Method for providing security service and electronic device therefor

Abstract

An electronic device comprising a communication circuit, a memory, and a processor is disclosed. The electronic device may be configured to: acquire a voice input; acquire additional information on the basis of the voice input; identify one or more candidate functions on the basis of at least one of the voice input or the additional information; encrypt personal information if the personal information is included in the additional information; generate a prompt on the basis of the one or more candidate functions, the encrypted personal information, and the voice input; transmit the generated prompt to an external artificial intelligence server by using the communication circuit; and receive, from the external artificial intelligence server, a response generated on the basis of the prompt.

Description

Method for providing security services and electronic devices therefor

[0001] Embodiments disclosed in this document relate to a method for providing a security service and an electronic device therefor.

[0002] Generative AI services utilizing AI models such as the large language model (LLM) or the large vision model (LVM) are widely used. Generative AI services receive prompts from users and generate outputs using those prompts. Generative AI services can train AI models using data from the received prompts. When prompt data is used to train an AI model, information contained in the training prompts may be included in the outputs. For example, if the prompts contain personal information about the user, the user's personal information may be exposed to other users through the AI ​​model.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0004] An electronic device according to an embodiment disclosed in the present document may include a communication circuit, a memory, and at least one processor communicatively connected to the communication circuit and the memory. The memory may store instructions that may be individually or collectively executed by the at least one processor. The instructions, when executed, may cause the electronic device to obtain a voice input, obtain additional information based on the voice input, and identify one or more candidate functions based on at least one of the voice input and the additional information. The instructions, when executed, may cause the electronic device to encrypt the personal information, if the additional information includes personal information, and generate a prompt based on the one or more candidate functions, the encrypted personal information, and the voice input. The instructions, when executed, may cause the electronic device to transmit the generated prompt to an external artificial intelligence server using the communication circuit, and to receive a response generated based on the prompt from the external artificial intelligence server. The above response may include at least a portion of the encrypted personal information.

[0005] In addition, a service providing method according to an embodiment disclosed in the present document may include an operation of obtaining a voice input, an operation of obtaining additional information based on the voice input, an operation of identifying one or more candidate functions based on at least one of the voice input or the additional information, an operation of encrypting the personal information when the additional information includes personal information, an operation of generating a prompt based on the one or more candidate functions, the encrypted personal information, and the voice input, an operation of transmitting the generated prompt to an external artificial intelligence server, and an operation of receiving a response generated based on the prompt from the external artificial intelligence server. The response may include at least a portion of the encrypted personal information.

[0006] A computer-readable storage medium according to an embodiment disclosed in this document can store instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the service providing method.

[0007] Figure 1 illustrates a service provision environment according to one embodiment.

[0008] FIG. 2 illustrates a block diagram of a user device and an electronic device according to one embodiment.

[0009] Figure 3a illustrates a block diagram of a service providing system according to one embodiment.

[0010] FIG. 3b illustrates a block diagram of a service providing system according to one embodiment.

[0011] Figure 4 is a flowchart of a service provision method according to one embodiment.

[0012] Figure 5 illustrates a personal information setting screen according to one embodiment.

[0013] Figure 6 illustrates a personal information encryption history screen according to one embodiment.

[0014] Figure 7 is a flowchart of a prompt generation method according to one embodiment.

[0015] Figure 8a illustrates a prompt structure according to one embodiment.

[0016] Figure 8b illustrates a system prompt according to one embodiment.

[0017] Figure 8c illustrates the contents of a candidate function according to one embodiment.

[0018] FIG. 8d illustrates the contents of user data according to one embodiment.

[0019] Figure 9 illustrates a service provision screen according to one embodiment.

[0020] Figure 10 illustrates a service provision screen according to one embodiment.

[0021] Figure 11 is a flowchart of a service provision method according to one embodiment.

[0022] FIG. 12 is a block diagram of an exemplary electronic device capable of performing the operations described in this document.

[0023] FIG. 13 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0024] Fig. 14 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0025] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0026] Hereinafter, various embodiments of the present invention will be described with reference to the attached drawings. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that the present invention encompasses various modifications, equivalents, and / or alternatives of the embodiments.

[0027] Figure 1 illustrates a service provision environment according to one embodiment.

[0028] Referring to FIG. 1, according to one embodiment, a service providing environment may include user devices (10a, 10, 10c), an electronic device (20), and an artificial intelligence server (30). For example, the user devices (10a, 10, 10c), the electronic device (20), and the artificial intelligence server (30) may be able to communicate with each other via a network (99) (e.g., an IP (Internet Protocol) network). The structure and form of the service providing environment described with respect to FIG. 1 is an example, and a person skilled in the art will understand that modified structures and forms may be utilized. For example, the number of user devices associated with a user (90) may be different from that in FIG. 1.

[0029] For example, the user devices (10a, 10b, 10c) may include any device configured to receive input from the user (90). For example, the first user device (10a) may be a mobile phone. The first user device (10a) may be configured to receive voice input, touch input, and / or button input from the user (90). The second user device (10b) may include an artificial intelligence speaker. The second user device (10b) may be configured to receive voice input from the user (90). The third user device (10c) may include any wearable device (e.g., a smart watch). The third user device (10c) may be configured to receive voice input, touch input, and / or button input from the user (90).

[0030] The user devices (10a, 10b, 10c) may be referred to as any device having a voice assistant application installed. The user (90) may invoke the voice assistant by performing a designated utterance (e.g., a wake-up word) or a designated input. After the voice assistant is invoked, the user devices (10a, 10b, 10c) may be configured to process the received user input.

[0031] In one example, the user devices (10a, 10b, 10c) can process user input without the cooperation of the electronic device (20). The user devices (10a, 10b, 10c) can be configured to provide a response by directly processing the received user input. In one example, the user devices (10a, 10b, 10c) can process the user input through cooperation with the electronic device (20). The user devices (10a, 10b, 10c) can transmit the received user input to the electronic device (20) and receive a response corresponding to the user input from the electronic device (20). The user devices (10a, 10b, 10c) can be configured to provide the received response.

[0032] For example, the electronic device (20) may include any server configured to provide voice services to the user devices (10a, 10b, 10c). The electronic device (20) may receive user input through a connection (e.g., a secure connection) with the user devices (10a, 10b, 10c) and generate a response to the user input. The electronic device (20) may transmit the generated response to the user devices (10a, 10b, 10c).

[0033] According to one embodiment, the user device (10a, 10b, 10c) and / or the electronic device (20) may process user input using an artificial intelligence server (30). The artificial intelligence server (30) may include a server configured to generate results using any artificial intelligence model (e.g., a large language model (LLM), a large vision model (LVM), or a large multimodal model (LMM)). The artificial intelligence server (30) may be configured to perform natural language processing using the artificial intelligence model.

[0034] For example, the user device (10a, 10b, 10c) and / or the electronic device (20) can generate a prompt based on a user input. The user device (10a, 10b, 10c) and / or the electronic device (20) can transmit the generated prompt to the artificial intelligence server (30). Based on the transmission of the prompt, the user device (10a, 10b, 10c) and / or the electronic device (20) can receive a result from the artificial intelligence server (30). For example, the result can be generated by inputting the prompt into an artificial intelligence model. The user device (10a, 10b, 10c) and / or the electronic device (20) can provide a response based on the received result. For example, the user device (10a, 10b, 10c) can provide a response to the user (90) based on the received result. The electronic device (20) can transmit a response based on the received result to the user device (10a, 10b, 10c).

[0035] As described above, the voice assistant service of the present disclosure can generate responses using an artificial intelligence model. By providing a voice assistant service using an artificial intelligence model, more complex user inputs can be flexibly processed. For example, the user (90) can perform input in a more intuitive manner.

[0036] Typically, the service provider of the user device (10a, 10b, 10c) and the provider of the voice assistant service (e.g., the operator of the electronic device (20)) are the same, while the artificial intelligence server (30) may be operated by a third party. Since the voice assistant service often processes personal information, the protection of personal information is required. When processing some operations of the voice assistant service using the artificial intelligence server (30), there is a risk of personal information leakage. When the user's information is used as training data by the artificial intelligence server (30), there is a risk that the user's information may be exposed to any third party. In addition, since one more external entity (e.g., the artificial intelligence server (30)) is involved compared to the provision of a typical voice assistant service, security vulnerabilities increase.

[0037] According to examples of the present disclosure, personal information transmitted to the AI ​​server (30) can be encrypted. Encrypting personal information can reduce the risk of personal information exposure. For example, personal information can be anonymized to the AI ​​server (30) through encryption. Examples of the present disclosure can encrypt personal information based on the essentiality of the data. By encrypting only information essential to generating a response, personalized responses can be provided while reducing the risk of personal information exposure.

[0038] FIG. 2 illustrates a block diagram of a user device and an electronic device according to one embodiment.

[0039] Referring to FIG. 2, according to one embodiment, the user device (10) may include a processor (120), a memory (130), a display (160), a camera (170), an interface (180), and / or a communication circuit (190). The user device (10) may correspond to the first user device (10a), the second user device (10b), the third user device (10c) of FIG. 1, and / or the electronic device (1200) of FIG. 12. The user device (10) may include a configuration similar to the electronic device (1200) described below with reference to FIG. 12. For example, the processor (120) may correspond to at least one processor (1210) of FIG. 12. For example, the memory (130) may correspond to the memory (1220) of FIG. 12. For example, the display (160) may correspond to the display (1240) of FIG. 12. For example, the camera (170) may correspond to the image sensor (1250) and / or the image signal processor (ISP) 1214 of FIG. 12. For example, the communication circuit (190) may correspond to the communication circuit (1260) of FIG. 12. The configuration of the user device (10) illustrated in FIG. 2 is exemplary, and the configuration of the user device (10) is not limited thereto. For example, the user device (10) may further include configurations not illustrated in FIG. 2. For example, the user device (10) may not include at least one of the configurations illustrated in FIG. 2.

[0040] The processor (120) may be communicatively, electrically, operatively, or functionally connected to the memory (130), the display (160), the camera (170), the interface (180), and / or the communication circuitry (190). In various embodiments of the present disclosure, when a component is “operatively” connected to another component, it may mean that the component is connected so as to be able to operate the other component. For example, the component may operate the other component by transmitting a control signal to the other component, either directly or via another component. In various embodiments of the present disclosure, when a component is “functionally” connected to another component, it may mean that the component is connected so as to be able to execute a function of the other component. For example, the component may execute a function of the other component by transmitting a control signal to the other component, either directly or via another component.

[0041] The processor (120) may include at least one processor. For example, the processor (120) may include an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), and / or a communication processor (CP). The processor (120) may include at least one chip or one chipset. In the present disclosure, the processor (120) may be referred to as a hardware component having an architecture by at least one processing circuit. For example, the processor (120) may be mounted on a substrate (e.g., a printed circuit board) located within the user device (10) and may communicate with other components of the user device (10) through at least one conductive path formed on the substrate.

[0042] The memory (130) can store instructions. When executed by the processor (120), the instructions can cause the user device (10) to perform various operations. For example, the instructions can be individually or collectively executed by at least one processor to cause the user device (10) to perform various operations. In various embodiments of the present disclosure, the operation of the user device (10) can be referred to as an operation performed by the processor (120) by executing instructions stored in the memory (130). The memory (130) can be referred to as a hardware component for data storage.

[0043] The display (160) may include at least one pixel configured to display an image. In one example, the display (160) may include multiple displays. For example, the display (160) may include a left-eye display and a right-eye display. The display (160) may include a front display and / or a rear display. The display (160) may include at least one of a see-through display, a flexible display, a rollable display, a foldable display, and / or a rigid display.

[0044] The camera (170) may include at least one camera configured to acquire visual information. The user device (10) may, for example, use an image acquired using the camera (170) as at least part of the user input.

[0045] The interface (180) may include at least one device configured to receive input. For example, the interface (180) may include a touch circuit (181) configured to receive a touch input (e.g., a touch screen display). The interface (180) may include at least one microphone (182) configured to receive a voice input. The interface (180) may include at least one device for output. For example, the interface (180) may include a haptic module for tactile output, at least one speaker (184) for sound output, and / or an indicator. The interface (180) may include any human interface device (HID). In one example, the interface (180) may include a button (185). According to one embodiment, the processor (120) may be configured to receive an input using the interface (180) and process the received input.

[0046] The communication circuit (190) may be configured to perform short-range wireless communication and / or long-range wireless communication. The communication circuit (190) may include a network interface card (NIC). The processor (120) may communicate with other external electronic devices based on wireless communication and / or wired communication, for example, using the communication circuit (190). The processor (120) may communicate with an external device via an IP (Internet Protocol) network, for example, using the communication circuit (190).

[0047] According to one embodiment, the electronic device (20) may include a processor (220), a memory (230), and / or a communication circuit (290). In one example, the electronic device (20) may configure a service provision system described below together with a user device (10) and an artificial intelligence server (30). As described below, the service provision system may be implemented by the user device (10) and the artificial intelligence server (30).

[0048] The processor (220) may be communicatively, electrically, operatively, or functionally connected to the memory (230) and the communication circuit (290). The processor (220) may include at least one processor. For example, the processor (220) may include an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), and / or a communication processor (CP). The processor (220) may include at least one chip or one chipset.

[0049] The memory (230) can store instructions. When executed by the processor (220), the instructions can cause the electronic device (20) to perform various operations. For example, the instructions can be individually or collectively executed by at least one processor to cause the electronic device (20) to perform various operations. In various embodiments of the present disclosure, the operation of the electronic device (20) can be referred to as an operation performed by the processor (220) by executing instructions stored in the memory (230). The memory (230) can be referred to as a hardware component for storing data.

[0050] The communication circuit (290) may be configured to perform short-range wireless communication and / or long-range wireless communication. The communication circuit (290) may include a network interface card (NIC). The processor (220) may communicate with other external electronic devices based on wireless communication and / or wired communication, for example, using the communication circuit (290). The processor (220) may communicate with an external device via an IP (Internet Protocol) network, for example, using the communication circuit (290).

[0051] Figure 3a illustrates a block diagram of a service providing system according to one embodiment.

[0052] Referring to FIG. 3A, according to one embodiment, the service providing system may include a client (301), a server (302), and a service server (303). For example, the client (301) may correspond to the user devices (10a, 10b, 10c) of FIG. 1. The server (302) may correspond to the electronic device (20) of FIG. 1. The service server (303) may correspond to the artificial intelligence server (30) of FIG. 1.

[0053] According to one embodiment, the client (301) may include a client application (310), a data platform (315), and an application database (DB, 320). The client (301) may include any electronic device configured to process user input through communication with the server (302). For example, the components of the client (301) may be software modules (e.g., threads, functions, databases, and / or programs) implemented by the client (301) executing instructions stored in a memory using a processor.

[0054] The client application (310) may act as an interface between a user and an application. For example, the client application (310) may include an application configured to provide a voice assistant service. The client application (310) may be invoked based on, for example, a specified input (e.g., a wake-up word). The client application (310) may acquire user input and transmit the acquired user input to the server (302). The client application (310) may be configured to receive a response to the user input from the server (302) and provide the received response to the user.

[0055] The data platform (315) can manage and retrieve data of applications stored in the client (301). For example, the data platform (315) can manage and retrieve data of various applications stored in the application DB (320). The application DB (320) can include data generated and / or user input data related to various applications. For example, the application data of the application DB (320) can include personal information. In one example, the data platform (315) can collect data related to various applications, convert the collected data into a format having a unified structure, and store the collected data in the application DB (320). In one example, the application DB (320) can be stored in a secure memory area or encrypted.

[0056] According to one embodiment, the server (302) may include a front end (325), an automatic speech recognition module (330), a natural language processing module (340), a retrieval augmented generation (RAG) service module (350), and / or a proxy service (360). The server (302) may process user input received from the client (301) to generate a response. The server (302) may reference a knowledge database of an external data source (e.g., a service server (303)) to generate the response. In the example of FIG. 3A, the server (302) may control the voice assistant service of the client (301) and simultaneously act as an interface for the service server (303). For example, the server (302) may correspond to the intelligent server (1400) of FIG. 13. The server (302) may include at least some of the components of the intelligent server (1400) of FIG. 13. For example, the components of the server (302) may be software modules (e.g., threads, functions, databases, and / or programs) implemented by the server (302) executing instructions stored in memory using a processor.

[0057] The front end (325) may act as an interface of the server (302) to the client application (310). For example, the front end (325) may correspond to the front end (1410) of FIG. 13. The front end (325) may receive input (e.g., voice input) and / or personal information from the client (301). The front end (325) may provide a generated response to the client (301).

[0058] The automatic speech recognition module (330) may be configured to perform automatic speech recognition on a speech input received from the client (301). The automatic speech recognition module (330) may be configured to extract text from the speech input. For example, the automatic speech recognition module (330) may correspond to the automatic speech recognition module (1421) of FIG. 13.

[0059] The natural language processing module (340) may be configured to process text extracted by the automatic speech recognition module (330). For example, the natural language processing module (340) may include a natural language understanding module (341), a feature search module (342), an LLM orchestrator (343), and / or a prompt generator (344).

[0060] The natural language understanding module (341) can identify a user's intent and / or parameters (e.g., entities and / or slots) based on text data of a speech input. The user's intent corresponds to the speech input and may include information indicating an action (or function) that the user wishes to execute using the client (301). The slot may be detailed information related to the user's intent. The slot may be acquired based on a domain corresponding to the utterance. The slot may be variable information required to perform the action. In one example, the variable information constituting the slot may include a named entity. For example, the natural language understanding module (341) may correspond to the natural language understanding module (1423) of FIG. 14.

[0061] The feature search module (342) may be configured to search for features associated with user input and / or additional information (e.g., information acquired using a data retriever (351) and / or information received from the client (301). For example, the feature search module (342) may search for features (e.g., candidate features) associated with the user input based on text data recognized by the automatic speech recognition module (330) and / or intent recognized by the natural language understanding module (341). For example, the feature search module (342) may identify features that include parameters corresponding to the additional information as candidate features. In one example, the user input may include a name stored in the contacts of the client (301). In this case, the feature search module (342) may identify features associated with the contact (e.g., making a call, sending a message, etc.) as candidate features. The feature search module (342) may identify candidate features by supplementing the additional information with the recognized intent.

[0062] The feature search module (342) can obtain additional information related to the searched feature. The feature search module (342) can transmit the associated feature (e.g., a sign feature) and additional information to the LLM orchestrator (343).

[0063] The function search module (342) can search for at least one candidate function based on text data (e.g., user input corresponding text data) and / or intent (e.g., user input corresponding intent). A function can be referred to as a unit operation that can be performed by, for example, the client application (310). For example, the function search module (342) can search for at least one candidate function based on the similarity between an input (e.g., text data and / or intent) and a description of the function. The function search module (342) can search for the candidate function using data (e.g., metadata) associated with the function. For example, the data associated with the function can include a description of the operation of the function. For example, the function search module (342) can identify functions having a similarity greater than or equal to a specified value and / or N functions having the highest similarity (e.g., N is a natural number) as candidate functions. If the user's intention corresponds to sequential execution of multiple functions, the function search module (342) can identify multiple functions to be sequentially performed as candidate functions.

[0064] For example, metadata associated with a function may include a name of the function, a description of the function (e.g., a description of its operation), parameter information of the function, and / or the result of performing the function. For example, parameter information of the function may include a list of parameters associated with the function, the requirement of the parameter, the allowability of anonymization of the parameter, the anonymization algorithm of the parameter, and / or a description of the parameter. The list of parameters may include the names of the parameters used in connection with the function. The requirement of a parameter may indicate whether the parameter is mandatory to execute the associated function. The allowability of anonymization of the parameter may include information indicating whether encryption of the parameter, described below, can be applied. The anonymization algorithm of the parameter may include information of an algorithm that can be applied when encryption is applied to the parameter. The description of the parameter may include any information for describing the parameter.

[0065] The feature search module (342) may obtain additional information related to at least one candidate feature. For example, the feature search module (342) may obtain additional information related to at least one candidate feature using a data retriever (351). The acquisition of the additional information will be described later with respect to the data retriever (351).

[0066] The LLM orchestrator (343) may receive additional information, including at least one candidate feature identified by the feature discovery module (342), additional information, and personal information received from the client (301). The LLM orchestrator (343) may use the encryption module (353) to encrypt (e.g., anonymize) at least a portion of the additional information. If personal information received from the client (301) exists, the LLM orchestrator (343) may use the encryption module (353) to encrypt at least a portion of the personal information. The LLM orchestrator (343) may determine, for example, whether the additional information contains personal information that requires encryption. If personal information exists, the LLM orchestrator (343) may use the encryption module (353) to encrypt (e.g., anonymize) the personal information. Hereinafter, the term “additional information” may mean information received from the function search module (342) and / or information received from the client (301).

[0067] In this disclosure, personal information may be referred to as information associated with the identification of an individual and / or sensitive information belonging to an individual. The LLM orchestrator (343) may identify personal information from additional information. The additional information may, for example, include information corresponding to parameters of at least one candidate function. The LLM orchestrator (343) may determine whether at least some of the additional information corresponds to personal information based on specified rules and / or descriptions of the parameters. In one example, the LLM orchestrator (343) may encrypt at least some of the personal information using the encryption module (353). The LLM orchestrator (343) may transmit all identified personal information to the encryption module (353), and the encryption module (353) may return at least some of the encrypted personal information to the LLM orchestrator (343). In one example, the LLM orchestrator (343) may pass additional information to the encryption module (353), and identification and encryption of personal information may be performed by the encryption module (353).

[0068] The LLM orchestrator (343) can transmit user input (e.g., text data corresponding to voice input), at least one candidate feature, and additional information (e.g., encrypted additional information, if encryption is performed) to the prompt generator (344). The prompt generator (344) can use the information received from the LLM orchestrator (343) to generate a prompt to be input to an artificial intelligence model (e.g., an artificial intelligence model of the service server (303)). The prompt generator (344) can be configured to generate a prompt based on the information received from the LLM orchestrator (343) and a specified template (e.g., a text structure defining the structure of the prompt). The prompt generator (344) can use the trained artificial intelligence model to generate the prompt. For example, the prompt can include structured text. The structure of a prompt according to an example can be described below with reference to FIGS. 8A to 8D.

[0069] The proxy service (360) may act as an interface between the server (302) and the service server (303) (e.g., the service server (30) of FIG. 1). Through the proxy service (360), the server (302) may communicate with the service server (303) independently of the type of artificial intelligence model of the service server (303). Through the proxy service, a prompt may be transmitted to the service server (303).

[0070] The service server (303) may store at least one artificial intelligence model (e.g., LLM, LVM, and / or LMM). The service server (303) may process the received prompt using the artificial intelligence model to generate a result. For example, the result may include a function corresponding to the intent of the user input and parameters of the function. The result may include, for example, information about candidate functions to be executed to achieve the purpose corresponding to the intent of the user input (e.g., which candidate functions are to be executed in what order) and information about parameters of the candidate functions to be executed (e.g., parameter values). In one example, the service server (303) may fill in parameters for the candidate functions and include information about the candidate functions including parameter values ​​in the result. The service server (303) may fill in parameters of the candidate functions using information included in the prompt (e.g., additional information). In one example, the result may include a sequence of multiple candidate functions.

[0071] The server (302) can receive a result corresponding to the prompt from the service server (303) via the proxy service (360). The LLM orchestrator (343) can determine whether the received result includes encrypted personal information. If encrypted personal information is included, the LLM orchestrator (343) can decrypt the encrypted personal information using the encryption module (353). The LLM orchestrator (343) can send a response based on the result to the client (301) via the front end (325). For example, the LLM orchestrator (343) can generate a response by converting the received result (e.g., after or together with decryption) into a form that can be executed by the client application (310). The client application (310) can use the received response to perform a task corresponding to the user input (e.g., a task corresponding to the user intent). For example, the received response can include at least one function and associated parameter information. The client application (310) can perform a task by executing at least one function according to associated parameter information. In one example, the client application (310) can perform a task by executing multiple functions in the order indicated by the response.

[0072] The RAG service module (350) may be referred to as a module for retrieving and protecting personal information. The RAG service module (350) may reduce the risk of personal information exposure by encrypting personal information when the personal information is transmitted to an external server (e.g., service server (303)). For example, the RAG service module (350) may include a data retriever (351), an encryption module (353), a named entity database (NE DB) 355, and / or a personal data database (PD DB) 357.

[0073] The data retriever (351) can retrieve data associated with a candidate function based on information received from the function retrieval module (342) (e.g., information associated with the candidate function and / or parameters of the candidate function) and / or user input. For example, the data retriever (351) can retrieve data associated with the candidate function from the NE DB (355), the PD DB (357), and / or the application DB (320). The data associated with the candidate function can include, for example, information corresponding to parameters of the candidate function. The NE DB (355) can include, for example, information on proper nouns, celebrity names, group names, and / or nouns commonly used among users. The PD DB (357) can include, for example, information received from the client (301) (e.g., personal information). The PD DB (357) can be synchronized with at least a portion of the application DB (320) based on a synchronization service. In one example, the data retriever (351) may retrieve data associated with the intent of the user input and / or data associated with an entity identified from the user input. The data retriever (351) may transmit the retrieved information to the function search module (342).

[0074] The data retriever (351) can retrieve data based on user input. For example, the data retriever (351) can retrieve information associated with information (e.g., intent and / or keywords) identified from the user input from the NE DB (355), the PD DB (357), and / or the application DB (320). The NE DB (355) can include, for example, information on proper nouns, celebrity names, group names, and / or nouns commonly used among users. The PD DB (357) can include, for example, information (e.g., personal information) received from the client (301). The PD DB (357) can be synchronized with at least a portion of the application DB (320) based on a synchronization service. In one example, the data retriever (351) can retrieve data associated with the intent of the user input and / or data associated with an entity identified from the user input. The data retriever (351) can transmit the retrieved information to the function search module (342).

[0075] The data retriever (351) can retrieve data associated with a candidate function based on information received from the function search module (342) (e.g., information associated with the candidate function and / or parameters of the candidate function). For example, the data retriever (351) can retrieve data associated with the candidate function from the NE DB (355), the PD DB (357), and / or the application DB (320). The data associated with the candidate function can include, for example, information corresponding to parameters of the candidate function. The data retriever (351) can transmit the retrieved information to the function search module (342).

[0076] According to one embodiment, the encryption module (353) may encrypt information received from the LLM orchestrator (343) and return the encrypted information to the LLM orchestrator (343). For example, the encryption module (353) may receive additional information from the LLM orchestrator (343), identify personal information from the received additional information, and then encrypt at least a portion of the personal information. For example, the encryption module (353) may receive personal information from the LLM orchestrator (343) and encrypt at least a portion of the received personal information. The 'encryption' of the encryption module (353) may be referred to as anonymization of data that presupposes 'decryption'. For example, data encrypted by the encryption module (353) may not be decrypted by the service server (303).

[0077] For example, the encryption module (353) may determine whether personal information is encryptable based on the essentiality and / or encryption permissibility of the personal information. For example, if the encryption permissibility of the personal information (e.g., encryption permissibility of parameter information) is positive, the encryption module (353) may encrypt the personal information. For example, if the personal information is essential for a related function, the function may not be executed if the personal information is encrypted. In this case, the encryption module (353) may not encrypt the essential personal information. Therefore, even for the same personal information, whether to encrypt or not may be determined differently depending on the related function. For example, the first personal information may be designated as an essential parameter for the first function and a non-essential parameter for the second function. In this case, the encryption module (353) may not encrypt the first personal information if the candidate function is the first function, but may encrypt the first personal information if the candidate function is the second function.

[0078] In one example, the encryption module (353) can encrypt personal information based on a predefined policy. For example, the predefined policy may include a policy set by the service provider and / or a policy set by the user of the client (301). The predefined policy may be set for a specified function and a parameter mapped to the specified function. For example, the predefined policy may include an encryption requirement, an encryption level, and / or an encryption algorithm for the specified parameter. The encryption requirement may indicate whether encryption is allowed for the corresponding parameter. The encryption algorithm may indicate information about the encryption algorithm to be applied to the corresponding parameter. The encryption level may include an encryption policy for specific information set by the user or the service provider, for example. For example, if the encryption level of the specified parameter is equal to or higher than the specified level, data corresponding to the specified parameter may always be encrypted independently of the candidate function. For example, if the encryption level of the specified parameter is lower than the specified level, data corresponding to the specified parameter may be encrypted dependently depending on the candidate function.

[0079] In one example, the encryption module (353) may encrypt personal information using an artificial intelligence model. For example, the encryption module (353) may use the internal artificial intelligence model of the server (302) to determine the identification of personal information to be encrypted and / or whether to encrypt the personal information. Determining whether to encrypt the personal information may include determining an encryption algorithm for the personal information to be encrypted. By using the artificial intelligence model, whether to encrypt can be determined even for parameters for which no policy has been set.

[0080] In one example, the encryption module (353) may perform encryption using the necessity of personal information, encryption allowance, a predefined policy, and / or an artificial intelligence model.

[0081] The encryption module (353) can encrypt personal information based on a designated algorithm. For example, the designated algorithm may include pseudonymization, masking, differential privacy, and / or homomorphic encryption. Homomorphic encryption may include additive homomorphic encryption, multiplicative homomorphic encryption, and / or fully homomorphic encryption. The above-described algorithms are exemplary, and embodiments of the present disclosure are not limited thereto. The encryption of the encryption module (353) may include any algorithm that can encrypt data so that it can be decrypted, while anonymizing it to the other party (e.g., the service server (303)). For example, the encryption module (353) may add noise to the data to make it impossible to identify the value, replace the data with another value, change the value of part of the data, or obscure or delete part of the data. Table 1 shows encrypted data of the encryption module (353) according to an example.

[0082] Original dataEncrypted dataApplied algorithm2024-05-05, 1800-19002024-05-05, afternoonDifferential Privacy2024-05-05, 1800-1900{YYYY}-{MM}-{DD}, 1800-1900Pseudonymization2024-05-05, 1800-19002**4-**-05, 1**0-*900Masking2024-05-05, 1800-1900Data omission

[0083] When encrypting personal information, the encryption module (353) may store mapping information of the encrypted information. For example, first personal information may be encrypted with first encrypted personal information. In this case, the prompt generated by the prompt generator (344) may include the first encrypted personal information. Similarly, a result generated based on the prompt (e.g., a result received from the service server (303)) may include the first encrypted personal information. The LLM orchestrator (343) may decrypt the first encrypted personal information using the encryption module (353). The encryption module (353) may identify the first personal information mapped to the first encrypted personal information using the mapping information. The encryption module (353) may perform decryption by changing the first encrypted personal information included in the result into the first personal information.

[0084] In the example of FIG. 3a, voice recognition services and RAG services are described as being provided by the server (302), but embodiments of the present disclosure are not limited thereto. For example, as described below with reference to FIG. 3b, at least some of the functions of the server (302) may be performed by the client (301).

[0085] FIG. 3b illustrates a block diagram of a service providing system according to one embodiment.

[0086] Referring to FIG. 3b, according to one embodiment, the client (301) may be configured to provide an on-device voice assistant service and / or a RAG service. Unless otherwise described, descriptions of components having the same reference numbers may be referred to by the description of FIG. 3a.

[0087] According to one embodiment, the client (301) may include a client application (310), an automatic speech recognition module (311), a RAG service module (312), a data platform (315), an artificial intelligence module (317), an encryption module (319), and an application database (DB, 320). For example, the components of the client (301) may be software modules (e.g., threads, functions, databases, and / or programs) implemented by the client (301) executing instructions stored in memory using a processor.

[0088] The automatic speech recognition module (311) may be configured to perform automatic speech recognition on a speech input received from a client application (310). The automatic speech recognition module (311) may be configured to extract text from the speech input.

[0089] The artificial intelligence module (317) can process the text extracted by the automatic speech recognition module (311) using at least one artificial intelligence model. For example, the at least one artificial intelligence model can include LLM and / or sLLM (smaller LLM). The artificial intelligence module (317) can identify at least one keyword and / or intent from the extracted text. In one example, the client (301) can include a natural language understanding module that does not utilize an artificial intelligence model.

[0090] In one example, the artificial intelligence module (317) may identify at least one candidate function corresponding to the user's intent. In one example, the client (301) may include a module configured to identify candidate functions as described above with respect to the function search module (342) of FIG. 3A.

[0091] The RAG service module (312) can obtain additional data corresponding to at least one candidate function. For example, the RAG service module (312) can obtain additional data corresponding to at least one candidate function by searching the application database (320). The RAG service module (312) can obtain the additional data in a manner similar to that described above with respect to the data retriever (351) of FIG. 3A.

[0092] The encryption module (319) may encrypt at least a portion of personal information. For example, the encryption module (319) may encrypt personal data in a manner similar to that described above with respect to the encryption module (353) of FIG. 3A.

[0093] The RAG service module (312) may generate a prompt based on encrypted personal data. For example, the RAG service module (312) may generate a prompt in a manner similar to that described above with respect to the prompt generator (344) of FIG. 3A.

[0094] The client (301) can transmit the generated prompt to the service server (303) through the server (302) or directly. The client (301) can provide a response based on the result received from the service server (303).

[0095] In the example of FIG. 3b, an example is described in which the voice service and RAG service are provided by the client (301), but a person skilled in the art will understand that at least some of the operations described above can be performed by the server (302).

[0096] Hereinafter, various examples are described focusing on the operation of an electronic device (e.g., electronic device (20) of FIG. 1) corresponding to a server (302), but embodiments of the present disclosure are not limited thereto. As described with respect to FIG. 3b, at least some of the operations of the electronic device described below may be performed by the client (301).

[0097] Figure 4 is a flowchart of a service provision method according to one embodiment.

[0098] The operations described below with respect to FIG. 4 may be referred to as operations of the electronic device (20) of FIG. 2. In one example, as described above with respect to FIG. 3B, at least some of the operations of FIG. 4 may be performed by the user device (10). The order of the operations described below with respect to FIG. 4 is merely an example, and embodiments of the present disclosure are not limited thereto. For example, at least some of the operations may be performed in a different order from that of FIG. 4, or may be performed substantially simultaneously with other operations of FIG. 4. At least some of the operations described below with respect to FIG. 4 may be omitted.

[0099] Referring to FIGS. 2 and 4 , in operation 405, the electronic device (20) may obtain a user input. For example, the user input may include a voice input obtained by the user device (10). The electronic device (20) may receive the voice input obtained by the user device (10) from the user device (10) using the communication circuit (290). The user input may include audio data corresponding to the user voice and / or text data based on a voice recognition result for the user voice. As described above with respect to FIG. 3A , the electronic device (20) may perform voice recognition for the user voice using the automatic voice recognition module (330).

[0100] In operation 410, the electronic device (20) may obtain personal data associated with the user input. For example, the electronic device (20) may obtain personal data using an intent and / or keyword (e.g., entity) identified from the user input. The electronic device (20) may search the PD DB (357) using the data retriever (351) of FIG. 3A. The electronic device (20) may obtain personal data associated with the intent and / or keyword from the PD DB (357). For example, the electronic device (20) may obtain personal data corresponding to a parameter mapped to the identified intent. The electronic device (20) may obtain personal data corresponding to a parameter whose correlation with the identified intent is greater than or equal to a specified value. The electronic device (20) may identify a category to which the keyword belongs and obtain personal data belonging to the identified category. If personal information is received from the user device (10) along with user input, the personal data obtained in operation 410 may include the personal information received from the user device (10).

[0101] In operation 415, the electronic device (20) may identify at least one candidate function using user input and personal data. The electronic device (20) may identify at least one candidate function corresponding to an intent included in the user input. The electronic device (20) may identify at least one candidate function associated with a keyword included in the user input. The electronic device (20) may identify a function that uses personal data as a parameter as at least one candidate function. For example, the electronic device (20) may identify at least one candidate function according to the operation of the function search module (342) of FIG. 3A.

[0102] In operation 420, the electronic device (20) may encrypt at least a portion of personal data corresponding to the parameters of the candidate function. For example, the electronic device (20) may encrypt at least a portion of personal data according to the operation of the encryption module (353) of FIG. 3A.

[0103] In operation 425, the electronic device (20) may generate a prompt using encrypted data and candidate functions. For example, the electronic device (20) may generate a prompt according to the operation of the prompt generator (344) of FIG. 3A. For example, the generated prompt may include a system prompt, a candidate function, user input, and / or encrypted data. As described above with respect to the encryption module (353), some of the personal information may not be encrypted. In this case, the electronic device (20) may generate a prompt using encrypted personal data, unencrypted personal data, and the candidate function.

[0104] In operation 430, the electronic device (20) can obtain a prompt-based result. For example, the electronic device (20) can obtain a prompt-based result by transmitting a prompt to the service server (303) of FIG. 3A and receiving a result generated based on the prompt from the service server (303).

[0105] In operation 435, the electronic device (20) can decrypt the encrypted personal data of the result. For example, the electronic device (20) can decrypt the encrypted personal data using the mapping information generated during the encryption process, as described above with respect to the encryption module (353) of FIG. 3A.

[0106] In operation 440, the electronic device (20) may execute a result-based function. For example, the electronic device (20) may decrypt the encrypted information of the result and then generate, as a response, execution information of a function including the decrypted information. By transmitting the response to the user device (10), the electronic device (20) may control the user device (10) to execute the result-based function.

[0107] Example 1

[0108] For example, the user input of operation 405 may be “Create a meeting schedule for next week.” In this case, the electronic device (20) may obtain the user’s schedule as personal data associated with the user input (e.g., operation 410). Based on the intent of the user input, the electronic device (20) may identify a function for creating a schedule (e.g., create_calendar_event) as a candidate function (e.g., operation 415). Table 2 may illustrate the structure of the identified schedule creation function.

[0109] NameCreate_calendar_eventdescriptionThis function creates a calendar event based on the user's command and personal data.ParametersIs_mandatoryParameter descriptionTitleTrueTitle of the new eventdatetimeTrueDate and time of the new eventlocationFalseLocation of the new eventReturnDescriptionEventCalendar event

[0110] As described in Table 2, the schedule creation function may require parameters such as title information, time information, and / or location information. To determine the time so as not to conflict with existing registered schedules, information from existing registered schedules may be required. Table 3 shows information from existing registered schedules.

[0111] IndexParameterValue1TitleWeekly meetingDatetime2024-05-29, 1500-1600LocationB tower meeting room 20042TitleLunchDatetime2024-06-05, 1200-1300LocationA tower dinning facility3TitleMonthly meetingDatetime2024-06-05, 1400-1500LocationB tower boardroom

[0112] For example, the electronic device (20) can obtain previously registered schedule information as personal data. In the case of time information, it can be considered essential data for schedule registration to prevent schedule conflicts. In this case, the electronic device (20) can provide the time information to an external server (e.g., an artificial intelligence service) without encrypting it. In the case of location information, as additional information in the generation of schedule information, there may be no problem in inferring the schedule even without accurate location information. In this case, the electronic device (20) can encrypt the location information (e.g., operation 420) and provide it to the external server. Table 4 below shows the personal data of Table 3 encrypted by the electronic device (20) (e.g., operation 420).

[0113] IndexParameterValue1TitleWeekly meetingDatetime2024-05-29, 1500-1600Location{location-1212}2TitleLunchDatetime2024-06-05, 1200-1300Location{location-0317}3TitleMonthly meetingDatetime2024-06-05, 1400-1500Location{location-1021}

[0114] In the example in Table 4, the location information is encrypted based on pseudonymization, but those skilled in the art will understand that any encryption method can be applied. When performing encryption, the electronic device (20) may store mapping information indicating the mapping between the original data and the encrypted data. Table 5 may show mapping information according to one example.

[0115] Original data, encrypted data, B tower meeting room 2004 {location-1212}, A tower dinning facility {location-0317}, B tower boardroom {location-1021}

[0116] The electronic device (20) can generate a prompt containing encrypted personal data (e.g., operation 425) and obtain a result based on the generated prompt (e.g., operation 430). For example, Table 6 shows information included in the obtained result.

[0117] NameCreate_calendar_eventParametersValueTitleWeeklymeetingdatetime2024-06-05, 1500-1600location{location-1212}

[0118] The electronic device (20) can decrypt the encrypted personal data (e.g., location-1212) of the result. The electronic device (20) can perform decryption using information from the original data mapped to the encrypted personal data. Table 7 can show the result decrypted by the electronic device (20).

[0119] NameCreate_calendar_eventParametersValueTitleWeekly meetingdatetime2024-06-05, 1500-1600locationB tower meeting room 2004

[0120] The electronic device (20) can execute (e.g., operation 440) a result-based function (e.g., schedule generation) by generating a response based on the decrypted result and transmitting the response to the user device (10).

[0121] Example 2

[0122] For example, the user input of operation 405 may be “Search for a route to the schedule report meeting place.” In this case, the electronic device (20) may obtain the user’s schedule as personal data associated with the user input (e.g., operation 410). Based on the intent of the user input, the electronic device (20) may identify a function for route search (e.g., find_navigation_path) as a candidate function (e.g., operation 415). Table 8 may represent the structure of the identified route search function.

[0123] NameFind_navigation_pathdescriptionThis function finds a path to the destination.ParametersIs_mandatoryParameter descriptionfromTruedepaturetoTruedestinationReturnDescriptionEventNavigation path

[0124] As described in Table 8, the route search function may require origin and destination information. Table 9 shows information about existing registered itineraries.

[0125] IndexParameterValue1TitleWeekly meetingDatetime2024-06-05, 1500-1600LocationB tower meeting room 20042TitleCompany dinnerDatetime2024-06-05, 1830-2000LocationSteak house AAA3TitleMonthly meetingDatetime2024-06-05, 1400-1500LocationB tower boardroom

[0126] For example, the electronic device (20) can obtain previously registered schedule information as personal data. In the case of location information, it can be considered essential data for route search. In this case, the electronic device (20) can provide the location information to an external server (e.g., an artificial intelligence service) without encrypting it. In the case of time information, as additional information for route search, route search may not cause problems even if accurate time information is not available. In this case, the electronic device (20) can encrypt the time information (e.g., operation 420) and provide it to the external server. Table 10 below shows the personal data of Table 9 encrypted by the electronic device (20) (e.g., operation 420).

[0127] IndexParameterValue1TitleWeekly meetingDatetime2**4-*5-*9, **00-1**0LocationB tower meeting room 20042TitleCompany dinnerDatetime20**-06-**, 1*30-**00LocationSteak house AAA3TitleMonthly meetingDatetime20*4-**-*5, **00-*500LocationB tower boardroom

[0128] In the example of Table 10, the time information is encrypted based on masking, but those skilled in the art will understand that any encryption method can be applied. When performing encryption, the electronic device (20) may store mapping information indicating the mapping between the original data and the encrypted data. Table 11 may show mapping information according to one example.

[0129] Original dataEncrypted data2024-06-05, 1500-16002**4-*5-*9, **00-1**02024-06-05, 1830-200020**-06-**, 1*30-**002024-06-05, 1400-150020*4-**-*5, **00-*500

[0130] The electronic device (20) can generate a prompt containing encrypted personal data (e.g., operation 425) and obtain a result based on the generated prompt (e.g., operation 430). If the result contains encrypted personal data, the electronic device (20) can decrypt the encrypted personal data. In the case of path finding, since time information is not used, the result may not contain encrypted personal data. In this case, the electronic device (20) may not perform a decryption operation (e.g., operation 435). The electronic device (20) can generate a response based on the result and execute a result-based function (e.g., schedule creation) by transmitting the response to the user device (10) (e.g., operation 440).

[0131] In the schedule creation and route search examples described above, encryption may be performed differently for the same information. For example, time information was not encrypted in the schedule creation example, but was encrypted in the route search example. According to one embodiment, the electronic device (20) may determine whether to encrypt personal data based on the candidate function.

[0132] In the schedule creation and route search examples described above, encryption may be performed differently for the same information. For example, time information was not encrypted in the schedule creation example, but was encrypted in the route search example. According to one embodiment, the electronic device (20) may determine whether to encrypt personal data based on the candidate function.

[0133] Example 3

[0134] For example, the user input of operation 405 may be “Call Steve.” In this case, the electronic device (20) may obtain contact information as personal data associated with the user input (e.g., operation 410). Based on the intent of the user input, the electronic device (20) may identify a function for searching for a contact (e.g., find_contact) and a function for making a call (e.g., make_call) as candidate functions (e.g., operation 415). Table 12 may represent the structure of the identified contact search function, and Table 13 may represent the structure of the identified call making function.

[0135] NameFind_contactdescriptionThis function searches contacts for a nameParametersnameReturncontact

[0136] Namemake_calldescriptionThis function makes a call to a caontact numberParametersContact number

[0137] The electronic device (10) can encrypt (e.g., operation 420) the contact information corresponding to “Steve” through anonymization and masking as shown in Table 14.

[0138] NameContact-name-1212Phone number010****5678

[0139] The electronic device (20) may generate a prompt containing encrypted personal data (e.g., operation 425) and obtain a result based on the generated prompt (e.g., operation 430). In one example, the result may include a sequence of multiple candidate functions. For example, a subsequent candidate function may be scheduled to be executed using the output of a preceding candidate function. In this example, Tables 15 and 16 illustrate examples of obtained results. For example, the result may include sequence information indicating an execution order among the multiple candidate functions.

[0140] NameFind_contactParametersContact-name-1212Returncontact

[0141] Namemake_callParameters${result.find_contact.number}

[0142] For example, in the example of Table 16, the parameter values ​​may be set to utilize the results of the contact search function. The electronic device (20) may determine, based on the received parameter values, that the contact search function is performed prior to the call origination function.

[0143] The electronic device (20) can decrypt encrypted personal data (e.g., contact-name-1212) (e.g., operation 435) and generate a response based on the decrypted personal data. The electronic device (20) can execute an outcome-based function (e.g., schedule creation) by transmitting the response to the user device (10) (e.g., operation 440). For example, the user device (20) can execute a contact search function included in the response to obtain contact information corresponding to “Steve.” Thereafter, the user device (10) can execute a call origination function using the phone number of the obtained contact information.

[0144] The encryption examples of the electronic device (20) described above with reference to FIG. 4 are for illustrative purposes only, and the embodiments of the present disclosure are not limited thereto. For example, as described below with reference to FIGS. 5 to 7, the electronic device (20) may be configured to perform various encryption operations.

[0145] Figure 5 illustrates a personal information setting screen according to one embodiment.

[0146] Referring to FIGS. 2 and 5, according to one embodiment, the electronic device (20) may be configured to perform encryption based on a preset policy. For example, the preset policy may include a user policy. The electronic device (20) may receive the user policy from the user device (10).

[0147] According to one embodiment, the user device (10) may provide a personal information setting screen (500) using the display (160). For example, the personal information setting screen (500) may include a user interface (UI) for setting whether to allow encryption of personal information and whether to allow encryption of each piece of personal information.

[0148] For example, the personal information settings screen (500) may include a personal information encryption activation UI (510). The user device (10) may determine whether encryption of personal information is activated based on an input to the personal information encryption activation UI (510). If personal information encryption is not activated, the encryption policy may be set to not perform encryption on personal information. If personal information encryption is activated, the encryption policy may be set to perform encryption on personal information.

[0149] In one example, the personal information encryption activation UI (510) may include a UI for setting whether to allow encryption based on artificial intelligence. If personal information encryption is activated, the user device (10) may identify personal information based on artificial intelligence and encrypt the identified personal information. In one example, the user device (10) may store an internal LLM. The user device (10) may use the internal LLM to determine whether data should be anonymized, the degree of anonymization required, and the algorithm to be used. During this process, the internal LLM may perform data anonymization by referring to the descriptions of candidate functions, parameters, parameter descriptions, and / or retrieved personal data.

[0150] In one example, the user device (10) can identify and encrypt personal information based on the method described below with reference to FIG. 7. If personal information encryption is disabled, the user device (10) can determine whether to encrypt personal information based on whether encryption is permitted for each piece of personal information described below.

[0151] In one example, the personal information settings screen (500) may include a UI for setting whether to allow encryption for each piece of personal information. For example, the location information UI (521) may indicate encryption settings for location information. The name UI (522) may indicate encryption settings for name information. The schedule UI (523) may indicate encryption settings for schedule information. The phone number UI (524) may indicate encryption settings for phone number information. The medical information UI (525) may indicate encryption settings for medical information.

[0152] In one example, the personal information settings screen (500) may include a change history UI (515). When an input for the change history UI (515) is received, the user device (20) may provide the personal information encryption history screen of FIG. 6.

[0153] Figure 6 illustrates a personal information encryption history screen according to one embodiment.

[0154] Referring to FIGS. 2 and 6, according to one embodiment, the user device (10) may provide a personal information encryption history screen (600). The personal information encryption history screen (600) may include information on the result of personal information encryption and the date of encryption.

[0155] Referring to the first record (610), personal information such as a name (e.g., James) was encrypted into an anonymous number (e.g., JohnDoe) through anonymization. Referring to the second record (620), a phone number (e.g., 01012345678) was encrypted into an unspecified number (e.g., 010****4567) through masking. Referring to the third record (630), medical information (e.g., Urology) was encrypted into an unspecified hospital through differential privacy. Referring to the fourth record (640), specific address information (e.g., 7th Street) was encrypted into an unspecified house through differential privacy.

[0156] Figure 7 is a flowchart of a prompt generation method according to one embodiment.

[0157] The operations described below with respect to FIG. 7 may be referred to as operations of the electronic device (20) of FIG. 2. In one example, as described above with respect to FIG. 3B, at least some of the operations of FIG. 7 may be performed by the user device (10). The order of the operations described below with respect to FIG. 7 is merely an example, and embodiments of the present disclosure are not limited thereto. For example, at least some of the operations may be performed in a different order from that of FIG. 7, or may be performed substantially simultaneously with other operations of FIG. 7. At least some of the operations described below with respect to FIG. 7 may be omitted.

[0158] Referring to FIGS. 2 and 7, according to one embodiment, the electronic device (20) may apply different encryption levels to the same type of data. For example, as described below, the electronic device (20) may determine whether to encrypt data of the same type (e.g., name) differently.

[0159] In operation 705, the electronic device (20) may detect a name from the input. For example, the input may be “Send Steve a text message about Mozart’s music.” The electronic device (20) may detect two names (e.g., Steve and Mozart) from the input.

[0160] In operation 710, the electronic device (20) can determine whether the detected name exists in the contact list. For example, the electronic device (20) can determine whether the detected name exists in the contact list by searching the PD DB (357) using the data retriever (351) of 3a. In the case of the detected name “Steve,” it can exist in the contact list of the user device (10). The contact information of the user device (10) can be stored in the PD DB (357). In this case, the data retriever (351) can obtain contact information (e.g., name and phone number) corresponding to Steve from the PD DB (357).

[0161] If the name exists in the contact (e.g., operation 710-YES), at operation 725, the electronic device (20) may encrypt the name to generate a prompt. For example, the electronic device (20) may anonymize the name, mask the phone number, and encrypt additional information (e.g., contact information) corresponding to the name. For example, the electronic device (20) may encrypt the name in a manner similar to the first detail (610) of FIG. 6 . The electronic device (20) may encrypt the phone number in a manner similar to the second detail (620) of FIG. 6 .

[0162] If the name does not exist in the contact (e.g., operation 710-NO), in operation 715, the electronic device (20) can determine whether the name corresponds to a named entity (NE). For example, the electronic device (20) can determine whether the name corresponds to the NE by searching the NE DB (355) using the data retriever (351) of 3a. If the detected name exists in the NE DB (355), the electronic device (20) can determine that the name corresponds to the NE. For example, the data retriever (351) can obtain the name corresponding to Mozart and domain (e.g., musician) information of the name from the NE DB (355).

[0163] If the name does not correspond to an NE (e.g., operation 715-NO), the electronic device (20) may encrypt the name according to operation 725. If the name does not exist in the contact list and does not correspond to an NE, it may be unclear whether the name corresponds to personal information. In this case, the electronic device (20) may apply encryption. In one example, the electronic device (20) may not apply encryption to information for which it is unclear whether the name corresponds to personal information, depending on policy.

[0164] If the name corresponds to NE (e.g., action 715-YES), the electronic device (20) may generate the prompt without encrypting the name. In the example of FIG. 7, the name “Mozart” detected in the user input may not be encrypted, while “Steve” may be encrypted.

[0165] Figure 8a illustrates a prompt structure according to one embodiment.

[0166] Referring to FIGS. 2 and 8A, according to one embodiment, the electronic device (20) may generate a structured prompt. The electronic device (20) may generate the prompt using the prompt generator (344) of FIG. 3A. In one example, the electronic device (20) may generate the structured prompt using the format of the prompt stored in the memory (230).

[0167] A prompt (800) according to an example may include a system prompt (810), a candidate function (820), user data, and / or user input (840). In the examples of FIGS. 8A-8D , the user input (840) may be referenced as “Create a meeting schedule for next week.” The user input (840) may include a speech recognition result (e.g., text data) of the user input.

[0168] Figure 8b illustrates a system prompt according to one embodiment.

[0169] Referring to FIG. 8B , for example, the system prompt (810) may include information regarding general limitations for the artificial intelligence model. The system prompt (810) may correspond to a preamble within the prompt (800). In the example of FIG. 8B , the system prompt (810) may instruct the voice assistant to derive a result for performing a function based on the information being conveyed (e.g., candidate functions, user data, user commands (e.g., user input)). Additionally, the system prompt (810) may include information indicating that at least some of the information being conveyed may be encrypted.

[0170] Figure 8c illustrates the contents of a candidate function according to one embodiment.

[0171] Referring to FIG. 8C, for example, the candidate function (820) may include information on candidate functions searched by the function search module (342) of FIG. 3A. For example, the candidate function (820) may include information on the schedule generation function described above with respect to Table 2. Although only information on the schedule generation function is illustrated in FIG. 8C, a person skilled in the art will understand that information on candidate functions other than the schedule generation function may be included in the candidate function (820).

[0172] FIG. 8d illustrates the contents of user data according to one embodiment.

[0173] Referring to FIG. 8d, user data (830) may include additional information retrieved by the data retriever (351). If encryption is performed on at least a portion of the additional information, user data (830) may include encrypted personal information. In the example of FIG. 8d, user data (830) may include certain information corresponding to Table 4.

[0174] Figure 9 illustrates a service provision screen according to one embodiment.

[0175] Referring to FIGS. 2 and 9, the user device (20) may display a service provision screen (900). For example, the user device (20) may display the service provision screen (900) based on a response received from the electronic device (10). In the example of FIG. 9, the user device (20) may receive a response including the information of Table 8 described above from the electronic device (10).

[0176] For example, the call service notification (910) may indicate information about the service provided through the service provision screen (900) (e.g., application information). The schedule title (920) may include the title of the schedule included in the response. The time information (930) may include time information of the schedule included in the response. The location information (940) may include location information of the schedule included in the response.

[0177] The service provision screen (900) may include a UI for controlling a schedule generated based on a response. When an input for the save UI (950) is received, the user device (20) may use an associated service (e.g., a calendar service) to generate a schedule according to the information displayed on the service provision screen (900). When an input for the modify UI (960) is received, the user device (20) may allow modification of the displayed schedule information. When an input for the cancel UI (970) is received, the user device (20) may stop generating the schedule and stop displaying the service provision screen (900).

[0178] Figure 10 illustrates a service provision screen according to one embodiment.

[0179] Referring to FIGS. 2 and 10 , the user device (20) may display a service provision screen (1000). For example, the user device (20) may display the service provision screen (1000) based on a response received from the electronic device (10). In the example of FIG. 10 , the user device (20) may receive a response from the electronic device (10) according to the examples described above with respect to Tables 8 to 11. In the example of FIG. 10 , the response received from the electronic device (10) may include information on candidate destinations. The user device (20) may display the service provision screen (1000) for selecting a destination based on the received response.

[0180] The call service notification (1010) may indicate information about the service provided through the service provision screen (1000) (e.g., information about the application). The guide information (1015) may include information prompting the user to select one of the candidate destinations.

[0181] For example, the service provision screen (1000) may include a first candidate destination (1020), a second candidate destination (1030), and a third candidate destination (1040). The first candidate destination (1020), the second candidate destination (1030), and the third candidate destination (1040) may be candidate destinations selected by the artificial intelligence service based on the schedule information of Table 10.

[0182] Upon receiving an input for one of the candidate destinations, the user device (10) may perform subsequent actions based on the received response. For example, the user device (10) may execute a map service and then provide route information for the selected candidate destination.

[0183] Figure 11 is a flowchart of a service provision method according to one embodiment.

[0184] The operations described below with respect to FIG. 11 may be referred to as operations of the electronic device (20) of FIG. 2. In one example, as described above with respect to FIG. 3B, at least some of the operations of FIG. 11 may be performed by the user device (10). The order of the operations described below with respect to FIG. 11 is merely an example, and embodiments of the present disclosure are not limited thereto. For example, at least some of the operations may be performed in a different order from that of FIG. 11, or may be performed substantially simultaneously with other operations of FIG. 11. At least some of the operations described below with respect to FIG. 11 may be omitted.

[0185] Referring to FIGS. 2 and 11, according to one embodiment, the electronic device (20) can provide a service based on voice input.

[0186] In operation 1105, the electronic device (20) may obtain a voice input. For example, the electronic device (20) may obtain a voice input from the user device (10) using the communication circuit (290).

[0187] In operation 1110, the electronic device (20) can obtain additional information based on a voice input. The electronic device (20) can extract text data corresponding to the voice input by performing automatic voice recognition on the voice input. The electronic device (200) can identify at least one keyword, task, and / or intent from the text data through a natural language understanding module. For example, the electronic device (20) can process the voice input using the natural language processing module (340) described above with respect to FIG. 3A.

[0188] In operation 1115, the electronic device (20) may identify one or more candidate functions based on the voice input and additional information. For example, the electronic device (10) may identify an intent or task from the voice input. The electronic device (10) may identify one or more candidate functions among the plurality of functions based on their similarity to the identified intent or task. The electronic device (20) may identify one or more candidate functions using the function search module (342) described above with respect to FIG. 3A .

[0189] In operation 1120, the electronic device (20) may encrypt personal information included in the additional information. For example, the electronic device (20) may identify essential data among the required data of one or more candidate functions. The electronic device (20) may perform encryption on personal information other than information corresponding to the essential data among the additional information. The electronic device (20) may encrypt the personal information by applying at least one of pseudonymization, masking, differential privacy, homomorphic encryption, or deletion to at least a portion of the personal information. For example, the electronic device (20) may encrypt the personal information as described above with respect to the encryption module (353) of FIG. 3A .

[0190] As described above with reference to FIG. 5 , the electronic device (20) can identify personal information based on artificial intelligence and encrypt the identified personal information. In one example, the electronic device (20) can encrypt personal information according to the user settings described above with reference to FIG. 5 . For example, if the user has set location information encryption, the electronic device (20) can encrypt the location information.

[0191] In operation 1125, the electronic device (20) may generate a prompt based on one or more candidate functions, encrypted personal information, and a voice input. For example, the prompt may include one or more candidate functions, encrypted personal information, additional information other than the encrypted personal information, and text data corresponding to the voice input. The prompt may have the prompt structure described above with respect to FIG. 8A . For example, the electronic device (20) may generate the prompt using the prompt generator (344) described above with respect to FIG. 3A .

[0192] In operation 1130, the electronic device (20) may transmit a prompt to an external artificial intelligence server (30). The electronic device (20) may transmit the prompt to the artificial intelligence server (30) using a communication circuit (290).

[0193] For example, the artificial intelligence server (30) may be configured to process a prompt using at least one artificial intelligence model. The artificial intelligence server (30) may transmit a response including a result generated based on the prompt to the electronic device (20).

[0194] In operation 1135, the electronic device (20) may receive a response generated based on the prompt. For example, the response may include at least a portion of encrypted personal information. For example, the response may include at least one candidate function among the one or more candidate functions identified in operation 1115.

[0195] The electronic device (20) can generate a decrypted response by decrypting the encrypted personal information included in the response. The electronic device (20) can store mapping information between the additional information and the encrypted personal information in the memory (230). The electronic device (20) can decrypt the encrypted personal information included in the response using the mapping information. For example, the electronic device (20) can decrypt the encrypted personal information using the encryption module (353) of FIG. 3A.

[0196] The electronic device (20) can transmit the decrypted response to the user device (10). The electronic device (20) can provide a service corresponding to the voice input by controlling the user device (10). For example, the user device (10) can provide a service corresponding to the voice input by processing the decrypted personal information using at least one candidate function included in the decrypted response.

[0197] FIG. 12 is a block diagram of an exemplary electronic device (1200) capable of performing the operations described in this document.

[0198] Referring to FIG. 12, the electronic device (1200) may be one of various forms of electronic devices, such as a notebook (1290), smartphones (1291) having various form factors (e.g., a bar-type smartphone (1291-1), a foldable-type smartphone (1291-2), or a sliderable (or rollable) type smartphone (1291-3)), a tablet (1292), a cellular phone (not shown), and other similar computing devices (not shown). The components, their relationships, and their functions illustrated in FIG. 12 are exemplary only and do not limit the implementations described or claimed in this document. The electronic device (1200) may be referred to as a mobile device, a user device, a multi-function device, a portable device, or a server.

[0199] The electronic device (1200) may include components including at least one processor (1210) (hereinafter referred to as processor (1210)), at least one memory (1220) (hereinafter referred to as memory (1220)), at least one display (1240) (hereinafter referred to as display (1240)), at least one image sensor (1250) (hereinafter referred to as image sensor (1250)), at least one communication circuit (1260) (hereinafter referred to as communication circuit (1260)), and / or at least one sensor (1270) (hereinafter referred to as sensor (1270)). The above components are merely exemplary. For example, the electronic device (1200) may include other components (e.g., power management integrated circuitry (PMIC), audio processing circuitry, an antenna, a rechargeable battery, or an input / output interface). For example, some components may be omitted from the electronic device (1200). For example, several components can be combined into one component.

[0200] The processor (1210) may be implemented as one or more IC (integrated circuit (or circuitry)) chips and may perform various data processing. The processor (1210) may include at least one electrical circuit and may individually or collectively perform distributed processing of instructions (or programs, data, etc.) stored in the memory (1220). The processor (1210) may include a processor assembly including one or more processing circuits. The processor (1210) may include any processing circuit operative to control the performance and operations of one or more components of the electronic device (1200) (e.g., the memory (1220), the display (1240), the image sensor (1250), the communication circuit (1260), and / or the sensor (1270)). For example, the processor (1210) (e.g., an application processor (AP)) may be implemented as a system on chip (SoC) (e.g., a single chip or chipset). For example, the processor (1210) may be implemented as multiple cores (or at least one core circuit), multiple chips, or multiple chipsets. For example, the processor (1210) may include one or more processing circuits. For example, the processor (1210) may include one or more processing circuits configured to individually and / or collectively perform various functions of the present disclosure. As a non-limiting example, at least a portion of the processor (1210) may be included in a first chip of the electronic device (1200), and at least another portion of the processor (1210) may be included in a second chip of the electronic device (1200) that is different from the first chip of the electronic device (1200).

[0201] For example, the processor (1210) may include a central processing unit (CPU) (1211), a graphics processing unit (GPU) (1212), a neural processing unit (NPU) (1213), an image signal processor (ISP) (1214), a display controller (1215), a memory controller (1216), a storage controller (1217), a communication processor (CP) (1218), and / or a sensor interface (1219). These components of the processor (1210) are merely exemplary. For example, the processor (1210) may further include other components. For example, some components of the processor (1210) may be omitted from the processor (1210). For example, some components of the processor (1210) may be included as separate components of the electronic device (1200) outside the processor (1210). For example, some components of the processor (1210) (e.g., memory controller (1216)) may be included within other components (e.g., at least a portion of memory (1220), an interface (e.g., available for connection to at least one component of the electronic device (100)), a display (1240) and / or an image sensor (1250)).

[0202] The processor (1210) may cause other components of the electronic device (1200) to perform various operations by executing instructions stored in the memory (1220). The CPU (1211) (or central processing circuit) may be configured to control components of the processor (1210) based on the execution of instructions stored in the memory (1220) (e.g., volatile memory (1221) and / or non-volatile memory (1222)). The GPU (1212) (or graphics processing circuit) may be configured to execute parallel operations (e.g., rendering). The NPU (1213) (or neural processing circuit, or artificial intelligence (AI) chip) may be configured to execute operations for an artificial intelligence model (e.g., convolution computation). The ISP (1214) (or image signal processing circuit) may be configured to process a raw image acquired through the image sensor (1250) into a format suitable for a component within the electronic device (1200) or a component of the processor (1210). The display controller (1215) (or display control circuit, or display processing unit (DPU)) may be configured to process an image acquired from the CPU (1211), the GPU (1212), the ISP (1214), or the memory (1220) (e.g., the volatile memory (1221)) into a format suitable for the display (1240). The memory controller (1216) (or memory control circuit) may be configured to control reading data from the volatile memory (1221) and writing data to the volatile memory (1221). The storage controller (1217) (or storage control circuit) may be configured to control reading data from and writing data to the nonvolatile memory (1222).The CP (1218) (communication processing circuit) may be configured to process data obtained from a component of the processor (1210) into a format suitable for transmission to another electronic device via the communication circuit (1260), or to process data obtained from another electronic device via the communication circuit (1260) into a format suitable for processing by the component of the processor (1210). For example, the communication circuit (1260) may include one or more communication circuits. The sensor interface (1219) (or sensing data processing circuit, sensor hub) may be configured to process data on the state of the electronic device (1200) and / or the state of the surroundings of the electronic device (1200), obtained via the sensor (1270), into a format suitable for the component of the processor (1210).

[0203] The memory (1220) may include one or more storage media (or one or more storage devices). For example, the memory (1220) may include a memory assembly including one or more storage media. For example, the one or more storage media may include permanent memory (e.g., non-volatile memory (1222)) such as a hard drive, flash memory, read-only memory (ROM), semi-permanent memory (e.g., volatile memory (1221)) such as random access memory (RAM), any other suitable type of storage (or storage assembly), or any combination thereof. The memory (1220) may include cache memory, which is one or more different types of memory used to temporarily store data for a function or feature of the electronic device (1200). As a non-limiting example, the cache memory may be included within the processor (1210). The memory (1220) may be fixedly embedded within the electronic device (1200) or incorporated into one or more suitable types of components (e.g., a subscriber identity module (SIM) card and / or a secure digital (SD) card) that may be repeatedly inserted into and removed from the electronic device (1200).

[0204] For example, the memory (1220) may store one or more software applications, such as an operating system (or system) software application, a firmware software application, a driver software application, a plug-in (e.g., add-in, add-on, and / or applet) software application, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by the processor (1210). For example, the memory (1220) may store instructions callable by an application programming interface (API). For example, the memory (1220) may store instructions within a library.

[0205] FIG. 13 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0206] Referring to FIG. 13, an integrated intelligent system of one embodiment may include a first electronic device (1301) (e.g., electronic device (1200) of FIG. 12), a second electronic device (1302) (e.g., any device including a headset, earbuds, or microphone), an intelligent server (1400), and a service server (1499).

[0207] According to the illustrated embodiment, the first electronic device (1301) may include a communication interface (1310), an input / output (I / O) interface (1320), a processor (1330), and / or a memory (1340). The components listed above may be operatively or electrically connected to each other. For example, the electronic device (1301) may include at least some of the components of the electronic device (1200) of FIG. 12.

[0208] The communication interface (1310) can be connected to an external device (e.g., an intelligent server (1400) and / or a service server (1499)) via a first network (1399) (e.g., any network including a cellular network and / or a wireless local area network (WLAN)) to transmit and receive data. For example, the communication interface (1310) can correspond to the CP (1218) and / or the communication circuit (1260) of FIG. 12. The communication interface (1310) can support data transmission and reception with an external device (e.g., a second electronic device (1302)) via a second network (1398) (e.g., a short-range wireless communication network).

[0209] The I / O interface (1320) may receive user input, process received user input, and / or output results processed by the processor (1330) using input / output devices (not shown) (e.g., a microphone, a speaker, and / or a display (e.g., a display (1240) of FIG. 12)).

[0210] The processor (1330) may be operatively or electrically connected to a communication interface (1310), an I / O interface (1320), and / or a memory (1340) (e.g., the memory (1220) of FIG. 12) to perform a designated operation. For example, the processor (1330) may correspond to the processor (1210) of FIG. 12. The processor (1330) may execute a program (or one or more instructions) stored in the memory (1340) to perform a designated operation. For example, the processor (1330) may receive a user's voice input (e.g., a user's speech) through the I / O interface (1320). For example, the processor (1330) may receive a user's voice input received by the second electronic device (1302) from the second electronic device (1302) through the communication interface (1310). The processor (1330) can transmit voice input received through the communication interface (1310) to the intelligent server (1400). For example, the processor (1330) can include one or more processors.

[0211] The processor (1330) may receive a result corresponding to a voice input from the intelligent server (1400). For example, the processor (1330) may receive a plan corresponding to the voice input and / or a result calculated using the plan from the intelligent server (1400). For example, the plan may include, but is not limited to, information regarding a plurality of sequential operations to be executed by the first electronic device (1301) and / or another electronic device in relation to the voice input. The processor (1330) may receive a request from the intelligent server (1400) to obtain information (e.g., entities, slots, and / or parameters) necessary to generate a plan corresponding to the voice input. The processor (1330) may transmit the necessary information to the intelligent server (1400) in response to the request.

[0212] The processor (1330) can visually, tactilely, and / or audibly output the results of executing the operations specified according to the plan through the I / O interface (1320). For example, the processor (1330) can sequentially display the execution results of multiple operations on the display. As an example, the processor (1330) can display only the execution results of executing multiple operations (e.g., the execution result of one of the multiple operations or the last operation) on the display. The processor (1330) can provide feedback through the second electronic device (1302) by transmitting the execution results of the multiple operations or the execution results of at least some of the multiple operations to the second electronic device (1302).

[0213] The processor (1330) can recognize voice input. For example, the processor (1330) can execute an intelligent app (or a voice recognition app) to process voice input in response to a specified voice input (e.g., "Wake up!"). The processor (1330) can provide a voice recognition service through the intelligent app. The processor (1330) can transmit voice input to the intelligent server (1400) through the intelligent app and receive a result corresponding to the voice input from the intelligent server (1400).

[0214] In one example, the second electronic device (1302) may include a communication interface (1311), an input / output (I / O) interface (1321), a processor (1331), and / or a memory (1341). The components listed above may be operatively or electrically connected to each other. In one example, the second electronic device (1302) may be a set of multiple electronic devices configured as a single set (e.g., a left earbud and a right earbud).

[0215] The communication interface (1311) may support connection with an external device (e.g., the first electronic device (1301)) via a second network (1398). The I / O interface (1321) may receive user input, process received user input, and / or output a result processed by the processor (1331) using input / output devices (not shown) (e.g., at least one microphone, at least one speaker, and / or button).

[0216] The processor (1331) may be operatively and / or electrically connected to the communication interface (1311), the I / O interface (1321), and / or the memory (1341) to perform a designated operation. The processor (1331) may execute a program (or one or more instructions) stored in the memory (1341) to perform the designated operation. For example, the processor (1331) may receive a user's voice input (e.g., a user's speech) through the I / O interface (1321). In one example, the processor (1331) may perform voice activity detection (VAD) using at least one sensor (not shown) of the second electronic device (1302). The processor (1331) may detect a user's speech of the second electronic device (1302) using an acceleration sensor and / or a microphone.

[0217] The processor (1331) can transmit voice input received through the second network (1398) to the first electronic device (1301) using the communication interface (1311).

[0218] The processor (1331) may receive a result corresponding to a voice input from the first electronic device (1301). For example, the processor (1331) may receive data (e.g., text data) corresponding to the result corresponding to the voice input from the first electronic device (1301). The processor (1331) may output the received result through the I / O interface (1321).

[0219] The processor (1331) can recognize a voice input. For example, the processor (1331) can request the first electronic device (1301) to execute an intelligent app (or a voice recognition app) to process the voice input in response to a specified voice input (e.g., wake up!).

[0220] An intelligent server (1400) of one embodiment can receive a user's voice input from a first electronic device (1301) via a first network (1399). The intelligent server (1400) can convert audio data corresponding to the received voice input into text data. The intelligent server (1400) can generate at least one plan for performing a task corresponding to the user's voice input based on the text data. The intelligent server (1400) can transmit the generated plan or a result according to the generated plan to the first electronic device (1301) via the first network (1399).

[0221] An intelligent server (1400) of one embodiment may execute one or more programs including a front end (1410), a natural language platform (1420), a capsule database (1430), an execution engine (1440), and / or an end user interface (1450).

[0222] The front end (1410) can receive a voice input received by the first electronic device (1301) or the second electronic device (1302) from the first electronic device (1301). The front end (1410) can transmit a response corresponding to the voice input to the first electronic device (1301).

[0223] The natural language platform (1420) may include an automatic speech recognition (ASR) module (1421), a natural language understanding (NLU) module (1423), a planner module (1425), a natural language generator (NLG) module (1427), and / or a text-to-speech (TTS) module (1429).

[0224] The automatic speech recognition module (1421) can convert the voice input received from the first electronic device (1301) into text data. The natural language understanding module (1423) can identify the user's intent and / or parameters (e.g., entities and / or slots) based on the text data of the voice input. The user's intent corresponds to the voice input and may include information indicating an action (or function) that the user wishes to perform using the device. The slot may be detailed information related to the user's intent. The slot may be acquired based on a domain corresponding to the utterance. The slot may be variable information required to perform the action. In one embodiment, the variable information constituting the slot may include a named entity.

[0225] The planner module (1425) can generate a plan using the intent and / or parameters determined by the natural language understanding module (1423). For example, the planner module (1425) can determine at least one domain necessary to perform a task based on the determined intent. The domain may correspond to a category (or service) associated with an action (or function) that the user wishes to perform using the device. The domain may be classified according to a service (e.g., an app) related to the text. The domain may be related to the user's intent corresponding to the text. The domain may be classified according to, for example, the type of application that received the voice input and / or the type of service to be provided based on the voice input, but is not limited thereto. In one example, the determination of the domain may be performed by another module (e.g., the natural language understanding module (1423)). The planner module (1425) may determine a plurality of actions included in each of the at least one domain determined based on the intent. The planner module (1425) can determine parameters required to execute a plurality of determined actions or result values ​​output by the execution of the plurality of actions. The parameters and result values ​​can be defined as concepts of a specified format (or class). For example, the plan can include a plurality of actions and / or a plurality of concepts determined by the user's intention. The planner module (1425) can determine the relationship between the plurality of actions and / or the plurality of concepts in a step-by-step (or hierarchical) manner. For example, the planner module (1425) can identify the execution order of the plurality of actions (e.g., the plurality of actions determined based on the user's intention) based on the plurality of concepts (e.g., parameters required to execute the plurality of actions and results output by the execution of the plurality of actions). The planner module (1425) can generate a plan including association information (e.g., ontology) between the plurality of actions and the plurality of concepts.The planner module (1425) can create a plan using information (e.g., at least one capsule) stored in a capsule database (1430) in which a set of relationships between concepts and actions is stored.

[0226] The planner module (1425) can generate a plan based on an artificial intelligence (AI) system. For example, the AI ​​system can include one or more electronic devices and / or one or more processing circuits to execute a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN) and / or a recurrent neural network (RNN)), or a combination thereof. The AI ​​system described above is exemplary, and the AI ​​system can be an AI system based on any machine learning-based model. The planner module (1425) can select a plan corresponding to a user request from a set of predefined plans, or generate a plan in real time in response to a user request.

[0227] The natural language generation module (1427) can convert specified information into text. The information converted into text may be in the form of natural language speech. The text-to-speech module (1429) can convert text-to-speech information into speech information.

[0228] The capsule database (1430) can store information on the relationship between multiple concepts and actions corresponding to multiple domains (e.g., applications). The capsule database (1430) can store at least one capsule (e.g., capsule (1431) and / or capsule (1433)) in the form of a concept action network (CAN). For example, the capsule database (1430) can store actions for processing tasks corresponding to a user's voice input and / or parameters required for the actions in the form of a CAN. A capsule can include multiple action objects (or action information) and / or concept objects (or concept information) included in a plan. For example, capsules (1431, 1433) can be created for each domain and stored in the capsule database (1430), but are not limited thereto.

[0229] The execution engine (1440) can produce results using the generated plan. The end user interface (1450) can transmit the produced results to the first electronic device (1301).

[0230] According to one embodiment, some functions (e.g., natural language platform (1420)) or all functions of the intelligent server (1400) may be implemented in the first electronic device (1301). For example, the first electronic device (1301) may execute one or more programs including a natural language platform (e.g., natural language platform (1350) of FIG. 14) separately from the intelligent server (1400). For example, the electronic device (1301) may directly perform at least some of the operations of the natural language platform (1420) of the intelligent server (1400) (e.g., automatic speech recognition module (1421), natural language understanding module (1423), planner module (1425), natural language generation module (1427), and / or text-to-speech module (1429)).

[0231] In one embodiment, a service server (1499) may provide a service (e.g., food ordering or hotel reservation) designated to a first electronic device (1301). The service server (1499) may be a server operated by a different operator than the intelligent server (1400). The service server (1499) may communicate with the intelligent server (1400) and / or the first electronic device (1301) via the first network (1399). The service server (1499) may communicate with the intelligent server (1400) via a separate connection (not shown). The service server (1499) may provide the intelligent server (1400) with information for generating a plan corresponding to a voice input received by the first electronic device (1301) (e.g., operation information and / or concept information for providing a designated service). The provided information may be stored in a capsule database (1430). The service server (1499) can provide the result information according to the plan received from the first electronic device (1301) to the intelligent server (1400).

[0232] FIG. 14 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0233] Referring to FIG. 14, the integrated intelligence system may include a first electronic device (1301), a second electronic device (1302), and an intelligent server (1402). The first electronic device (1301) and the intelligent server (1402) may be connected to each other via a network and may transmit and receive data. The first electronic device (1301) and the second electronic device (1302) may be connected to each other via a short-range network and may transmit and receive data. According to one embodiment, the integrated intelligence system may be composed of a single device or multiple devices. For example, each device may include identical or similar functional configurations, and the configuration of one device may be replaced with the configuration of another device.

[0234] According to one embodiment, the intelligent server (1402) may include the entire configuration or at least a portion of the configuration of the intelligent server (1400) illustrated in FIG. 13. For example, the intelligent server (1402) may execute one or more programs including the natural language platform (1420) of the intelligent server (1400) of FIG. 13 and / or store the capsule database (1430) of FIG. 13. The configuration of the intelligent server (1402) is not limited to that illustrated in FIG. 14. For example, at least a portion of the natural language platform (1420) (e.g., the automatic speech recognition module (1421), the natural language understanding module (1423), the planner module (1425), the natural language generation module (1427), and / or the text-to-speech module (1429)) may be omitted from the intelligent server (1402). For example, the intelligent server (1402) may further include some components of the intelligent server (1400) of FIG. 13 (e.g., a front end (1410), an execution engine (1440), and / or an end user interface (1450)).

[0235] The first electronic device (1301) may execute one or more programs including a natural language platform (1350) and / or store a capsule database (1360). For example, the first electronic device (1301) may further execute one or more programs including a natural language platform (1350) and / or store a capsule database (1360) while including components of the first electronic device (1301) of FIG. 13.

[0236] The natural language platform (1350) may include an automatic speech recognition module (1351), a natural language understanding module (1353), a planner module (1355), a natural language generation module (1357), and / or a text-to-speech module (1359). The automatic speech recognition module (1351), the natural language understanding module (1353), the planner module (1355), the natural language generation module (1357), and the text-to-speech module (1359) may perform functions identical to or similar to those of the automatic speech recognition module (1421), the natural language understanding module (1423), the planner module (1425), the natural language generation module (1427), and the text-to-speech module (1429) of FIG. 13, respectively.

[0237] The capsule database (1360) may perform the same or similar functions as the capsule database (1430) of the intelligent server (1400, 1402). The capsule database (1360) may store information about the relationships between multiple operations and multiple concepts included in the plan generated by the planner module (1355). For example, the capsule database (1360) may store at least one capsule (e.g., capsule (1361) and / or capsule (1363)).

[0238] According to one embodiment, the first electronic device (1301) (e.g., the natural language platform (1350) and / or the capsule database (1360)) and the intelligent server (1402) (e.g., the natural language platform (1420) and / or the capsule database (1430)) may perform at least one function (or operation) in conjunction with each other, or may independently perform at least one function (or operation). For example, the first electronic device (1301) may perform voice recognition on its own without transmitting the received user's voice input to the intelligent server (1402). As an example, the first electronic device (1301) may convert the received voice input into text data through the automatic voice recognition module (1351). The first electronic device (1301) may transmit the converted text data to the intelligent server (1402). The intelligent server (1402) can determine (or identify) the user's intent and / or parameters from text data through the natural language understanding module (1423). The intelligent server (1402) can generate a plan through the planner module (1425) based on the determined intent and parameters and transmit the plan to the first electronic device (1301), or can transmit the determined intent and parameters to the first electronic device (1301) so that the plan is generated through the planner module (1355) of the first electronic device (1301). The planner module (1355) of the first electronic device (1301) can generate at least one plan for performing a task corresponding to a voice input using information stored in the capsule database (1360).

[0239] For example, the first electronic device (1301) can convert voice input received through the automatic speech recognition module (1351) into text data, and determine (or identify) the user's intention and / or parameters based on the text data through the natural language understanding module (1353). The first electronic device (1301) can generate a plan through the planner module (1355) based on the determined intention and parameters, or transmit the determined intention and parameters to the intelligent server (1402) so that the intelligent server (1402) can generate a plan through the planner module (1425). For example, if the planner module (1355) and / or the capsule database (1360) are not included in the first electronic device (1301), the first electronic device (1301) can generate a plan through the intelligent server (1402).

[0240] For example, the first electronic device (1301) can detect a speech pattern that is difficult to learn in an automatic speech recognition module (1351) or a natural language understanding module (1353), and transmit a voice input corresponding to the detected speech pattern to an intelligent server (1402) so that the automatic speech recognition module (1421) or the natural language understanding module (1423) of the intelligent server (1402) can process it.

[0241] Embodiments of the present disclosure are not limited to the examples described above. For example, the first electronic device (1301) may process the received voice input only within the terminal and produce a result corresponding to the voice input. For example, the first electronic device (1301) and the intelligent server (1402) may not only divide the voice input into modules and process it, but may also collaborate with each other to process it. For example, the natural language understanding module (1353) of the first electronic device (1301) and the natural language understanding module (1423) of the intelligent server (1402) may work together to produce a single result value (e.g., the user's intention and / or parameters).

[0242] The second electronic device (1302) can execute one or more programs including an automatic speech recognition (ASR) module (1352) and / or a text-to-speech (TTS) module (1354). For example, the second electronic device (1302) can include components of the second electronic device (1302) of FIG. 13 and execute one or more programs including an automatic speech recognition module (1352) and / or a text-to-speech module (1354). The automatic speech recognition module (1352) and the text-to-speech module (1354) can perform functions identical to or similar to the automatic speech recognition module (1421) and the text-to-speech module (1429) of FIG. 13, respectively.

[0243] According to one embodiment, the first electronic device (1301) and the second electronic device (1302) may perform at least one function (or operation) in conjunction with each other, or may independently perform at least one function (or operation). For example, the second electronic device (1302) may perform voice recognition for a voice input using an automatic voice recognition module (1352). The second electronic device (1302) may perform a function corresponding to the voice input based on the voice recognition. For example, the second electronic device (1302) may transmit a command corresponding to the recognized voice command to the first electronic device (1301). The second electronic device (1302) may output data received from the first electronic device (1301). For example, the second electronic device (1302) may convert data received from the first electronic device (1301) into voice using a text-to-speech conversion module (1354) and output the converted voice.

Claims

1. In an electronic device (20), Communication circuit (290); memory (230); and At least one processor (220) communicatively connected to the communication circuit and the memory, The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: Acquire voice input, Obtain additional information based on the above voice input, Identifying one or more candidate features based on at least one of the above voice input or the above additional information, If the above additional information contains personal information, the personal information will be encrypted, Generate a prompt based on said one or more candidate features, said encrypted personal information, and said voice input; Using the above communication circuit, the generated prompt is transmitted to an external artificial intelligence server, Store instructions for receiving a response generated based on the prompt from the external artificial intelligence server, An electronic device wherein the response includes at least a portion of the encrypted personal information.

2. In paragraph 1, The above instructions, when individually or in combination executed by the at least one processor, cause the electronic device to: Identifying an intent or task from the above voice input, An electronic device that identifies one or more candidate functions among a plurality of functions based on the similarity with the identified intention or the identified task.

3. In paragraph 1, An electronic device wherein the response comprises at least one candidate function among the one or more candidate functions.

4. In paragraph 3, The above instructions, when individually or in combination executed by the at least one processor, cause the electronic device to: Obtaining a decrypted response by decrypting the encrypted personal information included in the above response, An electronic device that provides the decrypted response.

5. In paragraph 4, The above instructions, when individually or in combination executed by the at least one processor, cause the electronic device to: Store mapping information between the above additional information and the above encrypted personal information, An electronic device that decrypts the encrypted personal information using the stored mapping information.

6. In paragraph 1, The above instructions, when individually or in combination executed by the at least one processor, cause the electronic device to: Transmitting the decrypted response to the user device, An electronic device that controls the user device to provide a service corresponding to the voice input by processing the decrypted personal information using at least one candidate function included in the decrypted response.

7. In paragraph 1, The above instructions, when individually or in combination executed by the at least one processor, cause the electronic device to: Identify essential data among the required data of one or more of the above candidate functions, An electronic device that encrypts personal information other than information corresponding to the essential data among the above additional information.

8. In paragraph 1, The above instructions, when individually or in combination executed by the at least one processor, cause the electronic device to: An electronic device that encrypts the personal information by applying at least one of pseudonymization, masking, differential privacy, homomorphic encryption, or deletion to at least a portion of the personal information.

9. In paragraph 1, An electronic device wherein the prompt comprises one or more candidate functions, the encrypted personal information, additional information other than the encrypted personal information, and text data corresponding to the voice input.

10. In paragraph 1, An electronic device, wherein the external artificial intelligence server is configured to process the prompt using at least one artificial intelligence model and transmit the response including the result generated based on the prompt to the electronic device.

11. In the method of providing service of electronic device (20), The act of acquiring voice input; An action to obtain additional information based on the above voice input; An action of identifying one or more candidate features based on at least one of the voice input or the additional information; If the above additional information contains personal information, the action of encrypting the personal information; An action of generating a prompt based on said one or more candidate functions, said encrypted personal information, and said voice input; An action of transmitting the generated prompt to an external artificial intelligence server; and Including an action of receiving a response generated based on the prompt from the external artificial intelligence server, A method for providing a service, wherein the response includes at least a portion of the encrypted personal information.

12. In paragraph 11, The act of identifying one or more of the above candidate features comprises: An action of identifying an intent or task from the above voice input; and A service providing method comprising an operation of identifying one or more candidate functions among a plurality of functions based on the similarity with the identified intention or the identified task.

13. In paragraph 11, A method for providing a service, wherein the response includes at least one candidate function among the one or more candidate functions.

14. In paragraph 13, An operation of obtaining a decrypted response by decrypting at least a portion of the encrypted personal information; and A service providing method further comprising an action of providing the decrypted response.

15. In paragraph 14, Further comprising an operation of storing mapping information between the above additional information and the above encrypted personal information, A service providing method, wherein the operation of obtaining the decrypted response includes an operation of decrypting at least a portion of the encrypted personal information using the stored mapping information.

Images