Electronic device based on artificial intelligence and voice recognition operation and control method thereof

Abstract

According to an embodiment, an electronic device may include: a microphone receiving ambient sound; a speaker; a memory in which parameter information is stored; and a processor that controls the speaker to perform speech recognition of the user's utterance speech when the user's utterance speech is detected through the microphone, and outputs a first response message corresponding to a speech recognition result. The processor may: detect an utterance expression in the speech recognition result indicating a re-request for the first response message, recognize at least one text constituting the first response message, determine at least one speech signal to be included in the second response message based on at least one parameter corresponding to the text, and generating a second response message emphasizing the determined at least one voice signal.

Description

Technical Field

[0001] Various embodiments disclosed herein relate to electronic devices (eg, intelligent agents) operating based on artificial intelligence and voice recognition and control methods thereof. Background Art

[0002] With the development of voice recognition technology in recent years, voice recognition functions can be implemented in various electronic devices each having a voice input device (e.g., a microphone). For example, through the voice recognition function, the electronic device can recognize the voice spoken by the user and perform a specific function based on the spoken voice. Such voice recognition functions have become more and more common recently in order to replace physical input in electronic devices and control the electronic devices using voice.

[0003] It is becoming increasingly popular to utilize the services of electronic devices by using intelligent agents. Intelligent agents can provide integrated functions to users by controlling various external devices functionally connected to the electronic devices. When the electronic device provides voice-based intelligent agent services, the user of the electronic device can operate various functions of the electronic device by using voice.

[0004] As the Internet of Things (IoT) begins to be used, devices used in users' living environments are connected to each other through wired and wireless networks and share information, and an environment is gradually being built in which various electronic devices such as TVs or refrigerators can be used to perform voice recognition of other external devices connected through the network.

[0005] In order to help understand the present disclosure, the above information can be provided as related technology. No judgment or determination is made as to whether any of the above content can be used as prior art related to the present disclosure. Summary of the invention

[0006] In the user's living environment, electronic devices that provide voice-based intelligent agent functions are increasing. When a user views the result of a voice instruction after a voice instruction, if the content on the screen cannot be seen, the user can mostly only rely on voice feedback, which has limitations. When the user "requests" voice feedback again, the electronic device may make an unexpected response that does not match the user's intention. When the user receives voice feedback from an electronic device under the same conditions as the previously provided voice feedback, it may be difficult for the user to correctly identify the desired information as before.

[0007] According to an embodiment, the electronic device may include: a microphone configured to receive ambient sound; a speaker; a memory configured to store parameter information; and a processor configured to perform voice recognition of a user's voice when the user's voice is detected through the microphone, and control the speaker to output a first response message corresponding to the result of the voice recognition. The processor is configured to detect a speech expression indicating a re-request for the first response message in the voice recognition result, identify at least one segment of the text constituting the first response message, and determine at least one voice signal to be included in the second response message based on at least one parameter corresponding to the text, and generate a second response message emphasizing at least one determined voice signal.

[0008] According to an embodiment, a method for providing voice feedback by an electronic device may include: in a case where a voice spoken by a user is detected through a microphone, performing voice recognition of the voice spoken by the user, and outputting a first response message corresponding to a result of the voice recognition, detecting an utterance in the voice recognition result indicating a re-request for the first response message, identifying at least one fragment of text constituting the first response message, determining at least one voice signal to be included in a second response message based on at least one parameter corresponding to the text, and generating a second response message emphasizing the at least one determined voice signal.

[0009] According to an embodiment, even after providing voice feedback, if the user “re-requests” the voice feedback, the electronic device can provide the voice feedback more efficiently than before, thereby improving the user experience.

[0010] According to the embodiment, the electronic device can more effectively deliver voice feedback to the user, thereby providing differentiated services. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram illustrating an integrated intelligence (AI) system according to an embodiment.

[0012] Figure 2 A form in which information on the relationship between concepts and actions is stored in a database according to an embodiment is shown.

[0013] Figure 3 A screen showing a user terminal processing a received voice input through a smart application according to an embodiment.

[0014] Figure 4 A configuration of an electronic device according to an embodiment is shown.

[0015] Figure 5 is a block diagram of an electronic device in a network environment according to various embodiments.

[0016] Figure 6aA configuration of a specified discourse processor for natural language processing according to an embodiment is shown. Figure 6b is a block diagram illustrating a text-to-speech (TTS) algorithm according to an embodiment.

[0017] Figure 7 is a flow chart illustrating a method for providing voice feedback by an electronic device according to an embodiment.

[0018] Figure 8a and Figure 8b is a flow chart illustrating a method for providing voice feedback by an electronic device according to an embodiment. DETAILED DESCRIPTION

[0019] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that they can be easily implemented by those skilled in the art to which the present disclosure belongs. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. When describing the accompanying drawings, the same or similar reference numerals may be used for the same or similar elements. In addition, in the accompanying drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for the sake of clarity and brevity.

[0020] Figure 1 is a block diagram illustrating an integrated intelligent system according to an embodiment.

[0021] Reference Figure 1 The integrated intelligent system 10 in the embodiment may include a user terminal 100, an intelligent server 200 and a service server 300.

[0022] The user terminal 100 according to an embodiment may be a terminal device (or electronic device) capable of connecting to the Internet, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a notebook computer, a TV, white goods, a wearable device, an HMD, or a smart speaker.

[0023] According to an embodiment, the user terminal 100 (eg, Figure 5 The electronic device 501 in the embodiment may include a communication interface 110, a microphone 120, a speaker 130, a display 140, a memory 150, and a processor 160. The constituent elements listed above may be operably or electrically connected to each other.

[0024] According to an embodiment, the communication interface 110 may be connected to an external device to send / receive data. According to an embodiment, the microphone 120 may receive a sound (e.g., a user's speech) and may convert the sound into an electrical signal. According to an embodiment, the speaker 130 may output the electrical signal as a sound (e.g., audio). According to an embodiment, the display 140 may be configured to display an image or video. According to an embodiment, the display 140 may display a graphical user interface (GUI) of an application (or application program) being run.

[0025] In an embodiment, display 140 may be configured to display an image or video. According to an embodiment, display 140 may also display a graphical user interface (GUI) of an application (or application program) being run. According to an embodiment, display 140 may receive touch input via a touch sensor. For example, display 140 may receive text input via a touch sensor in an on-screen keyboard area displayed in display 140.

[0026] According to an embodiment, the memory 150 may store a client module 151, a software development kit (SDK) 153, and a plurality of applications 155. The client module 151 and the SDK 153 may constitute a framework (or solution program) for performing general functions. In addition, the client module 151 or the SDK 153 may constitute a framework for processing user input (e.g., voice input, text input, touch input).

[0027] According to an embodiment, the plurality of applications 155 stored in the memory 150 may be programs for executing a specified function. According to an embodiment, the plurality of applications 155 may include a first application 155_1 and a second application 155_3. According to an embodiment, each of the plurality of applications 155 may include a plurality of operations for executing a specified function. For example, the plurality of applications 155 may include at least one of an alarm application, a message application, and a scheduling application. According to an embodiment, the plurality of applications 155 may be executed by the processor 160 to sequentially execute at least some of the plurality of operations.

[0028] According to an embodiment, the processor 160 may control the overall operation of the user terminal 100. For example, the processor 160 may be electrically connected to the communication interface 110, the microphone 120, the speaker 130, the display 140, and the memory 150 to perform a designated operation.

[0029] According to an embodiment, the processor 160 may also run a program stored in the memory 150 to perform a designated function. For example, the processor 160 may run at least one of the client module 151 or the SDK 153 to perform the following operation for processing a user input. For example, the processor 160 may control the operation of the plurality of applications 155 through the SDK 153. The following operation described as the operation of the client module 151 or the SDK 153 may be an operation performed by the processor 160.

[0030] According to an embodiment, the client module 151 may receive user input. For example, the client module 151 may generate a voice signal corresponding to the user's speech detected by the microphone 120. Alternatively, the client module 151 may receive a touch input detected by the display 140. Alternatively, the client module 151 may receive a text input detected by a keyboard or a screen keyboard. In addition, various types of user input detected by an input module included in the user terminal 100 or an input module connected to the user terminal 100 may be received. The client module 151 may send the received user input to the intelligent server 200. According to an embodiment, the client module 151 may send the status information of the user terminal 100 together with the received user input to the intelligent server 200. For example, the status information may be the running status information of the application.

[0031] According to an embodiment, the client module 151 may receive a result corresponding to the received user input. For example, the client module 151 may receive a result corresponding to the user input from the intelligent server 200. The client module 151 may display the received result on the display 140. In addition, the client module 151 may output the received result as audio through the speaker 130.

[0032] According to an embodiment, the client module 151 may receive a plan corresponding to the received user input. The client module 151 may display the results of multiple operations of running the application according to the plan on the display 140. For example, the client module 151 may sequentially display the results of running multiple operations on the display, and may output audio through the speaker 130. In another example, the user terminal 100 may display only some results of running multiple operations (e.g., the result of the last operation) on the display, and may output audio through the speaker 130.

[0033] According to an embodiment, the client module 151 may receive a request for obtaining information required for calculating a result corresponding to a user input from the smart server 200. The information required for calculating the result may be, for example, state information of the user terminal 100. According to an embodiment, the client module 151 may send the required information to the smart server 200 in response to the request.

[0034] According to an embodiment, the client module 151 may send information about the result of running a plurality of operations according to the plan to the smart server 200. The smart server 200 may use the result information to recognize that the received user input has been properly processed.

[0035] According to an embodiment, the client module 151 may include a voice recognition module. According to an embodiment, the client module 151 may recognize a voice input for running a limited function through the voice recognition module. For example, the client module 151 may execute an intelligent application for processing voice input to perform an organic action (e.g., wake up!) by specifying the input.

[0036] According to an embodiment, the intelligent server 200 may receive information related to the user voice input from the user terminal 100 through the communication network. According to an embodiment, the intelligent server 200 may change the data related to the received voice input into text data. According to an embodiment, the intelligent server 200 may generate a plan for executing a task corresponding to the user voice input based on the text data.

[0037] According to an embodiment, the plan may be generated by an artificial intelligence (AI) system. The artificial intelligence system may be a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN)) or a recurrent neural network (RNN)). Alternatively, the artificial intelligence system may be a combination of the foregoing systems or an artificial intelligence system different from the foregoing systems. According to an embodiment, the plan may be selected from a set of predetermined plans, or the plan may be generated in real time in response to a user request. For example, the artificial intelligence system may select at least one of a plurality of predetermined plans.

[0038] According to an embodiment, the intelligent server 200 may send the result calculated according to the generated plan to the user terminal 100, or may send the generated plan to the user terminal 100. According to an embodiment, the user terminal 100 may display the result calculated according to the plan on a display. According to an embodiment, the user terminal 100 may display the result of running the operation according to the plan on a display.

[0039] The intelligent server 200 according to the embodiment may include a front end 210 , a natural language platform 220 , a capsule database (capsule DB) 230 , a runtime engine 240 , a terminal user interface 250 , a management platform 260 , a big data platform 270 , and an analysis platform 280 .

[0040] According to an embodiment, the front end 210 may receive a user input received from the user terminal 100. The front end 210 may send a response corresponding to the user input.

[0041] According to an embodiment, the natural language platform 220 may include an automatic speech recognition module (ASR module) 221, a natural language understanding module (NLU module) 223, a planner module 225, a natural language generator module (NLG module) 227 and a text-to-speech module (TTS module) 229.

[0042] According to an embodiment, the ASR module 221 may convert the voice input received from the user terminal 100 into text data. According to an embodiment, the NLU module 223 may identify the user's intention by using the text data of the voice input. For example, the NLU module 223 may perform syntactic analysis or semantic analysis on the user input in the form of text data to identify the user's intention. According to an embodiment, the NLU module 223 may identify the meaning of the words extracted from the user input by using the language features (e.g., grammatical elements) of morphemes or phrases, and may determine the user's intention by matching the recognized meaning of the words with the intention. The NLU module 223 may obtain the intention information corresponding to the user's utterance. The intention information may be information representing the user's intention determined by interpreting the text data. The intention information may include information indicating the action or function that the user wants to perform using the device.

[0043] According to an embodiment, the planner module 225 can generate a plan by using the intent and parameters determined by the NLU module 223. According to an embodiment, the planner module 225 can determine multiple domains necessary for performing a task based on the determined intent. The planner module 225 can determine multiple actions included in each of the multiple domains determined based on the intent. According to an embodiment, the planner module 225 can determine the parameters required for running multiple determined actions, or the result values ​​output by running multiple actions. Parameters and result values ​​can be defined as concepts related to a specified format (or class). Therefore, the plan can include multiple concepts and multiple actions determined by the user's intent. The planner module 225 can determine the relationship between multiple actions and multiple concepts in a step-by-step (or hierarchical) manner. For example, the planner module 225 can determine the running order of multiple actions determined based on the user's intent based on multiple concepts. In other words, the planner module 225 can determine the running order of multiple actions based on the parameters required for running multiple actions and the results output by the running of multiple actions. Therefore, the planner module 225 can generate a plan including relationship information (e.g., ontology) between multiple actions and multiple concepts. Ontology indicates that the relationship between all entities can be defined in the form of “subject,” “predicate,” and “object.” Ontology may mean that the electronic device can define the relationship between all entities to understand the meaning of all entities.

[0044] The planner module 225 may generate a plan by using information stored in the capsule DB 230 in which a set of relations between concepts and actions is stored.

[0045] According to an embodiment, the NLG module 227 may change the designated information into a text form. The information changed into a text form may be in the form of a natural language speech. The TTS module 229 according to an embodiment may change the information in a text form into information in a voice form.

[0046] According to an embodiment, the capsule database 230 may store information about the relationship between a plurality of concepts and actions corresponding to a plurality of domains. For example, the capsule database 230 may store a plurality of capsules including a plurality of planned action objects (or action information) and concept objects (or concept information). According to an embodiment, the capsule database 230 may store a plurality of capsules in the form of a concept action network (CAN). According to an embodiment, a plurality of capsules may be stored in a function register included in the capsule database 230.

[0047] According to an embodiment, the capsule DB 230 may include a policy registry storing policy information required to determine a plan corresponding to the voice input. The policy information may include reference information for determining a plan when there are multiple plans corresponding to the user input. According to an embodiment, the capsule DB 230 may include a subsequent registry storing subsequent action information for suggesting subsequent actions to the user in a specified situation. The subsequent action may include, for example, a subsequent utterance. According to an embodiment, the capsule DB 230 may include a layout registry for storing layout information of information output through the user terminal 100. According to an embodiment, the capsule DB 230 may include a vocabulary registry storing vocabulary information included in the capsule information. According to an embodiment, the capsule DB 230 may include a conversation registry storing information about the user's conversation (or interaction).

[0048] According to an embodiment, capsule DB 230 can update the stored objects through a developer tool. The developer tool may include, for example, a function editor for updating an action object or a concept object. The developer tool may include a vocabulary editor for updating vocabulary. The developer tool may include a strategy editor for generating and registering a strategy for determining a plan. The developer tool may include a dialogue editor for generating a dialogue with a user. The developer tool may include a subsequent editor that can edit subsequent utterances for activating subsequent goals and providing prompts. The subsequent goal may be determined based on the currently configured goal, user preferences, or environmental conditions.

[0049] According to an embodiment, the capsule DB 230 may also be implemented in the user terminal 100. In other words, the user terminal 100 may include the capsule DB 230 storing information for determining an action corresponding to a voice input.

[0050] According to an embodiment, the operation engine 240 can calculate the result by using the generated plan. According to an embodiment, the terminal user interface 250 can send the calculated result to the user terminal 100. Therefore, the user terminal 100 can receive the result and can provide the received result to the user. According to an embodiment, the management platform 260 can manage the information used in the intelligent server 200. According to an embodiment, the big data platform 270 can collect user data. According to an embodiment, the analysis platform 280 can manage the quality of service (QoS) of the intelligent server 200. For example, the analysis platform 280 can manage the components and processing speed (or efficiency) of the intelligent server 200.

[0051] According to an embodiment, the service server 300 may provide a specified service (e.g., a food order or a hotel reservation) to the user terminal 100. According to an embodiment, the service server 300 may be a server operated by a third party. For example, the service server 300 may include a first service server 301, a second service server 303, and a third service server 305 operated by different third parties. According to an embodiment, the service server 300 may provide information for generating a plan corresponding to the received user input to the intelligent server 200. For example, the provided information may be stored in the capsule DB 230. In addition, the service server 300 may provide result information to the intelligent server 200 according to the plan.

[0052] In the above integrated intelligent system 10, the user terminal 100 can provide various intelligent services to the user in response to user input. The user input may include, for example, input using a physical button, touch input, or voice input.

[0053] According to an embodiment, the user terminal 100 can provide a voice recognition service through an internally stored intelligent application (or voice recognition application). In this case, for example, the user terminal 100 can recognize the user's speech or voice input (voice input) received through a microphone, and can provide a service corresponding to the recognized voice input to the user.

[0054] According to an embodiment, the user terminal 100 can perform a specified operation based on the received voice input alone or together with the intelligent server and / or the service server. For example, the user terminal 100 can run an application corresponding to the received voice input, and the specified operation can be performed by the running application.

[0055] According to an embodiment, when the client 100 provides a service together with the intelligent server 200 and / or the service server 300, the client 100 can detect user speech by using the microphone 120, and can generate a signal (or audio data) corresponding to the detected user speech. The client 100 can send the audio data to the intelligent server 200 by using the communication interface 110.

[0056] According to an embodiment, in response to the voice input received from the user terminal 100, the intelligent server 200 may generate a plan for performing a task corresponding to the voice input or a result of performing an action according to the plan. The plan may include, for example, a plurality of actions for performing a task corresponding to the user voice input, and a plurality of concepts related to the plurality of actions. These concepts may define parameter inputs for running a plurality of actions or result values ​​output by running a plurality of actions. The plan may include information about the relationship between the plurality of actions and the plurality of concepts.

[0057] The client 100 according to the embodiment may receive the response by using the communication interface 110. The client 100 may output a voice signal generated in the client 100 to the outside using the speaker 130 or may output an image generated in the client 100 to the outside using the display 140.

[0058] Figure 2 A form in which information on the relationship between concepts and actions according to various embodiments is stored in a database is shown.

[0059] Smart servers (e.g. Figure 1 The capsule database (e.g., Figure 1 The capsule DB 230 in the example may store a plurality of capsules in the form of a concept action network (CAN) 400. The capsule database may store actions for processing tasks corresponding to user voice input and parameters required for the actions in the form of a concept action network (CAN). The CAN may represent an organic relationship between an action and a concept defining parameters required to perform the action.

[0060] The capsule database may store a plurality of capsules (e.g., capsule A 401 and capsule B 404) each corresponding to a plurality of domains (e.g., applications). According to an embodiment, one capsule (e.g., capsule A 401) may correspond to one domain (e.g., application). In addition, one capsule may correspond to at least one service provider (e.g., CP1 402, CP2 403, CP3 405, or CP4 406) for performing a function of a domain related to the capsule. According to an embodiment, one capsule may include at least one action 410 and at least one concept 420 for performing a specified function.

[0061] According to an embodiment, a natural language platform (e.g., Figure 1 The natural language platform 220 in the example may generate a plan for performing a task corresponding to the received speech input by using capsules stored in the capsule database. For example, a planner module of the natural language platform (e.g., Figure 1 The planner module 225 in the embodiment can generate a plan by using capsules stored in the capsule database. For example, the plan 407 can be generated by using the actions 4011 and 4013 and concepts 4012 and 4014 of capsule A 401 and the action 4041 and concept 4042 of capsule B 404.

[0062] Figure 3 A screen showing a user terminal processing a received voice input through a smart application according to an embodiment.

[0063] The client 100 can run an intelligent application to communicate with the intelligent server (e.g. Figure 1 The intelligent server 200 in processes the user input.

[0064] According to an embodiment, in screen 310, when a specified voice input is recognized (e.g., wake up!) or input is received through a hardware key (e.g., a dedicated hardware key), the user terminal 100 may run an intelligent application for processing the voice input. For example, the user terminal 100 may run an intelligent application when running a scheduling application. According to an embodiment, the user terminal 100 may display a screen (e.g., Figure 1 An object (e.g., icon) 311 corresponding to the smart application is displayed on the display 140 in the user terminal 100. According to an embodiment, the user terminal 100 may receive a voice input by a user's speech. For example, the user terminal 100 may receive a voice input saying "Tell me about this week's schedule!" According to an embodiment, the user terminal 100 may display a user interface (UI) 313 (e.g., an input window) of the smart application on the display, in which the text data of the received voice input is displayed.

[0065] According to an embodiment, on screen 320, the user terminal 100 may display a result corresponding to the received voice input on the display 140. For example, the user terminal 100 may receive a plan corresponding to the received user input, and may display “this week’s schedule” on the display 140 according to the plan.

[0066] Figure 4 A configuration of an electronic device 500 according to an embodiment is shown.

[0067] In an embodiment, the electronic device 500 (eg, Figure 5 The electronic device 501 in the embodiment may include a microphone 450 (eg, Figure 5 , an input module 550 in the example, a speaker 460 (e.g., Figure 5 The sound output module 555 in the processor 430 (for example, Figure 5 processor 520) and memory 440 (e.g., Figure 5 Memory 530 in.

[0068] In an embodiment, the microphone 450 is an element for receiving sound from the outside and generating a signal corresponding to the input sound. Specifically, the microphone 450 can receive a user voice and generate a user voice signal corresponding to the user voice.

[0069] Furthermore, the microphone 450 may receive noise generated around the electronic device 500 and generate a signal corresponding to the input noise. The microphone 450 may include various elements, such as a microphone for collecting analog sound (user voice or environmental noise), an amplifier circuit for amplifying the collected user voice, an A / D conversion circuit for sampling the amplified user voice and converting the sampled user voice into a digital signal, and a filter circuit for removing noise components from the digital signal.

[0070] The element for receiving sound may be implemented as a microphone, but this is merely an example, and thus the element may be implemented as an input terminal capable of receiving a sound signal.

[0071] In an embodiment, the speaker 460 is an element that outputs various notification sounds or voice messages as well as various audio data on which the processor 430 has performed processing such as decoding. The speaker 460 can amplify the output of various notification sounds or voice messages.

[0072] In particular, the speaker 460 may output the response information about the user's voice in the form of a natural language voice message. Alternatively, when the electronic device 500 detects ambient noise and summarizes the response information about the user's voice, the speaker 460 may output the summarized response information in the form of a natural language voice message.

[0073] The element for outputting audio may be implemented as a speaker, but this is merely an embodiment, and thus the element may be implemented as an output terminal capable of outputting audio data.

[0074] In an embodiment, the processor 430 may be electrically connected to the memory 440 to control the overall operation and functions of the electronic device 500. For example, the processor 430 may drive an operating system or an application (e.g., Figure 5 540 in the processor 430) to control hardware or software elements connected to the processor 430 and perform various types of data processing and calculations. In addition, the processor 430 can load and process data from a volatile memory (e.g., Figure 5The volatile memory 532 in the memory may receive commands or data from at least one of the other elements in the memory, and various data blocks may be stored in the non-volatile memory (e.g., Figure 5 In the non-volatile memory 534).

[0075] To this end, the processor 430 can be implemented as a dedicated processor (e.g., an embedded processor) for performing corresponding operations or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor (AP)) capable of performing corresponding operations by running one or more software programs stored in a storage device.

[0076] In the present disclosure, the processor 430 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) for processing digital signals. However, the present disclosure is not limited thereto, and the processor 430 may include one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a graphics processing unit (GPU), a communication processor (CP), or an address resolution protocol (ARP) processor, or may be defined by corresponding terms. In addition, the processor 430 may be implemented as a system on chip (SoC) with a built-in processing algorithm or large-scale integration (LSI), or may be implemented as a field programmable gate array (FPGA).

[0077] In an embodiment, the processor 430 may detect a user voice spoken by a user through the microphone 450. Specifically, the processor 430 may receive a signal of a user voice input through the microphone 450 to detect the user voice.

[0078] In an embodiment, the processor 430 may perform speech recognition of the user's voice, and may control the speaker 460 to output information corresponding to the speech recognition result. In the present disclosure, the information corresponding to the speech recognition result may be information about a response to a query included in the user's voice. That is, when the user's voice includes a request for specific information, the information corresponding to the speech recognition result may be response information to the specific information included in the user's voice.

[0079] The processor 430 may detect noise around the electronic device 500 during a series of processes of receiving a user voice, performing voice recognition of the user voice, and outputting information corresponding to a voice recognition result.

[0080] Specifically, the processor 430 may recognize noise around the electronic device 500 by using the microphone 450. The processor 430 may receive a sound signal generated around the electronic device 500 through the microphone 450, and may recognize noise in the received sound signal.

[0081] In an embodiment, the processor 430 may identify the type of ambient noise by analyzing the frequency of the signal corresponding to the ambient noise. For example, the processor 430 may perform frequency analysis by extracting features such as the amplitude and period of the ambient noise signal from the received sound, thereby identifying the type of ambient noise. The processor 430 may identify whether the ambient noise corresponds to the voice of other users, to sounds generated by other electronic devices, or to sounds generated in situations requiring immediate action by the user, such as a baby crying or a window breaking.

[0082] Alternatively, the processor 430 can identify the type of environmental noise by using an artificial intelligence model. Specifically, the processor 430 can identify the type of noise around the electronic device 500 by using an artificial intelligence model trained to identify the type of sound. To this end, the electronic device 500 may include an artificial intelligence model trained to receive the sound input through the microphone 450 and identify the type of noise.

[0083] In an embodiment, the processor 430 may change the output state of the information corresponding to the speech recognition result based on the identified noise type. For example, the change of the output state of the information may include stopping the output of the information corresponding to the speech recognition result. The change of the output state of the information may include summarizing and outputting the information corresponding to the speech recognition result. The change of the output state of the information may include outputting the sound including the specific word louder than the sound including other words. The change of the output state of the information may include outputting the information about the noise instead of the information corresponding to the speech recognition result.

[0084] In an embodiment, the memory 440 is an element for storing various types of programs and data required for the operation of the electronic device 500 .

[0085] In an embodiment, the memory 440 may be implemented as an internal memory included in the processor 430, such as a ROM (e.g., an electrically erasable programmable read-only memory (EEPROM)) or a RAM, or may be implemented as a memory separate from the processor 430. In this case, the memory 440 may be implemented in the form of a memory embedded in the electronic device 500 or in the form of a removable memory in the electronic device 500, depending on the purpose of data storage. For example, data for driving the electronic device 500 may be stored in the memory embedded in the electronic device 500, and data for the extended functions of the electronic device 500 may be stored in the removable memory of the electronic device 500.

[0086] In an embodiment, the memory 440 may include an NLG abstraction module (not shown) that summarizes response information generated as a result of user voice recognition and generated by a natural language generator (NLG) module. The NLG abstraction module (not shown) may extract keywords from the response information generated by the NLG module of the dialogue system, and may generate summary information by summarizing the response information based on the extracted keywords.

[0087] To this end, keyword information of each domain may be pre-stored in the memory 440. In the present disclosure, a domain refers to a field related to a user's voice or response information to a user's voice. For example, when a user says "How is the weather today?", the domain for the voice spoken by the user may be "weather".

[0088] In an embodiment, the memory 440 may store an artificial intelligence model that is trained to output information about the user's reaction according to the noise type. In this case, the artificial intelligence model may be retrained using the type of noise generated by driving the electronic device 500 and / or information about the user's reaction to the noise.

[0089] Alternatively, according to an embodiment, the noise type and / or information about the user's reaction according to the noise type may be stored. Specifically, information about the user's reaction to the output of the speech recognition result when the noise is generated may be stored in the memory 440 according to the noise type. For example, if in the process of providing the speech recognition result, when a noise from an external device such as a vacuum cleaner sound occurs, the user does not summarize the speech recognition result but amplifies the sound of the speech information, then the memory 440 may store user reaction information such as "speech recognition result output step-vacuum cleaner sound-sound amplification".

[0090] In an embodiment, natural language understanding (NLU) can be implemented with various types of algorithms including rule-based or statistical-based algorithms. If it is necessary to process commands that are not applied to the rules or models of the algorithm used or commands that need to be processed separately, the processor 430 can identify and preprocess commands separately before processing natural language understanding. For example, a user-specified instruction (e.g., a shortcut command) is identified by a specified speech processor (not shown) through sentence matching (e.g., exact matching, Levenshtein distance (Levenshtein distance) or cosine distance), and when the user-specified instruction is identified, the processor 430 can sequentially run the commands configured by the user. The user-specified instruction (shortcut command) is a command generated with a user-specified name, and can refer to a command that cannot be processed in natural language understanding. Even if it is desired to temporarily specify a recognizable command in natural language understanding to perform other functions, the processor 430 can also sequentially run the command set by the user. In another example, in the case where a common command is to be processed in all domains, the processor 430 can use a specified speech processor (not shown) to process the common command, rather than identifying the common command in all domains and specifying an action. In the present disclosure, a domain refers to a field related to a user voice or response information to a user voice.

[0091] In an embodiment, according to a specific implementation, a designated speech processor (not shown) may be processed as a domain. The designated speech processing unit may be implemented on the processor 430. When the designated speech processor is processed as a domain, the domain may be selected with a higher priority than other domains. The designated speech processor performs domain classification on an input command, and when processed as a domain, the intent corresponding to the domain may be preferentially selected.

[0092] In an embodiment, the processor 430 may execute a function corresponding to a user request and then convert a message to be delivered to the user into sound. The processor 430 may use a text-to-speech (TTS) function. The processor 430 may generate speech based on a TTS model. The TTS model may be determined differently depending on the speech synthesis algorithm used (e.g., parameter speech synthesis, unit concatenation speech synthesis, or neural network speech synthesis).

[0093] In an embodiment, the processor 430 may include a plurality of TTS models corresponding to various voices. The processor 430 may include an emotional TTS model for representing various emotions or an emphasized TTS model for emphasizing voicing. When a reconfirmation request utterance is recognized, the processor 430 may generate a voice response by using a response to a previous utterance. The processor 430 may generate a voice response using other TTS models not used in the previous response. Alternatively, the processor 430 may generate a voice response using other synthesis parameters not used in the previous response. For example, the processor 430 may generate a waveform type synthesis in a vocoder by using an acoustic frame generated in a decoder. A vocoder is a composite word of voice and coder, and may refer to a technique for compressing a pulse code modulation (PCM) signal, which is a digitally converted voice signal. The processor 430 may adjust at least one of a plurality of features (e.g., pitch, spectrum, cepstrum, duration, and energy) for a portion that needs to be emphasized during the operation of the vocoder to apply a sense of emphasis to the synthesized sound. The cepstrum can be obtained by applying an inverse fast Fourier transform to the spectrum.

[0094] Figure 5 is a block diagram illustrating an electronic device 501 in a network environment 500 according to various embodiments. Figure 5 , the electronic device 501 in the network environment 500 may communicate with the electronic device 502 via the first network 598 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 504 or the server 508 via the second network 599 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 501 may communicate with the electronic device 504 via the server 508. According to an embodiment, the electronic device 501 may include a processor 520, a memory 530, an input module 550, a sound output module 555, a display module 560, an audio module 570, a sensor module 576, an interface 577, a connection terminal 578, a haptic module 579, a camera module 580, a power management module 588, a battery 589, a communication module 590, a user identification module (SIM) 596, or an antenna module 597. In some embodiments, at least one of the above components (e.g., the connection terminal 578) may be omitted from the electronic device 501, or one or more other components may be added to the electronic device 501. In some embodiments, some of the above-described components (eg, sensor module 576, camera module 580, or antenna module 597) may be implemented as a single integrated component (eg, display module 560).

[0095] The processor 520 may run, for example, software (e.g., program 540) to control at least one other component (e.g., hardware component or software component) of the electronic device 501 connected to the processor 520, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 520 may store commands or data received from another component (e.g., sensor module 576 or communication module 590) in the volatile memory 532, process the commands or data stored in the volatile memory 532, and store the resultant data in the non-volatile memory 534. According to an embodiment, the processor 520 may include a main processor 521 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 523 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 521 in operation. For example, when the electronic device 501 includes a main processor 521 and an auxiliary processor 523, the auxiliary processor 523 may be adapted to consume less power than the main processor 521, or adapted to be dedicated to a specific function. The auxiliary processor 523 may be implemented to be separate from the main processor 521, or to be implemented as part of the main processor 521.

[0096] When the main processor 521 is in an inactive (e.g., sleep) state, the auxiliary processor 523 (rather than the main processor 521) may control at least some of the functions or states related to at least one component among the components of the electronic device 501 (e.g., the display module 560, the sensor module 576, or the communication module 590), or when the main processor 521 is in an active state (e.g., running an application), the auxiliary processor 523 may control at least some of the functions or states related to at least one component among the components of the electronic device 501 (e.g., the display module 560, the sensor module 576, or the communication module 590) together with the main processor 521. According to an embodiment, the auxiliary processor 523 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., a camera module 580 or a communication module 590) that is functionally related to the auxiliary processor 523. According to an embodiment, the auxiliary processor 523 (e.g., a neural processing unit) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model may be generated by machine learning. For example, such learning may be performed by the electronic device 501 where the artificial intelligence is executed or via a separate server (e.g., server 508). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, for example. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q network or a combination of two or more thereof, but is not limited thereto. Additionally or alternatively, the artificial intelligence model may include a software structure in addition to a hardware structure.

[0097] The memory 530 may store various data used by at least one component of the electronic device 501 (e.g., the processor 520 or the sensor module 576). The various data may include, for example, software (e.g., the program 540) and input data or output data for commands related thereto. The memory 530 may include a volatile memory 532 or a non-volatile memory 534.

[0098] The program 540 may be stored as software in the memory 530 , and may include, for example, an operating system (OS) 542 , middleware 544 , or an application 546 .

[0099] The input module 550 may receive commands or data to be used by other components (e.g., the processor 520) of the electronic device 501 from outside (e.g., a user) of the electronic device 501. The input module 550 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0100] The sound output module 555 can output a sound signal to the outside of the electronic device 501. The sound output module 555 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive an incoming call. Depending on the embodiment, the receiver can be implemented as a separate speaker or as part of the speaker.

[0101] The display module 560 may visually provide information to the outside (e.g., user) of the electronic device 501. The display device 560 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display module 560 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of a force caused by a touch.

[0102] The audio module 570 can convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 570 can obtain sound via the input module 550, or output sound via the sound output module 555 or an earphone of an external electronic device (e.g., electronic device 502) directly (e.g., wired) or wirelessly connected to the electronic device 501.

[0103] The sensor module 576 may detect an operating state (e.g., power or temperature) of the electronic device 501 or an environmental state (e.g., a state of a user) outside the electronic device 501, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 576 may include, for example, a gesture sensor, a gyroscope sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0104] The interface 577 may support one or more specific protocols to be used to connect the electronic device 501 directly (e.g., wired) or wirelessly to an external electronic device (e.g., the electronic device 502). According to an embodiment, the interface 577 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0105] The connection end 578 may include a connector, wherein the electronic device 501 can be physically connected to an external electronic device (e.g., the electronic device 502) via the connector. According to an embodiment, the connection end 578 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0106] The haptic module 579 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 579 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0107] The camera module 580 may capture a still image or a moving image. According to an embodiment, the camera module 580 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0108] The power management module 588 may manage power supply to the electronic device 501. According to an embodiment, the power management module 588 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0109] The battery 589 may power at least one component of the electronic device 501. According to an embodiment, the battery 589 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0110] The communication module 590 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 501 and an external electronic device (e.g., electronic device 502, electronic device 504, or server 508), and perform communication via the established communication channel. The communication module 590 may include one or more communication processors capable of operating independently from the processor 520 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 590 may include a wireless communication module 592 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 594 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 598 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 599 (e.g., a long-range communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip), or these various types of communication modules can be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 592 can identify and authenticate the electronic device 501 in a communication network (such as the first network 598 or the second network 599) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 596.

[0111] The wireless communication module 592 can support 5G networks after 4G networks and next-generation communication technologies (e.g., new radio (NR) access technology). NR access technology can support enhanced mobile broadband (eMBB), massive machine type communication (mMTC), or ultra-reliable low-latency communication (URLLC). The wireless communication module 592 can support high-frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 592 can support various technologies for ensuring performance on high-frequency bands, such as, for example, beamforming, massive multiple-input multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 592 can support various requirements specified in the electronic device 501, an external electronic device (e.g., electronic device 504), or a network system (e.g., a second network 599). According to an embodiment, the wireless communication module 592 may support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane delay for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less round trip).

[0112] The antenna module 597 may transmit a signal or power to the outside of the electronic device 501 (e.g., an external electronic device) or receive a signal or power from the outside of the electronic device 501 (e.g., an external electronic device). According to an embodiment, the antenna module 597 may include an antenna including a radiating element, and the radiating element is formed of a conductive material or a conductive pattern formed in a substrate (e.g., a printed circuit board (PCB)) or formed on a substrate. According to an embodiment, the antenna module 597 may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 598 or the second network 599) may be selected from the plurality of antennas by, for example, the communication module 590 (e.g., the wireless communication module 592). Subsequently, a signal or power may be transmitted or received between the communication module 590 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element may be additionally formed as a part of the antenna module 597.

[0113] According to various embodiments, the antenna module 597 may form a millimeter wave antenna module. According to an embodiment, the millimeter wave antenna module may include a printed circuit board, a radio frequency integrated circuit (RFIC), and a plurality of antennas (e.g., array antennas), wherein the RFIC is disposed on a first surface (e.g., bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a specified high frequency band (e.g., millimeter wave band), and the plurality of antennas are disposed on a second surface (e.g., top surface or side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the specified high frequency band.

[0114] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0115] According to an embodiment, a command or data may be sent or received between the electronic device 501 and the external electronic device 504 via the server 508 connected to the second network 599. Each of the electronic device 502 or the electronic device 504 may be a device of the same type as the electronic device 501, or a device of a different type from the electronic device 501. According to an embodiment, all or some operations to be executed in the electronic device 501 may be executed in one or more of the external electronic device 502, the external electronic device 504, or the server 508. For example, if the electronic device 501 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 501 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 501 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices receiving the request may execute at least part of the function or service requested, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 501. The electronic device 501 may provide the result as at least a partial reply to the request in the case of further processing the result or in the case of not further processing the result. To this end, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology or client-server computing technology, for example, may be used. The electronic device 501 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 504 may include an Internet of Things (IoT) device. The server 508 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 504 or the server 508 may be included in the second network 599. The electronic device 501 may be applied to intelligent services (e.g., smart homes, smart cities, smart cars or health care) based on 5G communication technology or IoT-related technologies.

[0116] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.

[0117] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or alternative forms for corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one or all possible combinations of the items listed together with the corresponding one of the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish the corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “combined with another element (e.g., the second element)”, “combined to another element (e.g., the second element)”, “connected with another element (e.g., the second element)”, or “connected to another element (e.g., the second element)” when the term “operably” or “communicatively” is used or when the term “operably” or “communicatively” is not used, it means that the element may be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0118] As used in connection with various embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0119] The various embodiments described herein may be implemented as software (e.g., program 540) including one or more instructions stored in a storage medium (e.g., internal memory 536 or external memory 538) that can be read by a machine (e.g., electronic device 501). For example, under the control of a processor, a processor (e.g., processor 520) of the machine (e.g., electronic device 501) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be run by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" only means that the storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), but the term does not distinguish between data being semi-permanently stored in a storage medium and data being temporarily stored in a storage medium.

[0120] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be published in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be published online (e.g., downloaded or uploaded) via an application store (e.g., Play StoreTM), or may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a manufacturer's server, an application store's server, or a memory of a forwarding server).

[0121] According to various embodiments, each component (e.g., module or program) in the above-mentioned components may include a single entity or multiple entities, and some entities in the multiple entities may be separately arranged in different components. According to various embodiments, one or more components in the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each component in the multiple components in the same or similar manner as the corresponding one component in the multiple components before integration. According to various embodiments, the operations performed by a module, program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more operations in the operations may be run or omitted in different orders, or one or more other operations may be added.

[0122] Figure 6a A configuration of a specified discourse processor for natural language processing according to an embodiment is shown.

[0123] In an embodiment, an automatic speech recognizer (automatic speech recognition (ASR)) 605 may recognize speech, convert the speech to text, and send the text to a natural language understanding controller 610 .

[0124] Natural language understanding can be implemented using various types of algorithms including rule-based or statistical-based algorithms. A designated speech processor (predetermined command recognition) 612 within the natural language understanding controller 610 can independently identify and preprocess commands that are not applied to the algorithm model used or require separate processing before natural language understanding. For example, the designated speech processor 612 can recognize user-specified instructions (e.g., shortcut commands) generated by sentence matching (e.g., exact match, Levenshtein distance, or cosine distance) with user-specified names, and can execute user-configured commands sequentially.

[0125] In addition, for instructions that are desired to be temporarily designated to execute a function different from the designated function, the instructions for executing the different functions can be temporarily designated by a designated speech processor 612, rather than being recognized and designated in all domains like a common command. The designated speech processor 612 can use one module to process instructions to be processed on each domain.

[0126] The natural language understanding controller 610 may include a designated utterance processor 612, a domain classifier 615, a domain-specific intent classifier 617, and a domain-specific action runner 619. The natural language understanding controller 610 may determine the utterance intent of each domain by using the domain classifier 615, the domain-specific intent classifier 617, and the domain-specific action runner. The utterance intent may be a user intent (e.g., alarm setting or alarm dismissal) determined by a speech recognition action.

[0127] The designated speech processor 612 may be processed as a domain. In this case, one domain may have a higher priority than other domains. For example, as a result of domain classification of a user input command, when a first domain and a separate second domain corresponding to the designated speech processor 612 are detected, the processor (e.g., Figure 4 The processor 430 in the speech processor 612 may preferentially select the intent corresponding to the first domain rather than the intent corresponding to the second domain. The specified speech processor 612 may preferentially identify the intent corresponding to the first domain and send the identified intent to the speech response generator 620.

[0128] The natural language understanding controller 610 may recognize a reconfirmation request utterance (e.g., “What?”, “Tell me again”, or “What is the maximum temperature?”) in the user voice input. The natural language understanding controller 610 may recognize the reconfirmation request utterance and send the reconfirmation request utterance to the voice response generator 620.

[0129] According to an embodiment, when a reconfirmation request utterance is input within a specified time (e.g., 3 minutes) after an input time point of a previous utterance or a response time point to a previous utterance, the voice response generator 620 determines that there is a reconfirmation intention. When it is not determined that there is a reconfirmation intention, the voice response generator 620 may generate a specified response (e.g., "It is difficult to answer", "This is an incomprehensible command"). According to an embodiment, the specified utterance processor 612 may perform control so that a reconfirmation request utterance may be recognized only within a specified time (e.g., 30 seconds) after an input time point of a previous utterance or a response time point to a previous utterance. The time specified for the response is merely an example, but is not limited thereto.

[0130] According to an embodiment, when the user voice is not recognized as at least one of the designated commands, the designated speech processor 612 may analyze the input user voice by using the domain classifier 615 of the natural language understanding unit or the domain-specific intent classifier 617. The designated speech processor 612 may determine the intent of the user voice based on the analysis result, and may perform an action corresponding to the determined intent.

[0131] The voice response generator 620 may generate a corresponding response based on the reconfirmation request utterance received from the natural language understanding controller 610. The voice response generator 620 may use a text-to-speech (TTS) algorithm to convert a message to be sent to the user into sound. Figure 6b Describe the text-to-speech (TTS) algorithm. The speech response generator 620 can generate speech based on the TTS model, and the TTS model can vary according to the text-to-speech algorithm (e.g., parameter speech synthesis, unit concatenation speech synthesis, and neural network speech synthesis). The speech response generator 620 may include multiple TTS models 621, 622, and 623 corresponding to various voices or sounds.

[0132] In an embodiment, the voice response generator 620 may include an emotional TTS model for representing various emotions and / or an emphasis TTS model for generating emphasized utterances. The voice response generator 620 may recognize the reconfirmation request utterance received from the natural language understanding controller 610, and may generate a voice response by using a response to a previous utterance. Alternatively, the voice response generator 620 may generate a voice response by using a synthesis parameter different from that of the response to the previous utterance. Figure 6b Describes the process of generating responses by using individual synthesis parameters.

[0133] Figure 6b is a block diagram illustrating a text-to-speech (TTS) algorithm according to an embodiment.

[0134] In an embodiment, the encoder 630 can generate multiple potential vectors or potential frames based on the input text. The duration encoder 640 can predict the duration of each phoneme based on multiple potential vectors or potential frames. Alternatively, the duration encoder 640 can regenerate a potential vector or potential frame so that each phoneme has a predetermined level of duration. A potential vector can refer to a vector type variable possessed by an image or data. A potential vector or potential frame can be used to determine the potential space of the observed data. The potential space can be relatively small compared to the actual observed space. The deep learning learning model can improve efficiency by learning only specific features in the potential space when learning the observed data. The decoder 650 can generate the audio frame required for synthesis by using at least one of the generated potential vectors or potential frames. The decoder 650 can insert an emotional marker in an embedded form and generate a synthetic speech according to the emotion. The vocoder 660 can generate a synthetic sound in waveform form by using the acoustic frame generated by the decoder 650. The vocoder 660 may determine a portion that needs to be emphasized in the synthesized sound, and may apply a sense of emphasis to the synthesized sound by using at least one of synthesis features (eg, pitch, spectrum, cepstrum, duration, or energy) in the determined portion.

[0135] Figure 7 is a flow chart illustrating a method for providing voice feedback by an electronic device according to an embodiment.

[0136] It can be based on the information stored in the computer recording medium or memory (for example, Figure 5 The memory 530 in the instruction is implemented by Figure 7 The method 700 shown can be performed by the above reference Figure 1 6 (eg, Figure 5 The electronic device 501 in the embodiment is executed, and the technical features described above will be omitted below. Figure 7 The order of each operation in can be controlled, some operations can be omitted, and some operations can be performed in parallel.

[0137] In operation 710, when a microphone (eg, Figure 4 When the microphone 450 in the apparatus detects the user's speech, the processor (e.g., Figure 4 The processor 430 in the embodiment may perform voice recognition on the voice spoken by the user to output a first response message corresponding to the result of the voice recognition.

[0138] In operation 720, the processor 430 may detect a request to re-output the first response message in the speech recognition result. Alternatively, the designated speech processor (predetermined command recognition) may detect a request to re-output the first response message in the speech recognition result. The first response message may refer to an output message provided in response to a user request before receiving the re-output request. In an embodiment, the speech recognition operation may be performed by a natural language understanding controller (e.g., Figure 6a A domain classifier (e.g., Figure 6a 615) and domain-specific intent classifiers (e.g., Figure 6a The domain-specific intent determiner 617 in the domain-specific intent determiner 617 is executed. The domain-specific action runner (e.g., Figure 6a The domain-specific action runner 619 in the embodiment may generate a response text by using the user intent (e.g., alarm setting or alarm cancellation) or parameters (e.g., alarm time, number of repetitions, or alarm sound) determined by the speech recognition operation. A first response message may be generated by performing text-to-speech (TTS) on the generated response text based on synthesis parameters (e.g., pitch, spectrum, cepstrum, duration, or energy). In an embodiment, the processor 430 may receive user voice feedback such as “what?” or “say it again”. The processor 430 may specifically receive voice feedback specifying a specific item, such as “What is OO?” or “say OO again”. The processor 430 may provide a variety of response methods by using an AI agent (or intelligent agent) to respond to user voice feedback.

[0139] For example, the processor 430 can regenerate the first response message or the previous voice feedback again. In an embodiment, the processor 430 can generate a second response message by performing text-to-speech (TTS) on the response text generated for the speech previously spoken by the user based on a second synthesis parameter different from the first synthesis parameter. The processor 430 can adjust the volume of the response message according to the environment, and then provide the response message to the user. The processor 430 can emphasize and output a part of the voice feedback corresponding to a specific tag. The processor 430 can determine the emotional expression corresponding to at least one word in the first response message, and can generate a second response message based on the at least one word and the emotional expression. The second response message may refer to a response message generated in response to the user re-output request. The second response message may include at least a portion of the first response message. Alternatively, the second response message may have the same content as the first response message, differing only in volume.

[0140] In operation 730, the processor 430 may determine at least one word to be included in the second response message based on at least one parameter corresponding to at least one word constituting the first response message. In an embodiment, the processor 430 may emphasize and express important information in the entire content in response to the "re-request" for voice feedback. The processor 430 may determine at least one word by using a specified parameter in the first response message. The processor 430 may emphasize the determined word based on a tag near the specified parameter and may output the word through a speaker (e.g., Figure 2 The processor 430 may output the emphasized word through the speaker 460 in the voice feedback. For example, the tag may be operably linked to the variable phrase portion. The processor 430 may send the content of the message to the user more clearly by emphasizing a portion of the voice feedback content.

[0141] According to an embodiment, the domain-specific action runner 619 may select the following phrase as a dialog template for “Tell me about today's weather”.

[0142] {speech("Expected today for #{displayLocation} is #{conditionand} with a minimum temperature of #{MinTemp} and a maximum temperature of #{MaxTemp}. Fine particulate matter is #{fineDustLevel}. #{description}.")}

[0143] In an embodiment, the dialog template may include a fixed phrase part and a variable phrase part. The fixed phrase part may mean a part that does not change according to the request situation, such as "today", "lowest temperature", "highest temperature" or "expected". The variable phrase part may mean a part that can change according to the surrounding situation, such as "#{displayLocation}", "#{MinTemp}" or "#{MaxTemp}". The action runner 619 may be based on the electronic device (e.g., Figure 5 The action runner 619 may determine the value corresponding to the variable phrase portion based on the information collected by the electronic device 501 in the user's mobile phone and the external server or internal database. For example, the action runner 619 may determine that #{displayLocation} is "Seoul", #{MinTemp} is "24 degrees" and #{MaxTemp} is "29 degrees". The action runner may generate a response text (e.g., "It is expected that Seoul will be cloudy today, with a minimum temperature of 24 degrees and a maximum temperature of 29 degrees.") by using the determined variable phrase portion.

[0144] In an embodiment, the domain specific action runner 619 may use the dialog template for the previous utterance to generate a second response message corresponding to the user's re-request. The domain specific action runner 619 may synthesize speech by applying emphasis synthesis parameters to variable phrases of the dialog template.

[0145] In another embodiment, the domain-specific action runner 619 may not apply the emphasis synthesis parameter to some of the plurality of variable phrases. For example, when some of the plurality of variable phrase parts match words or phrases included in the input utterance, the domain-specific action runner 619 may determine that some of the variable phrase parts are information known to the user, and may not apply the emphasis synthesis parameter thereto.

[0146] The processor 430 can determine the words or phrases to be emphasized in the received conversation by using parameters. The conversation can refer to the conversation (or voice) history between the user and the intelligent agent. The parameter can refer to the parameter in the programming. The parameter can refer to the words generated and output based on specific information (e.g., time, place, maximum temperature or minimum temperature). For example, when the user voice expression is "What is the maximum temperature?", the parameter can be "maximum temperature". For example, when the user voice expression is "What time is it now?", the parameter can be "time". This is just an example, and the type of parameter is not limited to this.

[0147] For example, when a user requests weather information, the processor 430 may construct a response message using at least one of date, minimum temperature, maximum temperature, fine particle concentration, or precipitation. When a user requests weather information, the processor 430 may determine to emphasize an element corresponding to the parameter based on at least one parameter of date, minimum temperature, maximum temperature, fine particle concentration, or precipitation. Alternatively, the processor 430 may determine to include only the element corresponding to the parameter in the response message based on the parameter.

[0148] In operation 740, the processor 430 may generate a second response message including an emphasis tone for at least one determined word. In an embodiment, the processor 430 may include at least one determined word in the response message. In an embodiment, the processor 430 may generate a response message that emphasizes the at least one determined word. The second response message may include an emphasis tone for at least one word determined in the first response message. The second response message may refer to a response message generated in response to a user re-output request. The processor 430 may determine whether to emphasize a certain word by using a tag. A tag may refer to a keyword or classification given as metadata for a certain information. A tag may be located before or after a parameter in a dialog template. In an embodiment, the processor 430 may identify a tag near a parameter, and may emphasize a word corresponding to the parameter based on the tag and output the word.

[0149] In an embodiment, the processor 430 may determine at least one word in the first response message by using a specified parameter. The processor 430 may determine an emotional expression corresponding to the at least one determined word. The processor 430 may generate a second response message based on the at least one determined word and the determined emotional expression. The processor 430 may determine the tone or speech speed of the second response message based on the emotional expression. When providing voice feedback, the processor 430 may more realistically express the content to be conveyed by the emotional expression. Figure 8b Describes operations for emotional expression.

[0150] Figure 8a and Figure 8b is a flow chart illustrating a method for providing voice feedback by an electronic device according to an embodiment.

[0151] It can be based on the information stored in the computer recording medium or memory (for example, Figure 5 The memory 530) in the instruction is implemented by Figure 8a and Figure 8b The methods 800a and 800b shown can be referred to above. Figure 1 6 (eg, Figure 5The electronic device 501 in the embodiment is executed, and the technical features described above will be omitted below. Figure 8a and Figure 8b The order of each operation in can be controlled, some operations can be omitted, and some operations can be performed in parallel.

[0152] In an embodiment, a processor (e.g., Figure 4 The processor 430 in the embodiment may be able to use a microphone (e.g., Figure 4 The microphone 450 in the electronic device 500 can identify the noise around the electronic device 500, and the output state of the information corresponding to the voice recognition result can be changed based on the type of the identified noise.

[0153] In operation 810a, the processor 430 may recognize a user's re-request for voice feedback.

[0154] In operation 812a, processor 430 can analyze surrounding conditions or predetermined conditions. Most of the cases where the user "re-requests" voice feedback may be the case where the voice feedback is not properly transmitted due to external factors (e.g., far away from the user or noise). When processor 430 provides voice feedback with substantially the same volume as before, processor 430 may receive a "re-request" for voice feedback from the user. Processor 430 can provide effective voice feedback by "temporarily" or partially automatically adjusting the volume level according to surrounding conditions or predetermined conditions. In an embodiment, processor 430 can analyze surrounding conditions by using any one of the ambient noise level, the volume of the user's voice, or the information about the distance from the user measured by a sensor.

[0155] In operation 814a, the processor 430 may determine the volume level of the second response message based on the analysis result. The second response message may refer to a response message generated in response to a user re-output request. In an embodiment, the processor 430 may determine the volume level of the second response message based on the analyzed situation. The volume level of the second response message may be predefined or may be determined differently according to the analyzed situation. For example, when the ambient noise exceeds a predetermined level, the processor 430 may make the volume of the second response message relatively large compared to when the ambient noise is low.

[0156] In operation 816a, the processor 430 may provide a volume level that has been temporarily or partially adjusted to the second response message. The processor 430 may provide a response message with a volume that is temporarily adjusted differently based on the user's re-request for voice feedback. The processor 430 may analyze the user's current situation based on information about the ambient noise level, the user's voice volume, and / or the distance from the user. The processor 430 may temporarily or partially adjust the volume level based on the analyzed user's current situation. For example, the processor 430 may output a response message with a volume that is temporarily increased based on the ambient noise exceeding a specified level. Alternatively, in the case where loud sounds are not allowed (e.g., in a library), the processor 430 may output a response message with a volume that is temporarily reduced based on the user's voice volume being below a specified level. According to an embodiment, the volume of the user's voice based on the re-request message (e.g., "What?") exceeds a specified level, and the processor 430 may output a response message with a temporarily increased volume.

[0157] In an embodiment, the processor 430 may determine an emotional expression corresponding to at least one word determined based on the parameter in the first response message, and may output a second response message based on the determined at least one word and the determined emotional expression. The second response message may refer to a response message generated in response to a user re-output request.

[0158] In an embodiment, when providing voice feedback, the processor 430 can express the content to be conveyed by emotional expression more realistically. Emotional expression can change according to the sentence. When several sentences are continuous, the processor 430 can distinguish the sentences from the entire voice feedback paragraph, can analyze the emotion of each sentence, and can apply emotional effects to each sentence accordingly. The processor 430 can recognize specific words in the sentence, and can recognize the emotion of the speaker who said the sentence. For example, the processor 430 can recognize the word "tranquility" in the sentence "It is a quiet morning", and in response to this, the speed of the response message can be configured to a specified level or the tone of the response message can be adjusted, so as to apply the emotional effect corresponding to "tranquility".

[0159] In operation 820b, the processor 430 may separate and identify sentences from the identified paragraphs.

[0160] In operation 822b, the processor 430 may analyze the sentiment of each individual sentence.

[0161] In operation 824b, the processor 430 may apply the emotional effect analyzed for each sentence to the response message. In an embodiment, when the sentences have the same emotion, the processor 430 may group and process the continuous sentences. When the user "re-requests" the voice feedback, the processor 430 may provide realistic emotional expressions according to the content of the voice feedback. The processor 430 may provide various experiences to the user by providing realistic emotional expressions. The realistic emotions may include, for example, at least one of being angry, bored, satisfied, happy or happy. These are merely examples, and the types of realistic emotions are not limited thereto.

[0162] In an embodiment, when the user requests voice feedback more than a predetermined number of times, the processor 430 may provide emotional changes in voice feedback. The processor 430 may generate different types of response voices based on the number of re-request commands. For example, when the number of re-request commands is less than a certain level, the processor 430 may generate a response message with a happy or excited tone. Based on the number of re-request commands exceeding a predetermined level, the processor 430 may generate a response message with a fatigue or bored tone instead of a happy or excited tone. The number of re-request commands and / or the tone of the response message are not limited thereto and may vary according to the configuration. The processor 430 may provide emotional changes in voice feedback, thereby adding interest to the conversation.

[0163] According to an embodiment, an electronic device (e.g., Figure 4 The electronic device 500 in the embodiment may include a microphone (eg, Figure 4 microphone 450 in the example), a speaker (e.g., Figure 4 Speaker 460 in), memory (e.g., Figure 4 440) and a processor (e.g., Figure 4 The processor 430 in the embodiment of the present invention is configured to perform speech recognition of the user speech voice when the user speech voice is detected through the microphone, and control the speaker to output a first response message corresponding to the result of the speech recognition. The processor 430 may be configured to detect a speech expression indicating a re-request for the first response message in the speech recognition result, identify at least one segment of the text constituting the first response message, determine at least one speech signal to be included in the second response message based on at least one parameter corresponding to the text, and generate a second response message emphasizing the determined at least one speech signal.

[0164] According to an embodiment, a second response message can be generated based on a dialogue template, and the dialogue template may include a first area and a second area, the first area including predetermined phrases or words having fixed content, the second area including phrases or words having content that can be changed based on information received from an application, at least one phrase or word in the second area corresponds to at least one parameter, respectively, and the parameters may be generated and output based on time, place, maximum temperature or minimum temperature, and the parameters may include maximum temperature, minimum temperature, alarm time, alarm repetition number and alarm sound.

[0165] According to an embodiment, the processor may determine the content of at least one phrase or word in the second area based on information received from the application.

[0166] According to an embodiment, based on at least one parameter included in the voice recognition result, the processor may determine a phrase or word to be included in the second response message from at least one phrase or word in the second area.

[0167] According to an embodiment, the processor can analyze the voice spoken by the user to identify at least one voice signal included in the voice spoken by the user, can determine at least one parameter corresponding to the at least one identified voice signal, can determine the voice signal to be included in the second response message from the voice signal in the first response message based on the at least one determined parameter, and can generate a second response message that provides an emphasized tone to the at least one determined voice signal.

[0168] According to an embodiment, the processor may determine an emotional expression corresponding to at least one determined voice signal by using a specified parameter in the first response message, and may generate a second response message based on the at least one determined voice signal and the determined emotional expression.

[0169] According to an embodiment, the processor may determine the tone or output speed of the second response message based on the number of requests to re-output the first response message.

[0170] According to an embodiment, based on the fact that the number of re-output requests of the first response message is less than a predetermined level, the processor may determine the tone of the second response message to be a relatively high tone, and based on the fact that the number of re-output requests of the first response message exceeds a predetermined level, the processor may determine the tone of the second response message to be a relatively low tone.

[0171] According to an embodiment, based on the fact that the number of requests for re-outputting the first response message is less than a predetermined level, the processor may determine the output speed of the second response message to be relatively high, and based on the fact that the number of requests for re-outputting the first response message exceeds a predetermined level, the processor may determine the output speed of the second response message to be relatively low.

[0172] According to an embodiment, the processor may recognize noise around the electronic device by using a microphone, and may change an output state of information corresponding to a voice recognition result based on a volume or type of the recognized noise.

[0173] According to an embodiment, the processor may control at least one voice signal that does not correspond to the parameter specified in the first response message not to be included in the second response message.

[0174] According to an embodiment, a method for providing voice feedback by an electronic device may include: when a voice spoken by a user is detected through a microphone, performing voice recognition of the voice spoken by the user, and outputting a first response message corresponding to the result of the voice recognition, detecting an utterance in the voice recognition result indicating a re-request for the first response message, identifying at least one fragment of text constituting the first response message, and determining at least one voice signal to be included in a second response message based on at least one parameter corresponding to the text, and generating a second response message emphasizing at least one determined voice signal.

[0175] The embodiments disclosed in the specification and the drawings are only specific examples provided according to the embodiments of the present disclosure to easily describe technical problems and help understand the embodiments of the present disclosure, and do not limit the scope of the embodiments of the present disclosure. Therefore, it should be understood that, in addition to the embodiments disclosed herein, all modifications or variant forms that can be derived from the technical ideas of the embodiments of the present disclosure are included in the scope of the embodiments of the present disclosure.

Claims

1. An electronic device, comprising: microphone; speaker; processor; A memory configured to store parameter information and instructions, which when executed by the processor cause the processor to: performing speech recognition on a user's speech received by the microphone, Based on the voice recognition of the user's voice, controlling the speaker to output a first response message, detecting a re-request for the first response message, identifying text of the first response message, determining a second response message and a first voice signal based on a first parameter corresponding to the text, and The second response message is generated including the determined first voice signal.

2. The electronic device according to claim 1, wherein: The instructions, when executed by the processor, further cause the processor to: generating the second response message based on the dialogue template, and Wherein, the dialogue template includes: a first region including: (i) predetermined phrases or predetermined words and (ii) fixed content, and a second area including: a phrase or word having content that can be changed based on information received from the application, The phrase or word in the second region corresponds to the second parameter. wherein the processor is further configured to generate the second parameter based on at least one of time, location, maximum temperature or minimum temperature, and The generated second parameter includes at least one of a maximum temperature, a minimum temperature, an alarm time, an alarm repetition number or an alarm sound.

3. The electronic device of claim 2, wherein the instructions, when executed by the processor, further cause the processor to: The phrases or words of the second region are determined based on information received from the application.

4. The electronic device of claim 2, wherein the instructions, when executed by the processor, further cause the processor to: determining a phrase or word of the second region based on the second parameter, and The determined phrase or the determined word is inserted into the second response message.

5. The electronic device of claim 1 , wherein the instructions, when executed by the processor, further cause the processor to: Recognize a second voice signal of the user's voice by analyzing the user's voice; determining a third parameter corresponding to the recognized second speech signal; determining a third voice signal from the voice signal of the first response message based on the determined third parameter; as well as The second response message includes the determined third voice signal and provides an emphasis tone to the determined third voice signal.

6. The electronic device of claim 1 , wherein the instructions, when executed by the processor, further cause the processor to: Determine the emotional expression corresponding to the determined first voice signal by using the fourth parameter in the first response message; and The second response message is generated based on the determined first voice signal and the determined emotional expression.

7. The electronic device of claim 1, wherein the instructions, when executed by the processor, further cause the processor to: A tone or output speed of the second response message is determined based on the number of requests for repeatedly outputting the first response message.

8. The electronic device of claim 7, wherein the instructions, when executed by the processor, further cause the processor to: determining the tone of the second response message to be a relatively high tone based on determining that the number of requests for repeatedly outputting the first response message is equal to or less than a predetermined level; and Based on determining that the number of requests for repeatedly outputting the first response message exceeds the predetermined level, the tone of the second response message is determined to be a relatively low tone.

9. The electronic device of claim 7, wherein the instructions, when executed by the processor, further cause the processor to: determining an output speed of the second response message to be relatively high based on determining that the number of requests for repeatedly outputting the first response message is equal to or less than a predetermined level; and Based on determining that the number of requests for repeatedly outputting the first response message exceeds the predetermined level, the output speed of the second response message is determined to be relatively low.

10. The electronic device of claim 1, wherein the instructions, when executed by the processor, further cause the processor to: by recognizing noise around the electronic device using the microphone; and An output state of information corresponding to the speech recognition is changed based on the volume or type of the recognized noise.

11. The electronic device of claim 1 , wherein the instructions, when executed by the processor, further cause the processor to: controlling the fourth voice signal, in, The fourth voice signal does not correspond to the fourth parameter in the first response message, and The second response message does not include the fourth voice signal.

12. A method performed by an electronic device, the method comprising: performing speech recognition of a user's speech received by a microphone, and outputting a first response message based on the speech recognition of the user's speech; detecting a re-request for the first response message; identifying text of the first response message; Determining a second response message and a first voice signal based on a first parameter corresponding to the text; as well as The second response message is generated including the determined first voice signal.

13. The method according to claim 12, wherein: Generating the second response message further includes generating the second response message based on a dialogue template, wherein the dialogue template includes: A first area including: a predetermined phrase or a predetermined word including fixed content, and a second area including: a phrase or word having content that can be changed based on information received from the application, and The phrases or words in the second area correspond to the second parameter.

14. The method according to claim 13, further comprising: The phrases or words of the second region are determined based on information received from the application.

15. The method according to claim 13, further comprising: Based on the second parameter, a phrase or word of the second response message is determined based on the phrase or word of the second area.

Images