Electronic device for responding to user utterance using artificial intelligence, and operating method thereof

Abstract

This electronic device comprises: a deformable housing; a communication circuit; a microphone; a display that can at least partially deform so that the electronic device is in a first state or a second state according to the deformation of the housing; and a processor. A first display region of the display is activated when the electronic device is in the first state. A second display region larger than the first display region in the display is activated when the electronic device is in the second state. The processor can: receive a voice input from a user through the microphone; transmit a first query, including first text generated on the basis of the voice input, to a server via the communication circuit; obtain, from the server, a first response to the first query; display first content, corresponding to the first response, on the first display region while the electronic device is in the first state; transmit a second query, including second text different from the first text, to the server via the communication circuit at least partially on the basis of the electronic device changing from the first state to the second state; obtain, from the server, a second response to the second query; and display second content, corresponding to the second response, on the second display region simultaneously with the first content while the electronic device is in the second state.

Description

Electronic device for responding to user speech using artificial intelligence and method of operating the same

[0001] The present disclosure relates to an electronic device and a method of operating the same for responding to user speech using artificial intelligence.

[0002] Electronic devices can provide artificial intelligence (AI) agent services. For example, the electronic device can acquire user speech (speech) from audio signals input through a microphone, recognize the user speech, and, based on the results, construct a response and display it on a user interface (UI) screen for user interaction.

[0003] The above information is provided as background information to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] An electronic device can obtain a list of utterances associated with a user utterance. The electronic device can display the obtained list of associated utterances along with a response to the user utterance on a display. When a user selects an associated utterance from the list, the electronic device can obtain a response corresponding to the selected associated utterance and display it on the display.

[0005] Electronic devices may have a deformable housing structure. For example, the electronic device may have a foldable housing structure or a slideable housing structure. Depending on the state of the housing structure, the size of the display area for displaying visual information may be expanded or reduced, or the display area may change.

[0006] Embodiments of the present disclosure can provide an electronic device having a deformable housing structure and configured to provide a user with various responses associated with user speech depending on the state of the housing structure. The technical problems to be solved by the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the description below.

[0007] According to one embodiment, an electronic device includes a transformable housing; a communication circuit housed within the housing; a microphone housed within the housing; a display housed within the housing and at least partially deformable such that the electronic device has a first state or a second state according to deformation of the housing; and a processor housed within the housing. A first display area of ​​the display may be configured to be activated while the electronic device is in the first state. A second display area of ​​the display may be configured to be activated while the electronic device is in the second state. The second display area is wider than the first display area. The processor may be configured to receive a voice input from a user via the microphone. The processor may be configured to transmit a first query including a first text generated based on the voice input to a server via the communication circuit. The processor may be configured to obtain a first response to the first query from the server. The processor may be configured to display first content corresponding to the first response through the first display area while the electronic device is in the first state. The processor may be configured to transmit a second query including second text different from the first text to the server through the communication circuit, at least in part based on the electronic device changing from the first state to the second state. The processor may be configured to obtain a second response to the second query from the server. The processor may be configured to display second content corresponding to the second response through the second display area simultaneously with the first content while the electronic device is in the second state.

[0008] In one embodiment, a method is performed in an electronic device having a deformable housing. The method may include receiving a user's voice input through a microphone of the electronic device. The method may include transmitting a first query, including a first text generated based on the voice input, to a server through a communication circuit of the electronic device. The method may perform an operation of obtaining a first response to the first query from the server. The method may include displaying first content corresponding to the first response through a first display area while the electronic device is in a first state. The method may include transmitting a second query, including a second text different from the first text, to the server through the communication circuit based at least in part on the electronic device changing from the first state to a second state due to deformation of the housing. The method may include an operation of obtaining a second response to the second query from the server. The method may include an operation of displaying, while the electronic device is in the second state, second content corresponding to the second response through a second display area that is wider than the first display area, simultaneously with the first content.

[0009] According to one embodiment, an electronic device includes a transformable housing; a communication circuit housed within the housing; a microphone housed within the housing; a display housed within the housing and at least partially deformable such that the electronic device has a first state or a second state according to deformation of the housing; a memory including instructions; and a processor housed within the housing. A first display area of ​​the display may be configured to be activated while the electronic device is in the first state. A second display area of ​​the display may be configured to be activated while the electronic device is in the second state. The second display area is wider than the first display area. The processor may be configured to receive a voice input from a user via the microphone. The instructions, when executed by the processor, may cause the electronic device to transmit a first query, including a first text generated based on the voice input, to a server via the communication circuit. The instructions, when executed by the processor, may cause the electronic device to obtain a first response to the first query from the server. The instructions, when executed by the processor, may cause the electronic device to display first content corresponding to the first response through the first display area while the electronic device is in the first state. The instructions, when executed by the processor, may cause the electronic device to transmit a second query to the server through the communication circuit, the second query including second text different from the first text, at least in part based on the electronic device changing from the first state to the second state.The instructions, when executed by the processor, may cause the electronic device to obtain a second response to the second query from the server. The instructions, when executed by the processor, may cause the electronic device to display, through the second display area, second content corresponding to the second response simultaneously with the first content while the electronic device is in the second state.

[0010] According to one embodiment, a recording medium storing instructions readable by a processor of an electronic device is provided. The instructions, when executed by the processor, may cause the electronic device to perform an operation of receiving a voice input from a user through a microphone of the electronic device. The instructions, when executed by the processor, may cause the electronic device to perform an operation of transmitting a first query including a first text generated based on the voice input to a server through a communication circuit of the electronic device. The instructions, when executed by the processor, may cause the electronic device to perform an operation of obtaining a first response to the first query from the server. The instructions, when executed by the processor, may cause the electronic device to perform an operation of displaying first content corresponding to the first response through a first display area while the electronic device is in a first state. The instructions, when executed by the processor, may cause the electronic device to perform an operation of transmitting a second query to the server via the communication circuit, the second query including second text different from the first text, at least in part based on a change of the electronic device from the first state to the second state due to deformation of a housing of the electronic device. The instructions, when executed by the processor, may cause the electronic device to perform an operation of obtaining a second response to the second query from the server. The instructions, when executed by the processor, may cause the electronic device to perform an operation of displaying, while the electronic device is in the second state, second content corresponding to the second response through a second display area wider than the first display area, simultaneously with the first content.

[0011] According to one embodiment, an electronic device stores instructions executable by at least one processor. The instructions, when executed by the processor, may cause the electronic device to receive a user's voice input through a microphone and transmit a first query including a first text generated based on the voice input to a server through a communication circuit. The instructions, when executed by the processor, may cause the electronic device to obtain a first response to the first query from the server and, while the electronic device is in the first state, display first content corresponding to the first response through the first display area. The instructions, when executed by the processor, may cause the electronic device to transmit a second query including a second text different from the first text to the server through the communication circuit, at least in part based on a state of the electronic device changing from the first state to the second state. The instructions, when executed by the processor, may cause the electronic device to obtain a second response to the second query from the server. The above command, when executed by the processor, may cause the electronic device to display second content corresponding to the second response through the second display area simultaneously with the first content while the electronic device is in the second state.

[0012] According to embodiments of the present disclosure, an electronic device has a deformable housing structure and can provide various responses associated with user speech depending on the state of the housing structure. Additionally, various effects, directly or indirectly identified through this document, can be provided.

[0013] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0014] FIG. 2 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0015] FIG. 3 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0016] FIG. 4 is a block diagram illustrating an automatic speech recognition module according to one embodiment.

[0017] Figure 5 is a block diagram of a natural language understanding module according to one embodiment.

[0018] FIGS. 6A to 6J illustrate an electronic device having a multi-foldable housing structure in an in-folding manner, according to one embodiment.

[0019] FIGS. 7A to 7D illustrate an electronic device having a sliderable housing structure, according to one embodiment.

[0020] FIG. 8 is a block diagram of an electronic device having a deformable flexible display and a function to respond to user speech, according to one embodiment.

[0021] Figure 9 is a block diagram of an associated utterance agent module according to one embodiment.

[0022] FIG. 10 is a flowchart illustrating operations for obtaining associated utterances in a client device according to one embodiment.

[0023] FIG. 11 is a flowchart illustrating operations for obtaining associated utterances from a server according to one embodiment.

[0024] FIG. 12 is a flowchart illustrating operations for providing a response from an intelligent server to a user utterance based on the state of an electronic device, according to one embodiment.

[0025] FIG. 13 is a flowchart illustrating operations of an electronic device for providing a server response to a user utterance, according to one embodiment.

[0026] FIG. 14 is a flowchart illustrating operations of an electronic device for providing a server response to a user utterance based on the electronic device changing from a folded state to an unfolded state, according to one embodiment.

[0027] FIG. 15 is a flowchart illustrating operations of an electronic device for providing a server response to a user utterance based on the electronic device changing from a slide-in state to a slide-out state, according to one embodiment.

[0028] FIGS. 16a, 16b, 16c, and 16d are diagrams illustrating UI screens that provide a server response to a user utterance, according to one embodiment.

[0029] FIGS. 17a, 17b, and 17c are diagrams illustrating UI screens that provide a server response to a user utterance based on an enlargement of a display area, according to one embodiment.

[0030] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0031] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0032] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0033] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0034] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0035] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0036] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0037] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0038] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0039] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0040] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0041] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0042] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0043] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0044] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0045] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0046] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0047] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0048] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0049] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0050] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0051] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0052] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0053] FIG. 2 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0054] Referring to FIG. 2, an integrated intelligent system of one embodiment may include a first electronic device (201) (e.g., the electronic device (101) of FIG. 1), a second electronic device (202) (e.g., any device including a headset, earbuds, or microphone), an intelligent server (300), and a service server (399).

[0055] According to one embodiment, the first electronic device (201) may include a communication interface (210), an input / output (I / O) interface (220), a processor (230), and / or a memory (240). The components listed above may be operatively or electrically connected to each other. For example, the electronic device (201) may include at least some of the components of the electronic device (101) of FIG. 1.

[0056] The communication interface (210) can be connected to an external device (e.g., an intelligent server (300) and / or a service server (399)) via a first network (299) (e.g., any network including a cellular network and / or a wireless local area network (WLAN)) to transmit and receive data. For example, the communication interface (210) can correspond to the communication module (190) of FIG. 1. The communication interface (210) can support data transmission and reception with an external device (e.g., a second electronic device (202)) via a second network (298) (e.g., a short-range wireless communication network).

[0057] The I / O interface (220) may receive user input, process received user input, and / or output results processed by the processor (230) using input / output devices (not shown) (e.g., a microphone, a speaker, and / or a display (e.g., a display module (160) of FIG. 1).

[0058] The processor (230) may be operatively or electrically connected to a communication interface (210), an I / O interface (220), and / or a memory (240) (e.g., the memory (130) of FIG. 1) to perform a designated operation. For example, the processor (230) may correspond to the processor (120) of FIG. 1. The processor (230) may execute a program (or one or more instructions) stored in the memory (240) to perform a designated operation. For example, the processor (230) may receive a user's voice input (e.g., a user's speech) through the I / O interface (220). For example, the processor (230) may receive a user's voice input received by the second electronic device (202) from the second electronic device (202) through the communication interface (210). The processor (230) can transmit voice input received through the communication interface (210) to the intelligent server (300). For example, the processor (230) can include one or more processors.

[0059] The processor (230) may receive a result corresponding to the voice input from the intelligent server (300). For example, the processor (230) may receive a plan corresponding to the voice input and / or a result calculated using the plan from the intelligent server (300). For example, the plan may include, but is not limited to, information regarding a plurality of sequential operations to be executed by the first electronic device (201) and / or another electronic device in relation to the voice input. The processor (230) may receive a request from the intelligent server (300) to obtain information (e.g., entities, slots, and / or parameters) necessary to generate a plan corresponding to the voice input. The processor (230) may transmit the necessary information to the intelligent server (300) in response to the request.

[0060] The processor (230) can visually, tactilely, and / or audibly output the results of executing the operations specified according to the plan through the I / O interface (220). For example, the processor (230) can sequentially display the execution results of a plurality of operations on the display. As an example, the processor (230) can display only the execution results of executing a plurality of operations (e.g., the execution result of one of the plurality of operations or the execution result of the last operation) on the display. The processor (230) can provide feedback through the second electronic device (202) by transmitting the execution results of the plurality of operations or the execution results of at least some of the plurality of operations to the second electronic device (202).

[0061] The processor (230) can recognize voice input. For example, the processor (230) can execute an intelligent app (or a voice recognition app) to process the voice input in response to a specified voice input (e.g., "Wake up!"). The processor (230) can provide a voice recognition service through the intelligent app. The processor (230) can transmit the voice input to the intelligent server (300) through the intelligent app and receive a result corresponding to the voice input from the intelligent server (300).

[0062] In one example, the second electronic device (202) may include a communication interface (211), an input / output (I / O) interface (221), a processor (231), and / or a memory (241). The components listed above may be operatively or electrically connected to each other. In one example, the second electronic device (202) may be a set of a plurality of electronic devices configured as a single set (e.g., a left earbud and a right earbud).

[0063] The communication interface (211) may support connection with an external device (e.g., the first electronic device (201)) via a second network (298). The I / O interface (221) may receive user input, process received user input, and / or output a result processed by the processor (231) using input / output devices (not shown) (e.g., at least one microphone, at least one speaker, and / or button).

[0064] The processor (231) may be operatively and / or electrically connected to the communication interface (211), the I / O interface (221), and / or the memory (241) to perform a designated operation. The processor (231) may execute a program (or one or more instructions) stored in the memory (241) to perform a designated operation. For example, the processor (231) may receive a user's voice input (e.g., a user's speech) through the I / O interface (221). In one example, the processor (231) may perform voice activity detection (VAD) using at least one sensor (not shown) of the second electronic device (202). The processor (231) may detect a user's speech of the second electronic device (202) using an acceleration sensor and / or a microphone.

[0065] The processor (231) can transmit voice input received through the second network (298) to the first electronic device (201) using the communication interface (211).

[0066] The processor (231) can receive a result corresponding to a voice input from the first electronic device (201). For example, the processor (231) can receive data (e.g., text data) corresponding to the result corresponding to the voice input from the first electronic device (201). The processor (231) can output the received result through the I / O interface (221).

[0067] The processor (231) can recognize a voice input. For example, the processor (231) can request the first electronic device (201) to execute an intelligent app (or a voice recognition app) to process the voice input in response to a specified voice input (e.g., wake up!).

[0068] An intelligent server (300) of one embodiment can receive a user's voice input from a first electronic device (201) via a first network (299). The intelligent server (300) can convert audio data corresponding to the received voice input into text data. The intelligent server (300) can generate at least one plan for performing a task corresponding to the user's voice input based on the text data. The intelligent server (300) can transmit the generated plan or a result according to the generated plan to the first electronic device (201) via the first network (299).

[0069] An intelligent server (300) of one embodiment may execute one or more programs including a front end (310), a natural language platform (320), a capsule database (330), an execution engine (340), and / or an end user interface (350).

[0070] The front end (310) can receive a voice input received by the first electronic device (201) or the second electronic device (202) from the first electronic device (201). The front end (310) can transmit a response corresponding to the voice input to the first electronic device (201).

[0071] The natural language platform (320) may include an automatic speech recognition (ASR) module (321), a natural language understanding (NLU) module (323), a planner module (325), a natural language generator (NLG) module (327), and / or a text-to-speech (TTS) module (329).

[0072] The automatic speech recognition module (321) can convert the voice input received from the first electronic device (201) into text data. The natural language understanding module (323) can identify the user's intent and / or parameters (e.g., entities and / or slots) based on the text data of the voice input. The user's intent corresponds to the voice input and may include information indicating an action (or function) that the user wishes to perform using the device. The slot may be detailed information related to the user's intent. The slot may be acquired based on a domain corresponding to the utterance. The slot may be variable information required to perform the action. In one embodiment, the variable information constituting the slot may include a named entity.

[0073] The planner module (325) can generate a plan using the intent and / or parameters determined by the natural language understanding module (323). For example, the planner module (325) can determine at least one domain necessary to perform a task based on the determined intent. The domain may correspond to a category (or service) associated with an action (or function) that the user wishes to perform using the device. The domain may be classified according to a service (e.g., an app) related to the text. The domain may be related to the user's intent corresponding to the text. The domain may be classified according to, for example, the type of application that received the voice input and / or the type of service to be provided based on the voice input, but is not limited thereto. In one example, the determination of the domain may be performed by another module (e.g., the natural language understanding module (323)). The planner module (325) may determine a plurality of actions included in each of the at least one domain determined based on the intent. The planner module (325) can determine parameters required to execute a plurality of determined operations or result values ​​output by the execution of the plurality of operations. The parameters and result values ​​can be defined as concepts of a specified format (or class). For example, the plan can include a plurality of operations and / or a plurality of concepts determined by the user's intention. The planner module (325) can determine the relationship between the plurality of operations and / or the plurality of concepts in a step-by-step (or hierarchical) manner. For example, the planner module (325) can identify the execution order of the plurality of operations (e.g., the plurality of operations determined based on the user's intention) based on the plurality of concepts (e.g., parameters required to execute the plurality of operations and results output by the execution of the plurality of operations). The planner module (325) can generate a plan including association information (e.g., ontology) between the plurality of operations and the plurality of concepts.The planner module (325) can create a plan using information (e.g., at least one capsule) stored in a capsule database (330) in which a set of relationships between concepts and actions is stored.

[0074] The planner module (325) can generate a plan based on an artificial intelligence (AI) system. For example, the AI ​​system can include one or more electronic devices and / or one or more processing circuits to execute a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN) and / or a recurrent neural network (RNN)), or a combination thereof. The AI ​​system described above is exemplary, and the AI ​​system can be an AI system based on any machine learning-based model. The planner module (325) can select a plan corresponding to a user request from a set of predefined plans, or generate a plan in real time in response to a user request.

[0075] The natural language generation module (327) can convert specified information into text format. The information converted into text format may be in the form of natural language speech. The text-to-speech conversion module (329) can convert information in text format into information in speech format.

[0076] The capsule database (330) can store information on the relationship between multiple concepts and actions corresponding to multiple domains (e.g., applications). The capsule database (330) can store at least one capsule (e.g., capsule (331) and / or capsule (333)) in the form of a concept action network (CAN). For example, the capsule database (330) can store an action for processing a task corresponding to a user's voice input and / or parameters required for the action in the form of a CAN. A capsule can include multiple action objects (or action information) and / or concept objects (or concept information) included in a plan. For example, capsules (331, 333) can be created for each domain and stored in the capsule database (330), but are not limited thereto.

[0077] The execution engine (340) can produce results using the generated plan. The end user interface (350) can transmit the produced results to the first electronic device (201).

[0078] According to one embodiment, some functions (e.g., natural language platform (320)) or all functions of the intelligent server (300) may be implemented in the first electronic device (201). For example, the first electronic device (201) may execute one or more programs including a natural language platform (e.g., natural language platform (250) of FIG. 3) separately from the intelligent server (300). For example, the electronic device (201) may directly perform at least some of the operations of the natural language platform (320) of the intelligent server (300) (e.g., automatic speech recognition module (321), natural language understanding module (323), planner module (325), natural language generation module (327), and / or text-to-speech module (329)).

[0079] In one embodiment, a service server (399) may provide a service (e.g., food ordering or hotel reservation) designated to a first electronic device (201). The service server (399) may be a server operated by a different operator than the intelligent server (300). The service server (399) may communicate with the intelligent server (300) and / or the first electronic device (201) via the first network (299). The service server (399) may communicate with the intelligent server (300) via a separate connection (not shown). The service server (399) may provide the intelligent server (300) with information for generating a plan corresponding to a voice input received by the first electronic device (201) (e.g., operation information and / or concept information for providing a designated service). The provided information may be stored in a capsule database (330). The service server (399) can provide the result information according to the plan received from the first electronic device (201) to the intelligent server (300).

[0080] FIG. 3 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0081] Referring to FIG. 3, the integrated intelligence system may include a first electronic device (201), a second electronic device (202), and an intelligent server (302). The first electronic device (201) and the intelligent server (302) may be connected to each other via a network and may transmit and receive data. The first electronic device (201) and the second electronic device (202) may be connected to each other via a short-range network and may transmit and receive data. According to one embodiment, the integrated intelligence system may be composed of a single device or multiple devices. For example, each device may include a configuration having the same or similar functions, and the configuration of one device may be replaced with the configuration of another device.

[0082] According to one embodiment, the intelligent server (302) may include the entire configuration or at least a portion of the configuration of the intelligent server (300) illustrated in FIG. 2. For example, the intelligent server (302) may execute one or more programs including the natural language platform (320) of the intelligent server (300) of FIG. 2 and / or store the capsule database (330) of FIG. 2. The configuration of the intelligent server (302) is not limited to that illustrated in FIG. 3. For example, at least a portion of the configuration of the natural language platform (320) (e.g., the automatic speech recognition module (321), the natural language understanding module (323), the planner module (325), the natural language generation module (327), and / or the text-to-speech module (329)) may be omitted from the intelligent server (302). For example, the intelligent server (302) may further include some components of the intelligent server (300) of FIG. 2 (e.g., the front end (310), the execution engine (340), and / or the end user interface (350)).

[0083] The first electronic device (201) may execute one or more programs including a natural language platform (250) and / or store a capsule database (260). For example, the first electronic device (201) may further execute one or more programs including a natural language platform (250) and / or store a capsule database (260) while including components of the first electronic device (201) of FIG. 2.

[0084] The natural language platform (250) may include an automatic speech recognition module (251), a natural language understanding module (253), a planner module (255), a natural language generation module (257), and / or a text-to-speech module (259). The automatic speech recognition module (251), the natural language understanding module (253), the planner module (255), the natural language generation module (257), and the text-to-speech module (259) may perform functions identical to or similar to those of the automatic speech recognition module (321), the natural language understanding module (323), the planner module (325), the natural language generation module (327), and the text-to-speech module (329) of FIG. 2, respectively.

[0085] The capsule database (260) may perform the same or similar functions as the capsule database (330) of the intelligent server (300, 302). The capsule database (260) may store information about the relationships between multiple operations and multiple concepts included in the plan generated by the planner module (255). For example, the capsule database (260) may store at least one capsule (e.g., capsule (261) and / or capsule (263)).

[0086] According to one embodiment, the first electronic device (201) (e.g., the natural language platform (250) and / or capsule database (260)) and the intelligent server (302) (e.g., the natural language platform (320) and / or capsule database (330)) may perform at least one function (or operation) in conjunction with each other, or may independently perform at least one function (or operation). For example, the first electronic device (201) may perform voice recognition on its own without transmitting the received user's voice input to the intelligent server (302). As an example, the first electronic device (201) may convert the received voice input into text data through the automatic voice recognition module (251). The first electronic device (201) may transmit the converted text data to the intelligent server (302). The intelligent server (302) may determine (or identify) the user's intention and / or parameters from the text data through the natural language understanding module (323). The intelligent server (302) can generate a plan through the planner module (325) based on the determined intent and parameters and transmit the plan to the first electronic device (201), or can transmit the determined intent and parameters to the first electronic device (201) and cause the plan to be generated through the planner module (255) of the first electronic device (201). The planner module (255) of the first electronic device (201) can generate at least one plan for performing a task corresponding to a voice input using information stored in the capsule database (260).

[0087] For example, the first electronic device (201) can convert voice input received through the automatic speech recognition module (251) into text data, and determine (or identify) the user's intention and / or parameters based on the text data through the natural language understanding module (253). The first electronic device (201) can generate a plan through the planner module (255) based on the determined intention and parameters, or transmit the determined intention and parameters to the intelligent server (302) so that the planner module (325) of the intelligent server (302) can generate a plan. For example, when the planner module (255) and / or the capsule database (260) are not included in the first electronic device (201), the first electronic device (201) can generate a plan through the intelligent server (302).

[0088] For example, the first electronic device (201) can detect a speech pattern that is difficult to learn in an automatic speech recognition module (251) or a natural language understanding module (253), and transmit a voice input corresponding to the detected speech pattern to an intelligent server (302) so that the automatic speech recognition module (321) or the natural language understanding module (323) of the intelligent server (302) can process it.

[0089] Embodiments of the present disclosure are not limited to the examples described above. For example, the first electronic device (201) may process the received voice input only within the terminal and produce a result corresponding to the voice input. For example, the first electronic device (201) and the intelligent server (302) may not only divide the voice input into modules and process it, but may also collaborate with each other to process it. For example, the natural language understanding module (253) of the first electronic device (201) and the natural language understanding module (323) of the intelligent server (302) may work together to produce a single result value (e.g., the user's intention and / or parameters).

[0090] The second electronic device (202) can execute one or more programs including an automatic speech recognition (ASR) module (252) and / or a text-to-speech (TTS) module (254). For example, the second electronic device (202) can include components of the second electronic device (202) of FIG. 2 and execute one or more programs including an automatic speech recognition module (252) and / or a text-to-speech module (254). The automatic speech recognition module (252) and the text-to-speech module (254) can perform functions identical to or similar to the automatic speech recognition module (321) and the text-to-speech module (329) of FIG. 2, respectively.

[0091] According to one embodiment, the first electronic device (201) and the second electronic device (202) may perform at least one function (or operation) in conjunction with each other, or may independently perform at least one function (or operation). For example, the second electronic device (202) may perform voice recognition for a voice input using an automatic voice recognition module (252). The second electronic device (202) may perform a function corresponding to the voice input based on the voice recognition. For example, the second electronic device (202) may transmit a command corresponding to the recognized voice command to the first electronic device (201). The second electronic device (202) may output data received from the first electronic device (201). For example, the second electronic device (202) may convert data received from the first electronic device (201) into voice using a text-to-speech conversion module (254) and output the converted voice.

[0092] FIG. 4 is a block diagram illustrating an ASR module (400) according to one embodiment.

[0093] According to one embodiment, the ASR module (400) (e.g., the automatic speech recognition module (321) of FIG. 2 or the automatic speech recognition module (251) of FIG. 3) may include a front end (410), an end-point detector (EPD) (420), a wake-up module (430), and / or an automatic speech recognition model (440). The ASR module (400) may generate text data using a voice input. For example, the ASR module (400) may obtain a voice input through an I / O interface and / or an external device, and process the obtained voice input to output the generated text data. For example, the ASR module (400) may be a software module implemented by executing instructions by a processor. Hereinafter, operations performed by the ASR module (400) and / or its components may be referred to as operations of a processor of a device implementing the ASR module (400).

[0094] The front end (410) may perform preprocessing operations on the voice input. For example, the front end (410) may remove echo from the voice input using an echo cancellation module (e.g., an acoustic echo canceller (AEC) and / or a residual echo suppressor (RES). For example, the front end (410) may remove noise from the voice input using a noise cancellation module (e.g., a noise suppressor, NS).

[0095] The EPD (420) can detect the end point of a voice input. For example, the EPD (420) can detect the end point of a voice input and, based on the detection result, identify the end point of a user utterance corresponding to the voice input.

[0096] The wake-up module (430) can selectively transmit the voice input to the automatic speech recognition model (440) based on whether the voice input includes a specified wake-up word (or wake-up signal). For example, the wake-up module (430) can identify whether the specified wake-up word is present in the preprocessed voice input from the front end (410) through keyword spotting.

[0097] The automatic speech recognition model (440) may include any model for obtaining text data from speech input. For example, the automatic speech recognition model (440) may extract text data from speech input and / or convert speech input into text data based on a speech-to-text (STT) algorithm. For example, the STT (speech-to-text) algorithm may include a hidden Markov model, a Gaussian-mixture model, a deep neural network model, an N-gram language model, other statistical models, and / or combinations thereof. For example, the automatic speech recognition model (440) may include one or more models. The one or more models may, for example, be models for recognizing different languages. For example, the ASR module (400) can be implemented as an E2E (end to end) model (e.g., connectionist temporal classification (CTC), recurrent neural network transducer (RNN-T), listen, attend, and spell (LAS), or hybrid CTC / LAS).

[0098] For example, the ASR module (400) may further include other components. For example, the ASR module (400) may further include a feature extraction module (not shown) and an encoder (not shown). The ASR module (400) may extract features from a speech input (e.g., using the feature extraction module) to obtain a feature vector, and may encode the feature vector using the encoder.

[0099] FIG. 5 is a block diagram of a natural language understanding module (323) according to one embodiment.

[0100] Referring to FIG. 5, the natural language understanding module (323) may include a natural language understanding model (510) for intent classification, a dispatcher (520), a domain classifier (530), or a combination thereof. For example, the natural language understanding module (323) may be a software module implemented by executing instructions by a processor. Hereinafter, the operations performed by the natural language understanding module (323) may be referred to as operations of the processor of the device implementing the natural language understanding module (323).

[0101] The natural language understanding module (323) can obtain a speech recognition result (e.g., data converted from one or more phonemes included in the user's speech) from the automatic speech recognition module (321). The natural language understanding module (323) can provide the processing result of the speech recognition result to the planner module (325). The processing result of the speech recognition result can include an intent, a target device (e.g., information about the target device), a capsule, or a combination thereof.

[0102] The natural language understanding model (510) can determine intent by interpreting (e.g., syntactic analysis and / or semantic analysis) the speech recognition result from the automatic speech recognition module (321). The natural language understanding model (510) can use linguistic features (e.g., grammatical elements) of morphemes or phrases to identify the meaning of words extracted from the speech recognition result, and can determine the user's intent based on the identified meaning of the word and / or other parameters (e.g., domains or categories associated with the word). Grammatical analysis can include an act of dividing user input (e.g., user's utterance) into grammatical units (e.g., words, phrases, and / or morphemes) and identifying grammatical elements of the divided units. Semantic analysis can be performed through semantic matching, rule matching, and / or formula matching.

[0103] Here, data converted from one or more phonemes may represent one or more words included in the user's utterance, and / or tokens of each of one or more words. The intent may be data used by the natural language platform to generate a plan. The intent may include a goal and / or parameters. The goal may be used to specify the final goal of the plan in the planner module (325). The parameters may be values ​​input to one or more actions included in the plan in the planner module (325).

[0104] The dispatcher (520) can determine a target device and / or capsule (or domain, application) associated with one or more words included in the speech recognition result.

[0105] The dispatcher (520) may include a device dispatcher (521), a named dispatcher (523), a meta command dispatcher (525), or a combination thereof.

[0106] The device dispatcher (521) can determine one or more target devices based on the device names included in the voice recognition results. The target devices may be Internet of Things (IoT) devices. The target devices may include, but are not limited to, devices that perform the operations included in the plan and / or devices that receive the results of performing the operations included in the plan.

[0107] The named dispatcher (523) can determine one or more capsules based on the names of the capsules included in the speech recognition results.

[0108] The meta-command dispatcher (525) can determine one or more capsules based on specific commands included in the speech recognition result. The specific commands may be commands designated for specific situations. The specific situations may include situations in which the speech recognition service prompts the user, and / or situations in which content selection is possible from a content list including one or more contents. For example, in a prompting situation, the specific commands may include select (e.g., "first"), cancel (e.g., "cancel"), confirm (e.g., "understood"), or a combination thereof. For example, in situations in which content selection is possible, the specific commands may include repeat (e.g., "again"), next (e.g., "next"), previous (e.g., "back to previous"), or a combination thereof.

[0109] The domain classifier (530) can determine one or more capsules required to perform a task based on the speech recognition results. The domain classifier (530) can determine one or more capsules using predefined classification rules and / or artificial intelligence models. The predefined classification rules and / or artificial intelligence models can be learned using a learning algorithm.

[0110] Hereinafter, for convenience of explanation, the surface of the display (e.g., a flexible display or a main display) that is visually exposed to the user may be referred to as the front surface of the electronic device (101). In addition, the surface opposite the front surface may be referred to as the back surface of the electronic device (101). In addition, the surface surrounding the space between the front surface and the back surface may be referred to as the side surface of the electronic device (101). In this document, the term “state” may refer to the structural form, posture, shape, or configuration of the electronic device (101) (or, the display, slider, or housing constituting the electronic device (101).

[0111] In an electronic device, the display mode may be determined as a portrait mode or a landscape mode based on the state of the electronic device. For example, in FIG. 1, the sensor module (176) may include a sensor (e.g., an acceleration sensor, a gyro sensor) that identifies the relative positions between sides of the housing structure of the electronic device (101) and generates data used to identify the movement and direction of movement of the electronic device (101). The processor (120) may recognize the posture in which the electronic device is positioned based on the data received from the sensor, and determine the display mode as a portrait mode or a landscape mode based on the recognized posture. The processor (120) may display visual information on the display based on the determined display mode.

[0112] A bar-type housing structure can be applied to an electronic device (e.g., a smart phone, a tablet PC) (101). For example, the bar-type housing structure can include a plate (or cover) forming the front surface of the electronic device (101), a plate forming the rear surface of the electronic device (101), and a bezel structure forming a side surface surrounding the front and rear surfaces. A display can be placed on the front surface.

[0113] A foldable housing structure may be applied to an electronic device (e.g., a smart phone, a tablet PC, a notebook PC) (101). For example, the electronic device (101) may have a foldable housing structure that is divided into two housings centered on a folding axis. A first display area of ​​a display (e.g., a flexible display) may be arranged in the first housing, and a second display area of ​​the display may be arranged in the second housing. The foldable housing structure may be implemented in an in-folding manner in which the first display area and the second display area face each other when the electronic device (101) is in a folded state. Alternatively, the foldable housing structure may be implemented in an out-folding manner in which the first display area and the second display area face each other when the electronic device (101) is in a folded state. The electronic device may further include a sub-display. For example, a flexible display, which is a main display, may be arranged on the front of the electronic device, and a sub-display may be arranged on the back of the electronic device.

[0114] A slidable (or rollable) housing structure can be applied to an electronic device (e.g., a smart phone, a tablet PC, a notebook PC) (101). The electronic device (101) can include a slidable housing including a housing (or a first housing) and a slider (or a second housing), a rail structure (e.g., a rail structure by gear engagement between a rack gear and a pinion gear) that allows the slider to be inserted into the housing and the slider to be extracted from the housing, and a rollable display (e.g., a flexible display). The slider can be divided into a portion that can be inserted into the housing (hereinafter, referred to as an inlet portion) and a portion that remains exposed to the outside. When the inlet portion of the slider is completely extracted from the housing in a slide-out state (in other words, a first state, an open state, an extended state, a roll-out state), the entire display (or a majority of the display area) can be exposed to the outside through the front. As the slider's inlet portion is retracted into the housing, the display can also be retracted into the housing. The display can also be divided into a portion that remains exposed to the outside (e.g., a first display area, a first section) and a portion that can be retracted into the housing (e.g., a second display area, a second section, a bendable section). When the entire slider's inlet portion is switched to a slide-in state (in other words, a second state, a closed state, a reduced state, a roll-in state) in which the slider is retracted into the housing, the entire second display area of ​​the display can be retracted into the housing. In one embodiment, when switching from a slide-out state to a slide-in state, a portion of the display (e.g., a second display area) may be moved to the side and toward the rear without being retracted into the housing.As exemplified above, the electronic device (101) may have a sliding structure in which a portion of the display is retracted into the housing, or a sliding structure in which a portion of the display is moved from the front to the rear. In the display, only a portion exposed through the front may be determined as an activated display area (hereinafter, “active area”) that displays visual information. A portion retracted into the housing or moved to the rear may be determined as an inactive area in which no visual information is displayed. An electronic device having a slideable housing structure may include a separate button for switching states from slide-in to slide-out or vice versa.

[0115] FIGS. 6A to 6J illustrate an electronic device (600) having a multi-foldable housing structure in an in-folding manner according to one embodiment. Referring to FIGS. 6A to 6J, the electronic device (600) (e.g., the electronic device (101)) may include a first housing (610), a second housing (620), a third housing (630), a first hinge assembly (640), a second hinge assembly (650), a front display (660), a rear display (670), and a plurality of cameras (681, 682, 683, 684, 685). The first hinge assembly (640) may be configured to couple the first housing (610) to one side of the second housing (620) and allow the first housing (610) to rotate relative to the second housing (620). The second hinge assembly (650) may be configured to couple the third housing (630) to the other side of the second housing (620) and allow the third housing (630) to rotate relative to the second housing (620). The front display (660) and the rear display (670) may be positioned within the space formed by the multi-foldable housing structures (610, 620, 630).

[0116] Referring to FIG. 6A, a front display (or, in other words, a main display) (660) may be arranged from the first housing (610) across the first hinge assembly (640), the second housing (620), and the second hinge assembly (650) to the third housing (630). The front display (660) may be a component corresponding to the second display of FIGS. 12 to 14 and FIGS. 17A to 17C, which will be described later. According to one embodiment, the screen of the front display (660) may be visually exposed through the front of the electronic device (600). The front display (660) may be implemented as a flexible display such that at least a portion positioned above the first hinge assembly (640) and the second hinge assembly (650) may be bent. The front display (660) may be divided into a first front display area (or first main display area) (661) located in the first housing (610), a second front display area (or second main display area) (662) located in the second housing (620), and a third front display area (or third main display area) (663) located in the third housing (630), based on the first folding axis (A) of the first hinge assembly (640) and the second folding axis (B) of the second hinge assembly (650). The electronic device (600) may include at least one front camera arranged inside the electronic device (600) such that its lens is visually exposed through the front. For example, the first camera (681) may be located below the second front display area (662). The lens of the first camera (681) is positioned to face the second front display area (662), thereby allowing light to be received from outside the electronic device (600) through at least a portion of the second front display area (662) or at least one opening formed in the second front display area (662).In one embodiment, the first camera (681) may be referred to as an under display camera (UDC) as it is positioned under the display and receives light through the display.

[0117] Referring to FIG. 6B, a rear display (or, alternatively, a sub-display) (670) may be located in the first housing (610). The rear display (670) may be a component corresponding to the first display of FIGS. 12 to 14 and 17A to 17C, which will be described later. The screen of the rear display (670) may be visually exposed through the rear of the electronic device (600). The electronic device (600) may include at least one rear camera including a lens positioned toward the rear of the electronic device (600). For example, the electronic device (600) may include a second camera (682) positioned in the second housing (620) and including a lens facing the rear of the electronic device (600), and a third camera (683) positioned below the rear display (670) and including a lens facing the rear of the electronic device (600). According to one embodiment, the electronic device (600) may further include a fourth camera (684) and a fifth camera (685) having lenses facing the rear of the electronic device (600). The fourth camera (684) and the fifth camera (685) may be positioned adjacent to the second camera (682) in the second housing (620). For example, the second camera (682) may include a wide-angle lens, the fourth camera (684) may include an ultra-wide-angle lens, and the fifth camera (685) may include a telephoto lens.

[0118] In one embodiment, a state between two housings may be defined based on an angle formed between the two housings joined by a hinge assembly. For example, if the angle between the two housings is approximately 180 degrees, the state may be defined as an unfolded (flat) or open state. In one embodiment, if the angle between the two housings is less than or equal to approximately 10 degrees, the state may be defined as a folded (closed) state. In one embodiment, if the two housings form an angle that is greater than the angle when they are folded but less than the angle when they are unfolded (e.g., between approximately 10 degrees and 179 degrees), the state may be defined as an intermediate state. Here, the intermediate state may be expressed as a partially folded state or a partially unfolded state.

[0119] Referring to FIGS. 6A, 6C, 6D, and 6E, when the housings (610, 620, 630) are all in an unfolded state (FIG. 6A), the display areas (661, 662, 663) may all face substantially in the same direction. In one embodiment, in this state, the first hinge assembly (640) may rotate (C; FIG. 6C) so that the first foldable housing structure (610, 620) may enter an intermediate state. The first hinge assembly (640) may further rotate (D; FIG. 6D), thereby bringing the first foldable housing structure (610, 620) into a folded state and allowing the first front display area (661) to face the second front display area (662). As the second hinge assembly (650) rotates (E; FIG. 6e) while the first foldable housing structure (610, 620) is folded, the second foldable housing structure (620, 630) can be in an intermediate state. When the first foldable housing structure (610, 620) is in a folded state and the second hinge assembly (650) is in an unfolded state or an intermediate state, only the third front display area (663) can be exposed to the outside.

[0120] Referring to FIGS. 6a, 6f, 6g, 6h, and 6i, when the housings (610, 620, 630) are fully unfolded (FIG. 6a), the second hinge assembly (650) may rotate (F; FIG. 6f) so that the second foldable housing structure (620, 630) may transition to an intermediate state. The second hinge assembly (650) may further rotate (G; FIG. 6g), thereby bringing the second foldable housing structure (620, 630) into a folded state and allowing the third front display area (663) to face the second front display area (662). When the second foldable housing structure (620, 630) is folded, the first hinge assembly (640) may rotate (H; FIG. 6h). Accordingly, the first foldable housing structure (610, 620) can be converted to an intermediate state. When the second foldable housing structure (620, 630) is in a folded state and the first foldable housing structure (610, 620) is in an unfolded state or an intermediate state, only the first front display area (661) can be exposed to the outside. The first hinge assembly (640) can be further rotated (I; FIG. 6i), and thus, all of the housings (610, 620, 630) can be folded into a G shape.

[0121] Referring to FIGS. 6A and 6J, when the housings (610, 620, 630) are fully unfolded (FIG. 6A), the first hinge assembly (640) rotates (J1; FIG. 6J), thereby bringing the first foldable housing structure (610, 620) into an intermediate state. When the second hinge assembly (650) rotates (J2; FIG. 6J) when the first foldable housing structure (610, 620) is in the intermediate state, the second foldable housing structure (620, 630) may come into an intermediate state.

[0122] FIGS. 7A to 7D illustrate an electronic device (700) having a sliderable housing structure according to one embodiment. Specifically, FIGS. 7A and 7B are diagrams illustrating the front and back of the electronic device (700) in a slide-in state. FIGS. 7C and 7D are diagrams illustrating the front and back of the electronic device (700) in a slide-out state.

[0123] Referring to FIGS. 7A to 7D , an electronic device (700) (e.g., the electronic device (101) of FIG. 1 ) may include a first housing (or housing) (710), a second housing (or slider) (720) slidably coupled from the first housing (710) in a specified direction (e.g., direction ① or direction ②) (e.g., ± y-axis direction), and a rollable display (730) (e.g., a flexible display, an expandable display, or a stretchable display) arranged to be supported by at least a portion of the first housing (710) and the second housing (720). According to one embodiment, the electronic device (700) may be configured such that the second housing (720) slides out in a first direction (direction ①) or slides in in a second direction (direction ②) opposite to the first direction (direction ①) based on the first housing (710) held by the user. According to one embodiment, at least a portion of the second housing (720) including the second space (7201) may be accommodated in the first space (7101) of the first housing (710), thereby changing to a slide-in state. According to one embodiment, the electronic device (700) may include a support member (e.g., a bendable member, a multi-joint hinge module, or a multi-bar assembly) that, in a slide-out state, forms substantially the same plane as at least a portion of the second housing (720), and that, in a slide-in state, is at least partially accommodated into the first space (7101) of the first housing (710). According to one embodiment, at least a portion of the rollable display (730) may be accommodated in a bendable manner into the first space (7101) of the first housing (710) while being supported by the support member in the slide-in state, thereby being arranged so as to be invisible from the outside.According to one embodiment, at least a portion of the rollable display (730) may be positioned so as to be visible from the outside while being supported by a support member that forms at least partially substantially the same plane as the second housing (720) in the slide-out state.

[0124] The first housing (710) can include a first side member (711). In one embodiment, the first side member (711) can include a first side member (7111) having a first length along a first direction (e.g., a y-axis direction), a second side member (7112) extending from the first side member (7111) to have a second length along a direction substantially perpendicular to the first side member (7111) and shorter than the first length, and a third side member (7113) extending from the second side member (7112) substantially parallel to the first side member (7111) and having the first length. In one embodiment, the first housing (710) can include a first extension member (712) extending from at least a portion of the first side member (711) to at least a portion of the first space (7101). In one embodiment, the first extension member (712) can be formed integrally with the first side member (711). In some embodiments, the first extension member (712) may be formed separately from the first side member (711) and structurally coupled to the first side member (711).

[0125] The second housing (720) can include a second side member (721). The second side member (721) can include a fourth side member (7211) that corresponds at least partially with the first side member (7111) and has a third length, a fifth side member (7212) that extends from the fourth side member (7211) in a direction substantially parallel to the second side member (7112) and has a fourth length that is shorter than the third length, and a sixth side member (7213) that extends from the fifth side member (7212) to correspond with the third side member (7113) and has a third length. In one embodiment, at least a portion of the second side member (721) can include a second extension member (722) that extends to at least a portion of the second space (7201) of the second housing (720). In one embodiment, the second extension member (722) can be formed integrally with the second side member (721). In some embodiments, the second extension member (722) may be formed separately from the second side member (721) and structurally coupled to the second side member (721).

[0126] In one embodiment, the first side (7111) and the fourth side (7211) can be slidably coupled to each other. The third side (7113) and the sixth side (7213) can be slidably coupled to each other. In one embodiment, in the slide-in state, the fourth side (7211) can be arranged to overlap the first side (7111) so as to be substantially invisible from the outside. In the slide-in state, the sixth side (7213) can be arranged to overlap the third side (7113) so as to be substantially invisible from the outside. In some embodiments, at least a portion of the fourth side (7211) and the sixth side (7213) can be arranged to be at least partially visible from the outside in the slide-in state. In one embodiment, in the slide-in state, the second extension member (722) can be arranged to overlap the first extension member (712) so as to be substantially invisible from the outside.

[0127] The first housing (710) may include a first rear cover (713) coupled with at least a portion of the first side member (711). In one embodiment, the first rear cover (713) may be arranged to couple with at least a portion of the first extension member (712). In some embodiments, the first rear cover (713) may be formed integrally with the first side member (711). In some embodiments, the first rear cover (713) may extend to at least a portion of the first side member (711). In some embodiments, at least a portion of the first extension member (712) may be replaced by the first rear cover (713).

[0128] The second housing (720) may include a second rear cover (723) coupled with at least a portion of the second side member (721). In one embodiment, the second rear cover (723) may be arranged such that it couples with at least a portion of the second extension member (722). In some embodiments, the second rear cover (723) may be formed integrally with the second side member (721). In some embodiments, the second rear cover (723) may extend to at least a portion of the second side member (721). In some embodiments, at least a portion of the second extension member (722) may be replaced by the second rear cover (723).

[0129] According to one embodiment, the rollable display (730) may include a first portion (730a) (e.g., a flat portion) that is always visible from the outside, and a second portion (730b) (e.g., a bendable portion) that extends from the first portion (730a) and is at least partially accommodated in a first space (7101) of the first housing (710) so as not to be visible from the outside in a slide-in state. According to one embodiment, the first portion (730a) may be arranged to be supported by the second housing (720), and the second portion (730b) may be arranged to be at least partially supported by a support member. According to one embodiment, the second portion (730b) of the rollable display (730) may be arranged to form substantially the same plane as the first portion (730a) and be visible from the outside when the second housing (720) is slid-out along the first direction (① direction). According to one embodiment, the second part (730b) of the rollable display (730) may be accommodated in a bendable manner into the first space (7101) of the first housing (710) in a state where the second housing (720) is slid-in along the second direction (② direction) and may be arranged so as not to be visible from the outside. Accordingly, the electronic device (700) may have a display area that is exposed to the outside of the rollable display (730) and capable of displaying visual information may be varied as the second housing (720) is slidably moved from the first housing (710) in a specified direction (e.g., ±y-axis direction).

[0130] According to one embodiment, the rollable display (730) may have a variable length in the first direction (direction ①) according to the sliding movement of the second housing (720). For example, the rollable display (730) may have a first display area (e.g., first portion (730a)) corresponding to a first length (L1) in a slide-in state. According to one embodiment, the rollable display (730) may be expanded to have a third display area corresponding to a third length (L3) longer than the first length (L1) and including the first display area and the second display area (e.g., second portion (730b)) according to the sliding movement of the second housing (720) that is additionally moved by a second length (L2) based on the first housing (710) in a slide-out state.

[0131] The electronic device (700) may include at least one of an input device (e.g., a microphone (703-1)), an audio output device (e.g., a call receiver (706) and / or a speaker (707)), a sensor module (704, 717), a camera module (e.g., a first camera module (705) or a second camera module (716)), a connector port (708), a subscriber identification module (SIM) card (718), a key input device (719), or an indicator (not shown) disposed in a second space (7201) of a second housing (720). According to one embodiment, the electronic device (700) may include another input device (e.g., a microphone (703)) disposed in the first housing (710). In one embodiment, the electronic device (700) may be configured such that at least one of the above-described components is omitted, or other components are additionally included. In one embodiment, at least one of the above-described components may be disposed in the first space (7101) of the first housing (710).

[0132] The input device may include a microphone (703-1). In some embodiments, the input device (e.g., microphone (703-1)) may include multiple microphones arranged to detect the direction of sound. The sound output device may include, for example, a call receiver (706) and a speaker (707). According to one embodiment, sound output from the speaker (707) may be output to the outside through at least one speaker hole formed in a location (e.g., fifth side (7212)) that is always exposed to the outside regardless of the slide state. According to one embodiment, the connector port (708) may be exposed to the outside through a connector port hole formed in the second housing (720) in the slide-out state. In some embodiments, the connector port (708) may be formed in the first housing (710) in the slide-in state and may be exposed to the outside through an opening formed to correspond to the connector port hole. In some embodiments, the call receiver (706) may include an operative speaker (e.g., a piezo speaker) without a separate speaker hole.

[0133] The sensor modules (704, 717) can generate electrical signals or data values ​​corresponding to the internal operating state of the electronic device (700) or the external environmental state. The sensor modules (704, 717) can include, for example, a first sensor module (704) (e.g., a proximity sensor or a light sensor) disposed on the front of the electronic device (700) and / or a second sensor module (717) (e.g., a heart rate monitor (HRM) sensor) disposed on the rear of the electronic device (700). According to one embodiment, the first sensor module (704) can be disposed under the rollable display (730) when viewed facing the front of the electronic device (700). According to one embodiment, the first sensor module (704) and / or the second sensor module (717) may include at least one of a proximity sensor, an ambient light sensor, a time of flight (TOF) sensor, an ultrasonic sensor, a fingerprint recognition sensor, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, or a humidity sensor.

[0134] The camera module may include a first camera module (705) positioned on the front of the electronic device (700) and a second camera module (716) positioned on the rear of the electronic device (700). According to one embodiment, the electronic device (700) may also include a flash (not shown) positioned near the second camera module (716). According to one embodiment, the camera modules (705, 716) may include one or more lenses, an image sensor, and / or an image signal processor. According to one embodiment, the first camera module (705) may be positioned below the rollable display (730) when viewed facing the front of the electronic device (700) and configured to capture an object through a portion of an active area of ​​the rollable display (730).

[0135] According to one embodiment, the first camera module (705) and / or the first sensor module (704) may be arranged in the second space (7201) of the second housing (720) so as to be in contact with the external environment through a transparent area or a perforated opening formed in the rollable display (730). According to one embodiment, an area of ​​the rollable display (730) facing the first camera module (705) may be formed as a transparent area having a designated transmittance as part of a display area for displaying content. According to one embodiment, the transparent area may be formed to have a transmittance in a range of about 5% to about 20%. For example, the transparent area of ​​the rollable display (730) may include an area having a lower pixel arrangement density and / or wiring density than the surrounding area. For example, the transparent area may replace the above-described opening. The first camera module (705) may include a camera, also known as an under display camera (UDC), which receives light through a transparent area of ​​the display (730) as it is positioned under the display (730). In some embodiments, the first sensor module (704) may be positioned to perform its function without being visually exposed through the rollable display (730) in the internal space of the electronic device (700).

[0136] The electronic device (700) may include at least one antenna element electrically connected to a wireless communication circuit (e.g., the wireless communication module (192) of FIG. 1) disposed in a second housing (720). According to one embodiment, the electronic device (700) may also include a bezel antenna (A) disposed through a conductive first side member (711) of the first housing (710). For example, the bezel antenna (A) may include a conductive portion (727) disposed on at least a portion of a second side (7112) and a third side (7113) of the first side member (711) and electrically segmented through at least one segment (7271, 7272) formed of a non-conductive material (e.g., a polymer). According to one embodiment, the electronic device (700) may include a side cover (7112a) disposed on the second side (7112) to cover at least a portion of at least one segment (7271). In some embodiments, the bezel antenna (A) may be disposed on at least one of the first side (7111), the second side (7112), and the third side (7113). In some embodiments, the bezel antenna (A) may be disposed on at least one of the fourth side (7211), the fifth side (7212), and the sixth side (7213) of the second housing (720).

[0137] According to one embodiment, the state transition from a slide-in state to a slide-out state or from a slide-out state to a slide-in state can be performed automatically. For example, the state transition can be performed through gear engagement between a slide motor including a pinion gear disposed in a first space (7101) of a first housing (710) and a rack gear disposed in a second space (7201) of a second housing (720) and coupled with the pinion gear and the gear. For example, when a processor of the electronic device (700) (e.g., the processor (120) of FIG. 1 ) detects a triggering operation for a state transition, the processor can drive a slide motor disposed inside the electronic device (700). According to one embodiment, the triggering operation can include selecting (e.g., touching) an object displayed on the rollable display (730) or operating a physical button (e.g., a key button) included in the electronic device (700).

[0138] FIG. 8 is a block diagram of an electronic device (800) having a deformable flexible display and a function of responding to user speech, according to one embodiment.

[0139] Referring to FIG. 8, an electronic device (800) (e.g., the electronic device (101) of FIG. 1) may include a wireless communication circuit (810), a front display (or a main display or a second display) (821), a sensing circuit (830), an audio processing circuit (840), a microphone (843), a speaker (846), a state recognition module (851), a voice assistant module (852), a rendering module (853), a memory (888), and a processor (899). According to one embodiment, the wireless communication circuit (810), the front display (821), the sensing circuit (830), the audio processing circuit (840), the memory (888), and the processor (899) may be implemented substantially identically to the wireless communication module (192), the display module (160), the sensor module (176), the audio module (170), the memory (130), and the processor (120) of FIG. 1, respectively, thereby performing the same functions.

[0140] According to one embodiment, when the electronic device (800) has a foldable housing structure, the electronic device (800) may further include a rear display (or, sub-display or first display) (822). For example, in the electronic device (800), the front display (821) may correspond to a display disposed on the front of the electronic device (800) (e.g., the front display (660) of FIG. 6A) and the rear display (822) may correspond to a display disposed on the rear of the electronic device (800) (e.g., the rear display (670) of FIG. 6B).

[0141] According to one embodiment, when the electronic device (800) has a sliderable housing structure, the electronic device (800) may further include a motor (860) and a driving circuit (865) as an actuator that enables movement of a slider (e.g., a second housing (720)) in the sliderable housing structure. The motor (860) may be placed inside the electronic device (800) (e.g., inside the first housing (710) or the second housing (720) in FIGS. 7A to 7D ). The motor driving circuit (865) may drive the motor (860) based on the control of the processor (899), thereby causing the inlet portion of the slider to enter or be withdrawn from the housing (e.g., the first housing (710)). For example, the processor (899) may control the motor drive circuit (865) to perform a state transition in response to a user input for a button for a state transition (e.g., a button positioned on the side of the electronic device (800), a button displayed on the front display (821)). The motor drive circuit (865) may drive the motor (860) under the control of the processor (899) to transition the electronic device (800) from a slide-in state to a slide-out state or vice versa.

[0142] The sensing circuit (830) may include a grip sensor (831) that generates data necessary to determine where the user is holding the electronic device (800).

[0143] According to one embodiment, the grip sensor (831) may be electrically connected to a conductor formed on a side surface of the electronic device (800). For example, the grip sensor (831) may generate data (e.g., data indicating a change in capacitance) necessary to recognize that an external object, such as a user's hand, has contacted the conductor, and output the data to the processor (899). The grip sensor (831) may be disposed in an internal space of the electronic device (800) adjacent to the conductor. For example, when the electronic device (800) has a slidable housing structure (710, 720) and each housing includes a conductor, the grip sensor (831) may include a first grip sensor disposed adjacent to metal included in the first housing (710) and a second grip sensor disposed adjacent to metal included in the second housing (720). As another example, if the electronic device (800) has a multi-foldable housing structure (610, 620, 630) and at least a portion of the first housing (610), at least a portion of the second housing (620), and a portion of the third housing (630) include a conductor, the grip sensor (831) may include a first grip sensor disposed adjacent to metal included in the first housing (610), a second grip sensor disposed adjacent to metal included in the second housing (620), and a third grip sensor disposed adjacent to metal included in the third housing (630). The grip sensor (831) may generate data representing a change in electrostatic capacitance and output the data to the processor (899). For example, the processor (899) may determine a location on the electronic device (800) where a user's hand is in contact based on data received from the grip sensor (831).

[0144] The sensing circuit (830) may include a state sensor (832) that generates data necessary to recognize a state of the electronic device (800). When the electronic device (800) is implemented to have a foldable housing structure (e.g., multi-foldable housing structures (610, 620, 630) in FIGS. 6A to 6J), the state sensor (832) may generate data necessary to recognize an angle between the housings and a state of the foldable housing structure (e.g., unfolded state, intermediate state, folded state). When the electronic device (800) is implemented to have a sliderable housing structure (e.g., sliderable housing structures (710, 720) in FIGS. 7A to 7D), the state sensor (832) may also generate data necessary to recognize a state of the sliderable housing structure (e.g., slide-in state, slide-out state).

[0145] According to one embodiment, data generated by a state sensor (e.g., a gyro sensor and / or an acceleration sensor) (832) may be used to determine a display mode of the electronic device (800). The processor (899) may recognize a posture of the electronic device (800) based on the data received from the state sensor (832), and determine the display mode as a landscape mode or a portrait mode based on the recognized posture. The processor (899) may display visual information on the display based on the determined display mode. For example, a direction in which a first axis of the state sensor (832) (e.g., an x-axis in FIG. 7A) faces may be configured to be substantially the same as a direction in which a first side of a housing structure of the electronic device (800) extends, which is perpendicular to the first axis. A direction in which a second axis of the state sensor (832) (e.g., a y-axis in FIG. 7A) faces, which is perpendicular to the first side, may be configured to be substantially the same as a direction in which a second side of the housing structure extends, which is perpendicular to the first side. When the first axis direction of the state sensor (832) is recognized as being substantially the same as or opposite to the direction of gravity, the processor (899) can determine the display mode as a landscape mode and configure visual information to suit the landscape mode. When the second axis direction of the state sensor (832) is recognized as being substantially the same as or opposite to the direction of gravity, the processor (899) can determine the display mode as a portrait mode and configure visual information to suit the portrait mode.

[0146] In one embodiment, data generated by the state sensor (832) can be used to recognize the state of the housing structure in the electronic device (800).

[0147] When the electronic device (800) has a multi-foldable housing structure (610, 620, 630), for example, the state sensor (832) may include a first hall sensor attached to the first hinge assembly (640) and generating and outputting data corresponding to the angle between the two housings (610, 620), and a second hall sensor attached to the second hinge assembly (650) and generating and outputting data corresponding to the angle between the two housings (620, 630). As another example, the state sensor (832) may include a first inertial sensor disposed within the space inside the first housing (610) to generate data corresponding to a position and / or movement (e.g., angular velocity and / or acceleration along six or nine axes) of the first housing (610), a second inertial sensor disposed within the space inside the second housing (620) to generate data corresponding to the position and / or movement of the second housing (620), and a third inertial sensor disposed within the space inside the third housing (630) to generate data corresponding to the position and / or movement of the third housing (630). At least one of the inertial sensors may be used to determine a display mode.

[0148] When the electronic device (800) has a sliderable housing structure (710, 720), the status sensor (832) may include a sensor (e.g., an encoder, a hall sensor) that is attached to a gear (e.g., a pinion gear) that allows the slider (e.g., the second housing (720) referring to FIG. 7C) to move, and that generates and outputs data corresponding to the rotation angle of the gear when the gear is rotated by a force transmitted from a slide motor or a force transmitted from a user through the slider. According to one embodiment, the status sensor (832) may also include a sensor (e.g., a pressure sensor) that is arranged on a bendable portion of the front display (821) and generates data corresponding to the curvature of the portion.

[0149] The state recognition module (851) can recognize the state of the foldable housing structure using data received from the state sensor (832). For example, the state recognition module (851) can calculate the angle formed between the first housing (610) and the second housing (620) using data received from the state sensor (832), and can recognize the state of the first foldable housing structure (610, 620) based on the data obtained as the calculation result. The state recognition module (851) can calculate the angle formed between the second housing (620) and the third housing (630) using data received from the state sensor (832), and can recognize the state of the second foldable housing structure (620, 630) based on the data obtained as the calculation result. For example, the state recognition module (851) can determine the state of the foldable housing structure as a folded state when the calculated angle is between about 0 degrees and 10 degrees. The state recognition module (851) can determine the state of the foldable housing structure as an unfolded state when the calculated angle is between about 180 degrees. The state recognition module (851) can determine the state of the foldable housing structure as an intermediate state when the angle is between about 10 degrees and 179 degrees.

[0150] The state recognition module (851) can recognize the state of the sliderable housing structure (e.g., the sliderable housing structure (710, 720)) as a slide-in state or a slide-out state based on data received from the state sensor (832).

[0151] A voice assistant module (e.g., intelligent app) (852) can provide a user with a response from an AI agent (or voice secretary) (e.g., intelligent server (300 or 302)) to a query made by the user through voice.

[0152] According to one embodiment, a voice assistant module (e.g., an intelligent app) (852) may receive an audio signal from a microphone (843) via an audio processing circuit (840). Alternatively, the voice assistant module (852) may receive an audio signal from an external device (e.g., a wireless headset) that is wirelessly (e.g., via Bluetooth communication) connected to the electronic device (800) via a wireless communication circuit (810).

[0153] According to one embodiment, the voice assistant module (852) can recognize a call utterance (e.g., “Hi Bixby!” or “Wake up!”) for calling an AI agent (or voice assistant) from an audio signal. For example, the voice assistant module (852) can detect a starting point and an endpoint of a user utterance from an audio signal, thereby obtaining a portion of the user utterance from the audio signal (e.g., a first portion corresponding to “Hi” and a second portion corresponding to “Bixby”). The voice assistant module (852) can determine whether the audio signal includes a call utterance (or an actuation utterance) by comparing the obtained portion of the utterance with pre-stored voice data. According to one embodiment, the voice assistant module (852) can support a user to call an AI agent by a method other than voice. For example, the voice assistant module (852) may recognize an input (e.g., double-pressing) of a physical key (e.g., a power key) of an input device (e.g., the input module (150) of FIG. 1) as a call to the AI ​​agent. As another example, the voice assistant module (852) may also recognize a touch input received from a touch-sensitive display (e.g., the front display (821)) as a call.

[0154] According to one embodiment, the voice assistant module (852) may, in response to an AI agent call, obtain voice input (user speech) from audio data received from a microphone (843) via an audio processing circuit (840).

[0155] According to one embodiment, the voice assistant module (852) may transmit a voice input (user utterance) acquired through a microphone (843) to an intelligent server (e.g., the intelligent server (300) of FIG. 2 or the intelligent server (302) of FIG. 3) via a wireless communication circuit (810). As a response from the intelligent server to the transmission of the user utterance to the intelligent server, the voice assistant module (852) may receive a response to the user utterance (hereinafter, main utterance) (hereinafter, main response) and one or more responses associated with the main response (hereinafter, associated responses) from the intelligent server via the wireless communication circuit (810). The voice assistant module (852) may provide the main response and the associated response to the rendering module (853).

[0156] According to one embodiment, the electronic device (800) may further include an associated utterance agent module (854). As a response from the intelligent server in response to the user utterance being transmitted to the intelligent server, the voice assistant module (852) may receive the goal of the main utterance and the main response from the intelligent server via the wireless communication circuit (810). As another example, the voice assistant module (852) may receive information about an intent including a goal and parameters from the intelligent server via the wireless communication circuit (810). The associated utterance agent module (854) may generate a list of associated utterances including at least one utterance (hereinafter, “associated utterance”) associated with the main utterance based on the goal of the main utterance. The voice assistant module (852) may transmit the list of associated utterances to the intelligent server via the wireless communication circuit (810). As a response from the intelligent server to the transmission of the list of associated utterances to the intelligent server, the voice assistant module (852) can receive an associated response corresponding to each associated utterance in the list of associated utterances from the intelligent server via the wireless communication circuit (810). The voice assistant module (852) can provide the main response and associated response to the rendering module (853).

[0157] According to one embodiment, the electronic device (800) may further include an ASR module (855). The ASR module (855) (e.g., the automatic speech recognition module (251) of FIG. 3) may convert audio data corresponding to a main utterance into text data and return the converted audio data to the voice assistant module (852). The voice assistant module (852) may transmit the user utterance converted into text data to an intelligent server (e.g., the intelligent server (300) of FIG. 2 or the intelligent server (302) of FIG. 3) via a wireless communication circuit (810). As a response from the intelligent server to the transmission of the user utterance to the intelligent server, the voice assistant module (852) may receive a main response and one or more associated responses from the intelligent server via the wireless communication circuit (810). The voice assistant module (852) may provide the main response and the associated responses to the rendering module (853).

[0158] According to one embodiment, the electronic device (800) may further include an associated speech agent module (854) and an ASR module (855). The voice assistant module (852) may transmit a user speech converted into text data through the ASR module (855) to an intelligent server (e.g., the intelligent server (300) of FIG. 2 or the intelligent server (302) of FIG. 3) through a wireless communication circuit (810). As a response from the intelligent server in response to the transmission of the user speech to the intelligent server, the voice assistant module (852) may receive a goal of the main speech and a main response from the intelligent server through the wireless communication circuit (810). The associated speech agent module (854) may generate a list of associated speeches including at least one speech (hereinafter, “associated speeches”) related to the main speech, based on the goal of the main speech. The voice assistant module (852) can transmit a list of associated utterances to the intelligent server via the wireless communication circuit (810). As a response from the intelligent server in response to the transmission of the list of associated utterances to the intelligent server, the voice assistant module (852) can receive an associated response corresponding to each associated utterance in the list of associated utterances from the intelligent server via the wireless communication circuit (810). The voice assistant module (852) can provide the main response and associated response to the rendering module (853).

[0159] The rendering module (853) can render the main response and associated responses into content (in other words, a UI (user interface) screen or view) to be displayed to the user. Hereinafter, the content corresponding to the main response may be referred to as "main content," and the content corresponding to the associated response may be referred to as "associated content."

[0160] According to one embodiment, the rendering module (853) may provide the main content and / or at least one associated content to the user through the display (e.g., the front display (821) and / or the rear display (822)) based on the state of the electronic device (800). For example, if the state recognition module (851) determines that the electronic device (800) is in a folded state, the rendering module (853) may synthesize the main content with other content (e.g., an icon representing a voice assistant, an application icon, a status bar, a widget, and a navigation bar) to create one frame and display the frame on the rear display (822). If the state recognition module (851) determines that the electronic device (800) is in an unfolded state, the rendering module (853) may synthesize the associated content together with the main content with other content to be displayed on the display to create one frame and display the frame on the front display (821). As another example, if the state recognition module (851) determines that the electronic device (800) is in a slide-in state, the main content can be synthesized with other content to create a single frame and the frame can be displayed on the front display (821). If the state recognition module (851) determines that the electronic device (800) is in a slide-out state, the rendering module (853) can synthesize the main content and related content with other content to be displayed on the display to create a single frame and display the frame on the front display (821). Although the state recognition module (851) is illustrated in FIG. 9 as being operatively connected to the rendering module (853), it may also be connected to the rendering module (853) via the voice assistant module (852). For example, the state recognition module (851) can provide information about the state of the electronic device (800) to the rendering module (853) via the voice assistant module (852).

[0161] At least one of the above-described modules (851, 852, 853, 854, 855) may be stored as instructions in the memory of the electronic device (800). Here, the memory storing the instructions may include the memory (888) and / or the internal memory of the processor (899). For example, the instructions may be stored in the internal memory of the memory (888) or the processor (899). As another example, the instructions may be partially stored in the memory (888) and the internal memory of the processor (899). The instructions stored in the memory, when executed by the processor (899), may cause the electronic device (800) to perform given operations. According to one embodiment, the associated utterance agent module (854) and / or the ASR module (855) may be included in an intelligent server instead of the electronic device (800). Hereinafter, given operations of the electronic device (800) according to an embodiment of the present disclosure will be described in detail. In understanding the operations to be described below, reference may be made to FIGS. 1 to 8.

[0162] Fig. 9 is a block diagram of an associated utterance agent module (854) according to one embodiment. The associated utterance agent module (854) may be included in a client device (e.g., an electronic device (800) of Fig. 8) or an intelligent server (e.g., an intelligent server (300) of Fig. 2 or an intelligent server (302) of Fig. 3).

[0163] The voice assistant module (852) can transmit a main utterance (e.g., “Tell me the weather today”) (901) to the natural language platform (950) (e.g., the natural language platform (250 or 320) in FIG. 3). The main utterance (901) transmitted to the natural language platform (950) can include audio data and / or text data. As a response to receiving the main utterance (901) from the voice assistant module (852), the natural language platform (950) can generate a main response (902) (e.g., weather information including today’s lowest temperature, highest temperature, and current temperature based on the user’s location) and a target (903) (e.g., “SHOW WEATHER”) and provide them to the voice assistant module (852).

[0164] The voice assistant module (852) can transmit the target (903) of the main utterance (901) to the associated utterance agent module (854). According to one embodiment, the natural language platform (950) can directly transmit the target (903) to the associated utterance agent module (854). As a response to receiving the target (903) from the voice assistant module (852) or the natural language platform (950), the associated utterance agent module (854) can obtain a list of associated utterances (904) (e.g., “Tell me about fine dust,” “Tell me about tomorrow’s weather,” “Tell me about precipitation”) and provide it to the natural language platform (950). According to one embodiment, the list of associated utterances (904) can be provided from the associated utterance agent module (854) to the natural language platform (950) through the voice assistant module (852).

[0165] According to one embodiment, the associative utterance agent module (854) may include an associative utterance providing module (910), an associative utterance DB (920), a generation engine (930), and a priority manager (940).

[0166] According to one embodiment, the associated utterance DB (920) may include information about associated utterance(s) corresponding to each target in the information about the target. The associated utterance provision module (910) may obtain associated utterance(s) related to the target (903) from the associated utterance DB (920) and provide a list (904) including one or more obtained associated utterances to the natural language platform (950).

[0167] According to one embodiment, the generation engine (930) can generate a list of associated utterances (904) using the target (903). For example, the generation engine (930) can include an artificial intelligence model trained to generate utterances. If an associated utterance related to the target (903) does not exist in the associated utterance DB (920) (or, if information regarding the target (903) does not exist in the associated utterance DB (920), the generation engine (930) can input the target (903) into the model. As a result of the input value (target (903)) being input into the model, a result value can be output from the model. The generation engine (930) can obtain the list of associated utterances (904) from the result value output from the model. The associated utterance providing module (910) can provide the list of associated utterances (904) generated by the generation engine (930) to the natural language platform (950).

[0168] According to one embodiment, the associated utterance agent module (854) may generate an associated utterance list (904) using both the associated utterance DB (920) and the generation engine (930).

[0169] In one embodiment, the associated speech agent module (854) may generate the associated speech list (904) further based on context information about the user. For example, the associated speech agent module (854) may obtain data representing the environment around the user (e.g., speed, humidity, temperature, illuminance) through the sensing circuit (830). As another example, the associated speech agent module (854) may obtain data representing where the user is located (e.g., latitude / longitude and / or administrative district) and / or data representing what is around the user (e.g., restaurants, cafes, gas stations) from an external device through the wireless communication circuit (810). As another example, the associated speech agent module (854) may collect information related to a target (903) from an external device through the wireless communication circuit (810). For example, if the target (903) includes “weather,” the associated speech agent module (854) may collect weather information. As another example, the associated utterance agent module (854) can obtain data related to the user's situation (e.g., information indicating that the user is driving, information indicating the destination) from an application running on the electronic device (800). The associated utterance provision module (910) can obtain associated utterances related to the situation information (obtained data) and / or the target (903) (e.g., “Tell me the weather at the destination!”, “Tell me something special on the moving route!”) from the associated utterance DB (920) and provide the obtained associated utterances to the natural language platform (950) by including them in a list (904). According to one embodiment, the generation engine (930) can input the situation information (obtained data) together with the target (903) into a model and obtain an associated utterance list (904) from a result value output from the model.

[0170] According to one embodiment, the associated utterance agent module (854) may generate a list of associated utterances (904) based on the domain obtained from the main utterance (901). For example, the associated utterance agent module (854) may obtain the domain from the main utterance (901) through the natural language platform (950). For example, the voice assistant module (852) may receive, from the natural language platform (950), another domain (hereinafter, associated domain) (e.g., place (Samseong-dong), date (tomorrow)) associated with the domain (e.g., weather) of the main utterance (901) (e.g., “What is the weather in Samseong-dong tomorrow?”) along with the main response (902) and the target (903) as a response from the natural language platform (950) in response to transmitting the main utterance (901) to the natural language platform (950). The voice assistant module (852) can provide the associated domain together with the target (903) to the associated utterance agent module (854). The associated utterance providing module (910) can obtain associated utterances (e.g., “Find me a good restaurant near Samseong-dong,” “Check my schedule for tomorrow!”) corresponding to the associated domain from the associated utterance DB (920) and provide a list (904) including one or more obtained associated utterances to the natural language platform (950). According to one embodiment, when the associated utterance corresponding to the associated domain does not exist in the associated utterance DB (920), the generation engine (930) can input the associated domain together with the target (903) as input values ​​into the model and obtain the associated utterance list (904) from the result value output from the model. The associated utterance providing module (910) can provide the associated utterance list (904) generated by the generation engine (930) to the natural language platform (950).

[0171] According to one embodiment, the associated utterance agent module (854) may obtain a list of associated utterances (904) using a model (e.g., an NLU model) trained to extract features (e.g., intent, goal, or details related to the intent or goal) from a main utterance and generate associated utterances using the extracted features, and provide the list of associated utterances (904) to the natural language platform (950). For example, the generation engine (930) may input the main utterance (901) (e.g., “Tell me the weather today!”) identified through the voice assistant module (852) into the model, obtain a list of associated utterances (904) from the output value from the model, and provide the list to the natural language platform (950). As another example, the generation engine (930) may input at least two of the obtained input values ​​(e.g., goal, intent, contextual information, and domain) into the model, obtain a list of associated utterances (904) from the output value from the model, and provide the list to the natural language platform (950).

[0172] In one embodiment, the generation engine (930) may utilize a large language model (LLM) or an artificial neural network model generated based on the LLM. In one embodiment, the model used in the generation engine (930) may be trained using different datasets depending on the properties of input and output values. In one embodiment, when the LLM (or LLM-based model) is used to obtain associated utterances, the generation engine (930) may generate different prompts depending on the input values ​​and input them to the LLM.

[0173] According to one embodiment, the priority manager (940) can generate information (905) indicating the priority for each associated utterance by assigning a priority to each associated utterance in the associated utterance list (904).

[0174] According to one embodiment, the priority manager (940) may prioritize related utterances based on collected contextual information (e.g., weather information, location information) related to the user. For example, the priority manager (940) may determine from the collected contextual information that it is raining at the user's current location. Accordingly, the priority manager (940) may assign a relatively higher priority to "Tell me the rainfall amount" than other related utterances. If the collected contextual information determines that the concentration of fine dust is high, the priority manager (940) may assign a relatively higher priority to "Tell me the concentration of fine dust" than other related utterances. If the collected contextual information determines that the user is driving, the priority manager (940) may assign a relatively higher priority to "Tell me the weather at the destination" than other related utterances. If the collected situation information confirms that the user is driving and a traffic accident has occurred along the route, the priority manager (940) can give a relatively higher priority to “Tell me about anything unusual along the route” than other related utterances.

[0175] In one embodiment, the priority manager (940) may prioritize related utterances based on the domain acquired from the main utterance (901). For example, if the acquired domain is related to a location, the priority manager (940) may give a relatively higher priority to "What's the weather in Samseong-dong tomorrow?" than to other related utterances. If the acquired domain is related to a date, the priority manager (940) may give a relatively higher priority to "Check my schedule for tomorrow!" than to other related utterances.

[0176] According to one embodiment, priority information (905) may be acquired by an artificial intelligence model trained to generate associated utterances and assign priorities to the generated associated utterances. For example, the associated utterance agent module (854) may input input values ​​(e.g., main utterance (901), target (903), context information, domain) into the model and may acquire a list of associated utterances (904) and information (905) indicating priorities assigned to each associated utterance from the output values ​​from the model. For example, the artificial intelligence model may output multiple result values, and each result value may correspond to a associated utterance. When outputting multiple result values, the artificial intelligence model may determine a probability (likelihood) or confidence level associated with each result value. The artificial intelligence model may determine a priority based on the probability and / or confidence.

[0177] The priority manager (940) can provide priority information (905) and / or a list of associated utterances (904) to the voice assistant module (852).

[0178] In response to receiving a list of associated utterances (904) from an associated utterance agent module (854) or a voice assistant module (852), the natural language platform (950) may generate one or more associated responses (906) and provide them to the voice assistant module (852).

[0179] The voice assistant module (852) can provide a main response (902), priority information (905), and associated responses (906) to the rendering module (853).

[0180] The rendering module (853) can use the main response (902) and the associated response (906) to generate main content corresponding to the main response (902) and associated content corresponding to the associated response (906) as visual information to be shown to the user. The rendering module (853) can display the generated main content and / or associated content on a display.

[0181] According to one embodiment, the rendering module (853) may receive status information (907) from the status recognition module (851). The rendering module (853) may determine, based on the status information (907), whether a display area on which visual information may be displayed in the electronic device (800) has been enlarged or reduced. Based on the enlargement of the display area, the rendering module (853) may display related content together with the main content on the display. Based on the reduction of the display area, the rendering module (853) may display the main content on the display.

[0182] According to one embodiment, the rendering module (853) may determine which related content to display to the user among related content based on the priority information (905). For example, if the active area displaying visual information on the display is enlarged by approximately 1.5 times, the rendering module (853) may display the first-priority related content on the display along with the main content. If the active area is enlarged by approximately 2 times, the rendering module (853) may display the first-priority related content and the second-priority related content along with the main content.

[0183] FIG. 10 is a flowchart illustrating operations for obtaining associated utterances from a client device according to one embodiment. The client device may be a device having an associated utterance agent module (854), such as the electronic device (101) of FIG. 1 , the first electronic device (201) of FIG. 2 , or the electronic device (800) of FIG. 8 . Instructions stored in a memory of the client device (e.g., the voice assistant module (852), the associated utterance agent module (854), and the ASR module (855)), when executed by a processor of the client device, may cause the client device to perform the operations of FIG. 10 . Any content overlapping with FIGS. 8 and 9 will be briefly described or omitted.

[0184] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0185] In one embodiment, in operation 1010, the client device may acquire a main utterance (e.g., main utterance (901) of FIG. 9). For example, the client device may receive a voice input (main utterance) via a microphone of the client device. As another example, the client device may also receive a voice input from an external device via a wireless communication circuit.

[0186] According to one embodiment, in operation 1020, the client device may obtain a list of associated utterances (e.g., the list of associated utterances (904) of FIG. 9) based on the main utterance.

[0187] For example, a client device can transmit a main utterance to an intelligent server (e.g., an intelligent server (300) of FIG. 2 or an intelligent server (302) of FIG. 3). The client device can transmit the main utterance (audio data) to the intelligent server. Alternatively, the client device can convert the audio data into text data and transmit it to the intelligent server. As a response from the intelligent server to the transmission of the main utterance, the client device can receive a target of the main utterance from the intelligent server. The client device can obtain a related utterance corresponding to the target from a database (e.g., a related utterance database (920) of FIG. 9).

[0188] As another example, a client device may include an AI model trained to generate utterances based at least on the main utterance or its features (e.g., a goal, intent, or details about the goal or intent). The client device may input the goal as the main utterance or its features into the model and obtain a list of related utterances from the output of the model.

[0189] As another example, the client device may obtain a list of associated utterances (e.g., the list of associated utterances (904) of FIG. 9) based on the state of the device. For example, the client device may obtain a list of associated utterances based on whether the device is recognized as being in a slide-out state or an unfolded state.

[0190] As another example, a client device can collect contextual information related to a user using sensing circuits and / or wireless communication circuits. The client device can obtain contextual information and / or associated utterances corresponding to a target from a database (e.g., the associated utterance database (920) of FIG. 9). The client device can input contextual information along with the main utterance or its characteristics into a model and obtain a list of associated utterances from the output values ​​of the model.

[0191] As another example, the client device may receive, as a response from the intelligent server to the transmission of the main utterance, other domains associated with the main domain of the main utterance, along with the target of the main utterance. The client device may obtain contextual information and / or associated utterances corresponding to the associated domain from a database (e.g., the associated utterance database (920) of FIG. 9). The client device may input the associated domains together with the main utterance or its features into a model as input values, and obtain a list of associated utterances from the output values ​​of the model.

[0192] The client device may additionally perform operation 1030 to generate priority information for associated utterances. For example, the client device may generate priority information by prioritizing associated utterances based on collected contextual information. As another example, the client device may generate priority information by prioritizing associated utterances based on acquired associated domains. As another example, the client device may generate priority information by obtaining a value representing the priority assigned to each associated utterance from the output value of the aforementioned model learned for generating associated utterances.

[0193] FIG. 11 is a flowchart illustrating operations for obtaining associated utterances from a server according to one embodiment. Here, the server is a device having an associated utterance agent module (854), such as the server (108) of FIG. 1 , the intelligent server (300) of FIG. 2 , or the intelligent server (302) of FIG. 3 . Instructions stored in the memory of the server (e.g., the associated utterance agent module (854)) when executed by the processor of the server can cause the server to perform the operations of FIG. 11 . Any content overlapping with FIGS. 8 to 10 will be briefly described or omitted.

[0194] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0195] According to one embodiment, in operation 1110, the server may receive a main utterance (voice input) (e.g., the main utterance (901) of FIG. 9) from a client device (e.g., the electronic device (101) of FIG. 1, the first electronic device (201) of FIG. 2, or the electronic device (800) of FIG. 8). The server may receive the main utterance (text data). If the received main utterance is audio data, the server may convert the audio data into text data using an ASR module.

[0196] In one embodiment, at operation 1120, the server may obtain a list of associated utterances based on the main utterance and generate priority information for each associated utterance. For example, operation 1210 may be substantially identical to operations 1020 and 1030 performed by the client device.

[0197] According to one embodiment, at operation 1130, the server may transmit a list of associated utterances and priority information to the client device.

[0198] In one embodiment, operations 1120 and 1130 may be performed based on a request from the client device. For example, if the housing structure of the client device is a foldable or rollable type and the client device transmits information indicating the housing structure to the server along with the main utterance, the server may perform operations 1120 and 1130.

[0199] FIG. 12 is a flowchart illustrating operations for providing a response from an intelligent server to a user's utterance based on a state of an electronic device, according to one embodiment. Instructions stored in a memory of the electronic device (e.g., a state recognition module (851) and a rendering module (853)) may cause the electronic device (e.g., the electronic device (101) of FIG. 1 , the first electronic device (201) of FIG. 2 , or the electronic device (800) of FIG. 8 ) to perform the operations of FIG. 12 when executed by a processor of the electronic device. Contents overlapping with those of FIGS. 8 to 11 are briefly described or omitted.

[0200] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0201] According to one embodiment, in operation 1210, the electronic device may select a display target from among main content and associated content.

[0202] For example, if the electronic device is identified as being in a folded state (e.g., see FIG. 6i), the electronic device may select the main content as the display target. If the electronic device is identified as being in an unfolded state (e.g., see FIG. 6a), the electronic device may select related content as the display target along with the main content.

[0203] As another example, if the electronic device is identified as being in a slide-in state (e.g., see FIG. 7a), the electronic device may select the main content as the display target. If the electronic device is identified as being in a slide-out state (e.g., see FIG. 7c), the electronic device may select related content as the display target along with the main content.

[0204] According to one embodiment, in operation 1220, the electronic device may display a display target on a display.

[0205] For example, the electronic device may display the main content selected as the display target on a first display (e.g., the rear display (670) of FIG. 6 or the rear display (822) of FIG. 8) when the electronic device is confirmed to be in a folded state. The electronic device may display the main content and related content selected as the display target on a second display (e.g., the front display (660) of FIG. 6 or the front display (821) of FIG. 8) when the electronic device is confirmed to be in an unfolded state.

[0206] As another example, the electronic device may display the main content selected as the display target by confirming that the electronic device is in a slide-in state on a display area (e.g., the first portion (730a) of FIG. 7) that is activated in the slide-in state. The electronic device may display the main content selected as the display target and related content by confirming that the electronic device is in a slide-out state on a display area (e.g., the first portion (730a) and the second portion (730b) of FIG. 7) that is activated in the slide-out state.

[0207] FIG. 13 is a flowchart illustrating operations of an electronic device for providing a server response to a user utterance, according to one embodiment. Instructions stored in a memory of the electronic device (e.g., modules 851, 852, 853, 854, and 855 of FIG. 8 ) may, when executed by a processor of the electronic device, cause the electronic device (e.g., the electronic device 101 of FIG. 1 , the first electronic device 201 of FIG. 2 , or the electronic device 800 of FIG. 8 ) to perform the operations of FIG. 13 . Any content overlapping with that of FIGS. 8 to 12 will be briefly described or omitted.

[0208] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0209] In one embodiment, at operation 1310, the electronic device may receive voice input (main speech). For example, the electronic device may receive the voice input via a microphone of the electronic device. In another example, the electronic device may receive the voice input from an external device via a wireless communication circuit.

[0210] According to one embodiment, in operation 1320, the electronic device may transmit a first query generated based on a voice input to a server (e.g., server (108) of FIG. 1, intelligent server (300) of FIG. 2, or intelligent server (302) of FIG. 3). For example, the electronic device may generate the first query by converting the voice input (audio data) into text data using an ASR module.

[0211] According to one embodiment, in operation 1330, the electronic device may display first content corresponding to the server's first response to the first query in the first display area while the electronic device is in the first state. For example, the electronic device may receive the first response as the server's response to the first query. The electronic device may generate the first content using the first response and display it in the first display area.

[0212] An electronic device may be in a first state or a second state. For example, if the electronic device has a foldable housing structure, the first state and the second state may correspond to a folded state and an unfolded state, respectively. As another example, if the electronic device has a slideable housing structure, the first state and the second state may correspond to a slide-in state and a slide-out state, respectively.

[0213] The electronic device may activate a first display area while in a first state. The electronic device may activate a second display area that is wider than the first display area while in a second state. For example, if the electronic device has a foldable housing structure, the first display area may correspond to a display disposed on the rear of the electronic device (e.g., the rear display (670) of FIG. 6B)), and the second display area may correspond to a display disposed on the front of the electronic device and wider than the rear display (e.g., the front display (660) of FIG. 6A). As another example, if the electronic device has a slideable housing structure, the first display area may correspond to a portion of the display that is exposed outside the housing while the electronic device is in a slide-in state (e.g., the first portion (730a) in FIG. 7C). The second display area may correspond to the entirety or a majority of the display (e.g., the first portion (730a) and the second portion (730b) in FIG. 7C) while the electronic device is in a slide-out state.

[0214] According to one embodiment, in operation 1340, the electronic device may display, in a second display area, second content corresponding to a second response from the server to the second query, together with the first content, based on the electronic device changing from the first state to the second state. For example, as a reply to transmitting the first query to the server, the electronic device may receive, together with the first response, additional information related to the first query (e.g., the target of the main utterance). The electronic device may obtain the second query (e.g., the list of related utterances (904)) using the additional information. The electronic device may transmit the second query to the server and receive a second response as a reply from the server. The electronic device may generate second content using the second response and display the second content together with the first content in the second display area.

[0215] In one embodiment, the electronic device may obtain a second query (e.g., a list of related utterances (904)) using additional information while the electronic device is in the first state. Based on the change of the electronic device from the first state to the second state, the electronic device may transmit the second query to the server and receive a second response as a response from the server. The electronic device may generate second content using the second response and display it in the second display area together with the first content.

[0216] In one embodiment, the electronic device, while in the first state, may use additional information to obtain a second query (e.g., a list of related utterances (904)), transmit the second query to a server, and receive a second response as a response from the server. Based on the change of the electronic device from the first state to the second state, the electronic device may use the second response to generate second content and display it in a second display area together with the first content.

[0217] In one embodiment, the electronic device may, while in the first state, use additional information to obtain a second query (e.g., a list of related utterances (904)), transmit the second query to a server, receive a second response as a response from the server, and generate second content using the second response. The electronic device may display the second content in the second display area together with the first content based on the change of the electronic device from the first state to the second state.

[0218] FIG. 14 is a flowchart illustrating operations of an electronic device for providing a server response to a user's utterance based on the electronic device changing from a folded state to an unfolded state, according to one embodiment. The electronic device includes a foldable housing structure including a first housing and a second housing rotatably coupled thereto, a first display area disposed on a rear surface of the electronic device, and a second display area disposed on a front surface of the electronic device. The second display area may be formed to be wider than the first display area and may be implemented as a flexible display panel. For example, the second display area may be configured to fold when the foldable housing structure is folded. Instructions stored in the memory of an electronic device (e.g., modules (851, 852, 853, 854, 855) of FIG. 8) can cause the electronic device (e.g., the electronic device (101) of FIG. 1, the first electronic device (201) of FIG. 2, or the electronic device (800) of FIG. 8) to perform the operations of FIG. 14 when executed by a processor of the electronic device. Contents overlapping with those of FIGS. 8 to 13 are briefly described or omitted.

[0219] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0220] In one embodiment, at operation 1410, the electronic device may receive a voice input (main speech). For example, the electronic device may receive the voice input via a microphone of the electronic device. In another example, the electronic device may receive the voice input from an external device via a wireless communication circuit.

[0221] According to one embodiment, in operation 1420, the electronic device may transmit a first query generated based on a voice input to a server (e.g., server (108) of FIG. 1, intelligent server (300) of FIG. 2, or intelligent server (302) of FIG. 3). For example, the electronic device may generate the first query by converting the voice input (audio data) into text data using an ASR module.

[0222] According to one embodiment, in operation 1430, the electronic device may display first content corresponding to the first response from the server to the first query in the first display area while the electronic device is in a first state (e.g., a folded state). For example, the electronic device may receive the first response as a response from the server to the first query. The electronic device may generate the first content using the first response and display it in the first display area.

[0223] According to one embodiment, in operation 1440, the electronic device may display, in a second display area, second content corresponding to a second response from the server to the second query, together with the first content, based on the electronic device transitioning from the first state to the second state by the angle between the first housing and the second housing exceeding a first threshold angle value (e.g., 120 degrees). For example, as a reply to transmitting the first query to the server, the electronic device may receive, together with the first response, additional information related to the first query (e.g., the target of the main utterance). The electronic device may use the additional information to obtain the second query (e.g., the list of related utterances (904)). The electronic device may transmit the second query to the server and receive, as a reply from the server, a second response. The electronic device may use the second response to generate the second content and display the second content in the second display area together with the first content.

[0224] In one embodiment, the electronic device may obtain a second query (e.g., a list of related utterances (904)) using additional information and / or the first query based on the angle between the first housing and the second housing exceeding a specified second threshold angle value (less than the first threshold angle value) (e.g., 60 degrees). The electronic device may transmit the second query to a server based on the angle between the first housing and the second housing exceeding the first threshold angle value and receive a second response as a response from the server. The electronic device may generate second content using the second response and display the second content in the second display area together with the first content.

[0225] In one embodiment, the electronic device may obtain a second query (e.g., a list of related utterances (904)) using additional information and / or the first query based on whether the angle between the first housing and the second housing exceeds the second threshold angle value, transmit the second query to the server, and receive a second response as the server's response thereto. The electronic device may generate second content using the second response based on whether the angle between the first housing and the second housing exceeds the first threshold angle value, and display the second content in the second display area together with the first content.

[0226] According to one embodiment, the electronic device may obtain a second query (e.g., a list of related utterances (904)) using additional information and / or the first query based on whether the angle between the first housing and the second housing exceeds the second threshold angle value, transmit the second query to a server, receive a second response as a reply from the server, and generate second content using the second response. The electronic device may display the second content together with the first content in the second display area based on whether the angle between the first housing and the second housing exceeds the first threshold angle value.

[0227] FIG. 15 is a flowchart illustrating operations of an electronic device for providing a server response to a user utterance based on the electronic device changing from a slide-in state to a slide-out state, according to one embodiment. The electronic device includes a sliderable housing structure including a first housing and a second housing connected to slide out and in from the first housing, and a display disposed in the sliderable housing structure. At least a portion of the display is configured to be flexibly retractable into the sliderable housing structure. For example, a first portion of the display is exposed to the outside while the second housing is slid out and in with respect to the first housing. A second portion of the display is exposed to the outside while the second housing is slid out with respect to the first housing, and is bent and retracted into the sliderable housing structure while the second housing is slid in. In the following description, the first portion may be referred to as a first display area, and the first portion and the second portion may be referred to as second display areas. Instructions stored in the memory of an electronic device (e.g., modules (851, 852, 853, 854, 855) of FIG. 8) can cause the electronic device (e.g., the electronic device (101) of FIG. 1, the first electronic device (201) of FIG. 2, or the electronic device (800) of FIG. 8) to perform the operations of FIG. 15 when executed by a processor of the electronic device. Contents overlapping with those of FIGS. 8 to 14 are briefly described or omitted.

[0228] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0229] In one embodiment, at operation 1510, the electronic device may receive voice input (main speech). For example, the electronic device may receive the voice input via a microphone of the electronic device. In another example, the electronic device may receive the voice input from an external device via a wireless communication circuit.

[0230] According to one embodiment, in operation 1520, the electronic device may transmit a first query generated based on a voice input to a server (e.g., the server (108) of FIG. 1 , the intelligent server (300) of FIG. 2 , or the intelligent server (302) of FIG. 3 ). For example, the electronic device may generate the first query by converting the voice input (audio data) into text data using an ASR module.

[0231] According to one embodiment, in operation 1530, the electronic device may display first content corresponding to the first response of the server to the first query in the first display area while the electronic device is in the first state (e.g., the slide-in state). For example, the electronic device may receive the first response as a response from the server to the first query. The electronic device may generate the first content using the first response and display it in the first display area.

[0232] According to one embodiment, in operation 1540, the electronic device may display second content corresponding to a second response of the server to the second query in the second display area, together with the first content, based on whether a user input for transitioning from the first state to the second state (e.g., a slide-out state) is received or an activation area for displaying visual information on the display exceeds a first threshold size value (e.g., the activation area expands to cover the entire second portion).

[0233] For example, in response to transmitting a first query to a server, the electronic device may receive, along with the first response, additional information related to the first query (e.g., the target of the main utterance). The electronic device may use the additional information and / or the first query to obtain a second query (e.g., a list of related utterances (904)). The electronic device may transmit a second query to the server and receive a second response as the server's response thereto. The electronic device may use the second response to generate second content and display it in a second display area together with the first content.

[0234] According to one embodiment, the electronic device may receive a user input for transitioning the slide housing structure from a first state to a second state, either from a physical button or a virtual button displayed in the first display area. The electronic device may then drive a motor in response to the user input to extend the second portion, thereby expanding the size of the active area.

[0235] According to one embodiment, the electronic device may obtain a second query (e.g., a list of related utterances (904)) using additional information based on whether a user input for transitioning from a first state to a second state is received or whether a size of an activation area (or a value corresponding to a size of the activation area) exceeds a second threshold size value (smaller than the first threshold size value) (e.g., the activation area expands to half of the second portion). The electronic device may transmit the second query to a server based on whether the size of the activation area exceeds the first threshold size value and receive a second response as a response from the server. The electronic device may generate second content using the second response and display the second content in a second display area together with the first content.

[0236] In one embodiment, the electronic device may obtain a list of associated utterances based on a user input received for transitioning from a first state to a second state. The electronic device may display second content together with the first content in a second display area based on whether the size of the activated area exceeds a first threshold size value or whether the electronic device is identified as being in the second state.

[0237] In one embodiment, the electronic device may obtain a second query (e.g., a list of related utterances (904)) using additional information based on whether the size of the activated area exceeds a second threshold size value, transmit the second query to a server, and receive a second response as a response from the server. The electronic device may generate second content using the second response based on whether the size of the activated area exceeds a first threshold size value that is greater than the second threshold size value, and display the second content together with the first content in the second display area.

[0238] In one embodiment, the electronic device may obtain a second query (e.g., a list of related utterances (904)) using additional information based on whether the size of the activated area exceeds a second threshold size value, transmit the second query to a server, receive a second response as a response from the server, and generate second content using the second response. The electronic device may display the second content together with the first content in the second display area based on whether the size of the activated area exceeds the first threshold size value.

[0239] FIGS. 16A, 16B, 16C, and 16D are diagrams illustrating UI screens that provide a server response to a user utterance, according to one embodiment. The UI screens in FIGS. 16A, 16B, 16C, and 16D may be provided in an electronic device having a sliderable housing structure (e.g., the first housing (710) and the second housing (720) in FIG. 7C). At least a portion of the display of the electronic device is configured to be flexibly retractable into the sliderable housing structure. For example, a first portion of the display is exposed to the outside while the second housing is slid out and in with respect to the first housing. A second portion of the display is exposed to the outside while the second housing is slid out with respect to the first housing, and is bent and retracted into the inside of the sliderable housing structure while the second housing is slid in. When instructions stored in the memory of an electronic device (e.g., modules (851, 852, 853, 854, 855) of FIG. 8) are executed by a processor of the electronic device, the electronic device may perform operations for providing a UI screen according to one embodiment. Contents overlapping with those of FIGS. 8 to 15 are briefly described or omitted.

[0240] Referring to FIG. 16A, the electronic device may display a first UI screen (1610) corresponding to the main utterance while the electronic device is in a slide-in state. For example, the electronic device may transmit a user's voice input (main utterance) such as "Tell me the weather today" to a server and receive first information corresponding to the voice input from the server. As another example, the electronic device may generate first information corresponding to the voice input using a natural language platform (e.g., the natural language platform (250) of FIG. 3) provided in the electronic device. The electronic device may generate (e.g., render) a first UI screen (1610) using the first information and display it on a first portion (1601) of the display. Although not illustrated, the electronic device may display a list of related utterances on the first portion (1601) together with the first UI screen (1610). For example, the electronic device may display a first-ranked related utterance from the list of related utterances on the first portion (1601). The electronic device may display a button (1611) representing a voice assistant together with a first UI screen (1610).

[0241] Referring to FIG. 16B, the electronic device can recognize that the second portion (1602) of the display is exposed to the outside as the second housing (1604) slides out (1605) with respect to the first housing (1603). The electronic device can display the display positions of the first UI screen (1610) and the voice assistant button (1611) from the first portion (1601) to the second portion (1602) so that the first UI screen (1610) and the voice assistant button (1611) are recognized by the user as remaining in the same position until they are fully withdrawn. Alternatively, the electronic device can display the first UI screen (1610) and the voice assistant button (1611) by moving them in the direction in which the second housing (1604) slides out (1605) with respect to the first housing (1603).

[0242] Referring to FIG. 16C, the electronic device may display a second UI screen (1620) corresponding to the associated utterance on the display together with the first UI screen (1610) based on the electronic device transitioning to a slide-out state (e.g., the entire second portion (1602) is exposed to the outside). For example, while the first UI screen (1610) is displayed, when the state transition begins or when the state transition is completed, the electronic device may transmit a list of associated utterances to the server and receive second information corresponding to the list of associated utterances from the server. The electronic device may also generate the second information using an intelligent platform. The electronic device may generate (e.g., render) the second UI screen (1620) using the second information. For example, the electronic device may generate the second UI screen using a portion corresponding to the first-ranked associated utterance in the second information. Based on the completion of the state transition, the electronic device may display the second UI screen (1620) together with the first UI screen (1610). When the state transition is completed while the electronic device is displaying the first UI screen (1610) and the voice secretary button (1611), the electronic device may display the second UI screen (1620) together with the first UI screen (1610) and the voice secretary button (1611). The electronic device may display the voice secretary button (1611) by moving the display position. For example, the electronic device may display the voice secretary button (1611) by moving it from below the first UI screen (1610) (see FIG. 16b) to below the second UI screen (1620) (see FIG. 16c). According to one embodiment, the electronic device may reduce the size of the first UI screen (1610) or omit some of the contents of the first UI screen (1610) so that the first UI screen (1610) and the second UI screen (1620) are displayed simultaneously without overlapping. Although not shown, the electronic device may display the second-order associated utterance on the display along with the UI screens (1610, 1620).When a second-order related utterance is selected by the user, the electronic device can use the information to generate a third UI screen and display the third UI screen on the display, together with the first UI screen (1610), instead of the second UI screen (1620).

[0243] Referring to FIG. 16D, the electronic device may display a second UI screen (1620) on the display together with the first UI screen (1610) based on the start of a state transition and the external exposure of a designated size (e.g., half) of the second portion (1602). For example, while the first UI screen (1610) is displayed, if the state transition starts, the electronic device may transmit a list of associated utterances to the server and receive second information corresponding to the list of associated utterances from the server or generate the second information using an intelligent platform. The electronic device may generate (e.g., render) the second UI screen (1620) using the second information. For example, the electronic device may generate the second UI screen using a portion corresponding to the first-ranked associated utterance in the second information. When the designated size (e.g., half) of the second portion (1602) is externally exposed, the electronic device may display the second UI screen (1620) together with the first UI screen (1610). According to one embodiment, the electronic device may reduce the size of the first UI screen (1610) or omit some of the contents of the first UI screen (1610) so that the first UI screen (1610) and the second UI screen (1620) are displayed simultaneously without overlapping. Although not shown, the electronic device may display the second-priority related utterance on the display together with the UI screens (1610, 1620). When the second part (1602) is fully exposed, the electronic device may display the third UI screen corresponding to the second-priority related utterance on the display together with the first UI screen (1610) and the second UI screen (1620). The electronic device may display the third-priority related utterance on the display together with the UI screens. When the third-priority related utterance is selected by the user, the electronic device may use the information to generate a fourth UI screen and display the fourth UI screen on the display instead of another UI screen (e.g., the third UI screen) together with the other UI screens.

[0244] When transitioning from the state of FIG. 16c or FIG. 16d to the state of FIG. 16a, the electronic device can display only the first UI screen (1610) in the first part (1601).

[0245] FIGS. 17A, 17B, and 17C are diagrams illustrating UI screens that provide a server response to a user utterance based on an enlargement of a display area according to one embodiment. The UI screens in FIGS. 17A, 17B, and 17C may be provided in an electronic device having an infolding multi-foldable housing structure (e.g., the first housing (610), the second housing (620), and the third housing (630) in FIG. 6A). The electronic device includes a first display disposed on the rear (e.g., the rear display (670) or auxiliary display in FIG. 6B) and a second display disposed on the front (e.g., the front display (660) or main display in FIG. 6A). The second display may include a flexible display panel. For example, the second display may be configured to fold when the foldable housing structure is folded. When instructions stored in the memory of an electronic device (e.g., modules (851, 852, 853, 854, 855) of FIG. 8) are executed by a processor of the electronic device, the electronic device may perform operations for providing a UI screen according to one embodiment. Contents overlapping with those of FIGS. 8 to 16 are briefly described or omitted.

[0246] Referring to FIG. 17A, the electronic device may display a first UI screen (1710) corresponding to a main utterance on the first display (1702) while the electronic device is in a folded state. For example, the electronic device may generate (e.g., render) the first UI screen (1710) using first information corresponding to a voice input (main utterance) and display the first UI screen (1710) on the first display (1702). Although not illustrated, the electronic device may display a list of related utterances on the first display (1702) together with the first UI screen (1710). For example, the electronic device may display a first-ranked related utterance in the list of related utterances on the first display (1702).

[0247] Referring to FIG. 17B, the electronic device may display a second UI screen (1720) corresponding to the first associated utterance on the unfolded portion of the second display (e.g., the first display area (1701a) and the second display area (1701b)) together with the first UI screen (1710) while the electronic device is in a partially unfolded state (e.g., the first housing (1721) and the second housing (1722) are unfolded). For example, while the first UI screen (1710) is displayed, when a state transition begins or when a transition to a partially unfolded state is completed, the electronic device may transmit a list of associated utterances to a server and receive second information corresponding to the list of associated utterances from the server. The electronic device may also generate the second information using an intelligent platform. The electronic device may generate the second UI screen (1720) using the second information. For example, the electronic device may generate a second UI screen (1720) using a portion corresponding to a first associated utterance (e.g., a first-priority associated utterance) in the second information. Based on the completion of the state transition, the electronic device may display the second UI screen (1720) together with the first UI screen (1710) on the expanded portion of the second display. According to one embodiment, the electronic device may reduce the size of the first UI screen (1710) or omit some of the contents of the first UI screen (1710) so that the first UI screen (1710) and the second UI screen (1720) are displayed simultaneously without overlapping. Although not shown, the electronic device may also display the second associated utterance (e.g., a second-priority associated utterance) together with the UI screens (1710, 1720) on the expanded portion of the second display. When the second associated utterance is selected by the user, the electronic device can use the information to generate a third UI screen and display the third UI screen on the display, together with the first UI screen (1710), instead of the second UI screen (1720).The electronic device may change the display position of the voice assistant button (1711) based on the addition of the second UI screen (1720) as a display target. For example, the electronic device may display the voice assistant button (1711) from below the first UI screen (1710) (see FIG. 17a) to below the central portion of the first UI screen (1710) and the second UI screen (1720) (see FIG. 17b).

[0248] Referring to FIG. 17c, the electronic device may display a third UI screen (1730) corresponding to the second associated utterance on the second displays (1701a, 1701b, 1701c) together with the first UI screen (1710) and the second UI screen (1720) while all of the housings (1721, 1722, 1723) are unfolded. For example, while the first UI screen (1710) and the second UI screen (1720) are displayed, the electronic device may generate the third UI screen (1730) using a portion of the second information corresponding to the second associated utterance. Based on the completion of the state transition, the electronic device may display a third UI screen (1730) on the second displays (1701a, 1701b, 1701c) together with the first UI screen (1710) and the second UI screen (1720). According to one embodiment, the electronic device may reduce the size of the first UI screen (1710) and / or the second UI screen (1720) or display some of the contents thereof by omitting them so that the UI screens (1710, 1720, 1730) are displayed simultaneously without overlapping. Although not shown, the electronic device may also display a third associated utterance (e.g., a third-ranked associated utterance) on the second displays (1701a, 1701b, 1701c) together with the UI screens (1710, 1720, 1730). When the third associated utterance is selected by the user, the electronic device can use the information to generate a fourth UI screen and display the fourth UI screen on the display instead of the third UI screen (1730), together with the first UI screen (1710) and the second UI screen (1720). The electronic device can change the display position of the voice assistant button (1711) based on the addition of the third UI screen (1730) as a display target. For example, the electronic device can display the voice assistant button (1711) from below the central portion of the first UI screen (1710) and the second UI screen (1720) (see FIG. 17b) to below the second UI screen (1720) (see FIG. 17c).

[0249] According to one embodiment, an electronic device (e.g., electronic device (800) of FIG. 8) includes a transformable housing; a communication circuit housed within the housing; a microphone housed within the housing; a display housed within the housing and at least partially deformable such that the electronic device has a first state or a second state according to deformation of the housing; and a processor housed within the housing (e.g., processor (899)). A first display area of ​​the display may be configured to be activated while the electronic device is in the first state. A second display area of ​​the display may be configured to be activated while the electronic device is in the second state. The second display area is wider than the first display area. The processor may be configured to receive a voice input from a user via the microphone. The processor may be configured to transmit a first query including a first text generated based on the voice input to a server via the communication circuit. The processor may be configured to obtain a first response to the first query from the server. The processor may be configured to display first content corresponding to the first response through the first display area while the electronic device is in the first state. The processor may be configured to transmit a second query including second text different from the first text to the server through the communication circuit based at least in part on the electronic device changing from the first state to the second state. The processor may be configured to obtain a second response to the second query from the server. The processor may be configured to display second content corresponding to the second response through the second display area simultaneously with the first content while the electronic device is in the second state.

[0250] The processor may be configured to obtain the second query based at least in part on additional information generated from the server based on the first query.

[0251] The processor may be configured to obtain the second query based further on contextual information about the user.

[0252] The processor may be configured to perform an operation of generating the second query. Accordingly, the second query may include a first additional query having a first priority and a second additional query having a second priority lower than the first priority.

[0253] The processor may be configured to store the second query in association with at least one of the first query or additional information generated from the server based on the first query (e.g., the target of the main utterance) prior to transmitting the second query.

[0254] The housing may include a first housing; a second housing; and a hinge assembly rotatably connecting the first housing and the second housing. The display may include a first display (e.g., a rear display (822)) accommodated in at least one of the first housing or the second housing and disposed as visible in a first direction; and a second display (e.g., a front display (821)) accommodated in the first housing and the second housing and disposed as visible in a second direction opposite to the first direction, and foldable according to folding of the housing. The first display may be activated as at least a portion of the first display area while the electronic device is folded. The second display may be activated as at least a portion of the second display area while the electronic device is unfolded.

[0255] The electronic device may further include a sensor configured to generate information regarding an angle between the first housing and the second housing. The processor may be configured to perform an operation of transmitting the second query based at least in part on the angle reaching a first threshold. The processor may be configured to perform an operation of activating the second display area based on the angle reaching a second threshold different from the first threshold. The second threshold may be greater than the first threshold.

[0256] The processor may be configured to display an indicator in the second display area, simultaneously with the first response, indicating that the second response is being generated, after the second display area is activated and before the second content is displayed through the second display area.

[0257] The housing may include a first housing; and a second housing connected to slide out and in from one side of the first housing. The first display area (e.g., the first portion (1601) of FIG. 16A) may be arranged to be visible in a first direction while the second housing is slid out and in with respect to the first housing. The second display area (e.g., the second portion (1602) of FIG. 16B) may be arranged to be visible in the first direction while the second housing is slid out with respect to the first housing. The second display area may be bent toward the second direction while slid in and may be hidden by at least one of the first housing or the second housing.

[0258] The electronic device may further include an actuator (e.g., a motor (860) and a driving circuit (865) in FIG. 8) that enables a slide-out or slide-in of the second housing based on a user input. The processor may be configured to perform the operation of transmitting the second query based on the user input requesting a slide-out of the second housing. While the first display area remains as activated after the second housing is fully slid out with respect to the first housing, the processor may be configured to perform an operation of activating the second display area.

[0259] In one embodiment, a method is performed in an electronic device having a deformable housing. The method may include an operation of receiving a user's voice input through a microphone of the electronic device (e.g., operation 1310). The method may include an operation of transmitting a first query including a first text generated based on the voice input to a server through a communication circuit of the electronic device (e.g., operation 1320). The method may perform an operation of obtaining a first response to the first query from the server. The method may include an operation of displaying first content corresponding to the first response through a first display area while the electronic device is in a first state (e.g., operation 1330). The method may include an operation of transmitting a second query including a second text different from the first text to the server through the communication circuit based at least in part on the electronic device changing from the first state to a second state due to deformation of the housing. The method may include an operation of obtaining a second response to the second query from the server. The method may include an operation (e.g., operation 1340) of displaying, while the electronic device is in the second state, second content corresponding to the second response through a second display area that is wider than the first display area, simultaneously with the first content.

[0260] The method may include obtaining the second query based at least in part on additional information generated from the server based on the first query.

[0261] The act of obtaining the second query may include an act of generating the second query based further on contextual information about the user.

[0262] The operation of obtaining the second query may include an operation of generating the second query to include a first additional query having a first priority and a second additional query having a second priority lower than the first priority.

[0263] According to one embodiment, an electronic device stores instructions executable by at least one processor. The instructions, when executed by the processor, may cause the electronic device to receive a user's voice input through a microphone and transmit a first query including a first text generated based on the voice input to a server through a communication circuit. The instructions, when executed by the processor, may cause the electronic device to obtain a first response to the first query from the server and, while the electronic device is in the first state, display first content corresponding to the first response through the first display area. The instructions, when executed by the processor, may cause the electronic device to transmit a second query including a second text different from the first text to the server through the communication circuit, at least in part based on a state of the electronic device changing from the first state to the second state. The instructions, when executed by the processor, may cause the electronic device to obtain a second response to the second query from the server. The above command, when executed by the processor, may cause the electronic device to display second content corresponding to the second response through the second display area simultaneously with the first content while the electronic device is in the second state.

[0264] In the above explanation, the prefixes “first,” “second,” and “third” are only used to distinguish between the same names and do not have any special meaning in themselves, such as importance or order.

[0265] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0266] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0267] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. In one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0268] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0269] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0270] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, Deformable housing; communication circuit; mike; A display at least partially deformable such that the electronic device has a first state or a second state depending on deformation of the housing; processor; and Contains memory that stores instructions, A first display area in the above display is configured to be activated when the electronic device is in the first state, A second display area wider than the first display area in the above display is configured to be activated when the electronic device is in the second state, The above instructions, when executed by the processor, cause the electronic device to: Receives user's voice input through the above microphone; Transmitting a first query including a first text generated based on the voice input to a server through the communication circuit; Obtain a first response to the first query from the server; While the electronic device is in the first state, displaying first content corresponding to the first response through the first display area; At least in part based on the electronic device changing from the first state to the second state, transmitting a second query including a second text different from the first text to the server through the communication circuit; Obtaining a second response to the second query from the server; and An electronic device that displays second content corresponding to the second response through the second display area simultaneously with the first content while the electronic device is in the second state.

2. In the first paragraph, when the command is executed by the processor, the electronic device, An electronic device that obtains the second query based at least in part on additional information generated by the server using the first query.

3. In the second paragraph, when the command is executed by the processor, the electronic device, An electronic device that obtains the second query based further on contextual information about the user.

4. In the second paragraph, when the command is executed by the processor, the electronic device, An electronic device that generates the second query to include a first additional query having a first priority and a second additional query having a second priority lower than the first priority.

5. In the first paragraph, when the command is executed by the processor, the electronic device, An electronic device that stores the second query in association with at least one of the first query or additional information generated from the server based on the first query prior to the operation of transmitting the second query.

6. In paragraph 1, The above housing, 1st housing; Second housing; and A hinge assembly is included that rotatably connects the first housing and the second housing, The above display is, A first display accommodated in at least one of the first housing or the second housing and arranged to be viewed in a first direction; and A second display is accommodated in the first housing and the second housing, is arranged to be viewed in a second direction opposite to the first direction, and is foldable according to folding of the housing. The first display is configured to be activated as at least a portion of the first display area while the electronic device is folded, The second display is configured to be activated as at least a portion of the second display area while the electronic device is unfolded. Electronic devices.

7. In paragraph 6, Further comprising a sensor configured to generate information regarding an angle between the first housing and the second housing; The above instructions, when executed by the processor, cause the electronic device to: performing an operation of transmitting the second query based at least in part on the angle reaching the first threshold; and An electronic device that performs an operation of activating the second display area based on the angle reaching a second threshold value different from the first threshold value.

8. In the 7th paragraph, when the command is executed by the processor, the electronic device, An electronic device that displays an indicator indicating that the second response is being generated through the second display area, simultaneously with the first response, after the second display area is activated and before the second content is displayed through the second display area.

9. In paragraph 7, An electronic device wherein the second threshold value is greater than the first threshold value.

10. In paragraph 1, The above housing, First housing; and A second housing is included that is connected to slide out and slide in from one side of the first housing, The first display area is arranged to be visible in a first direction while the second housing is slid-out and slid-in with respect to the first housing, An electronic device wherein the second display area is arranged to be visible in the first direction while the second housing is slid out relative to the first housing, and to be bent toward the second direction while the second housing is slid in and to be hidden by at least one of the first housing or the second housing.

11. In paragraph 10, Further comprising an actuator that enables the second housing to slide out or slide in based on user input; The above instructions, when executed by the processor, cause the electronic device to: An operation of transmitting the second query based on a request for slide-out of the second housing by the user input; and An electronic device that performs an operation of activating the second display area while the first display area remains activated after the second housing is fully slid out with respect to the first housing.

12. A method for operating an electronic device having a deformable housing, An action of receiving a user's voice input through a microphone of the electronic device; An operation of transmitting a first query including a first text generated based on the voice input to a server via a communication circuit of the electronic device; An operation of obtaining a first response to the first query from the server; An operation of displaying first content corresponding to the first response through a first display area while the electronic device is in a first state; An operation of transmitting a second query including a second text different from the first text to the server through the communication circuit based at least in part on the electronic device changing from the first state to the second state according to deformation of the housing; An operation of obtaining a second response to the second query from the server; and A method comprising an operation of displaying, while the electronic device is in the second state, second content corresponding to the second response through a second display area wider than the first display area, simultaneously with the first content.

13. In paragraph 12, A method further comprising obtaining the second query based at least in part on additional information generated from the server based on the first query.

14. In the 13th paragraph, the operation of obtaining the second query is as follows: A method comprising generating said second query based further on contextual information about said user.

15. In the 13th paragraph, the operation of obtaining the second query is as follows: A method comprising generating a second query including a first additional query having a first priority and a second additional query having a second priority lower than the first priority.

Images