Electronic device for controlling audio output according to display state, operating method thereof, and storage medium

Abstract

According to an embodiment, an electronic device (101) may comprise: displays (160, 560); one or more sensors (176, 576); an audio module (570); one or more processors (120, 520); and a memory (530) for storing instructions. According to an embodiment, the instructions may be configured to, when executed by the one or more processors, cause the electronic device to identify the state of the displays by using the sensors in response to the execution of an application. According to an embodiment, the instructions may be configured to cause the electronic device to, on the basis of identifying that the state of the displays is a first state, generate a first stereoscopic audio signal from an audio signal by using an audio processing unit (521) set to a first gain and output the generated first stereoscopic audio signal via the audio module. According to an embodiment, the instructions may be configured to cause the electronic device to change the gain of the audio processing unit from the first gain to a second gain on the basis of identifying that the state of the displays has changed from the first state to a second state. According to an embodiment, the instructions may be configured to cause the electronic device to generate a second stereoscopic audio signal from the audio signal by using the audio processing unit set to the second gain and output the generated second stereoscopic audio signal via the audio module.

Description

Electronic device for controlling audio output according to display status, method of operation thereof and storage medium

[0001] Embodiments disclosed in this document relate to an electronic device for controlling audio output according to a display state, a method of operating the same, and a storage medium.

[0002] Advances in electronic technology are driving the development of various types of flexible electronic devices. These flexible electronic devices can offer larger displays while maintaining portability. For example, flexible electronic devices can be foldable, bendable, slidable, or rollable, allowing users to modify their shapes. For example, displays using organic light-emitting diodes (OLEDs) (or electronic devices equipped with them) can operate stably even when manufactured in a very thin form factor.

[0003] Furthermore, user demands for the performance of electronic devices are becoming increasingly sophisticated and diverse, and audio performance can be a key indicator of product performance. Current electronic devices typically incorporate multiple speakers and utilize them to deliver audio output.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0005] According to one embodiment, an electronic device includes a display, at least one sensor, an audio device, at least one processor, and a memory storing instructions.

[0006] According to one embodiment, the instructions, when executed by the at least one processor, may be configured to cause the electronic device to identify a state of the display by using the sensor in response to execution of an application.

[0007] According to one embodiment, the instructions may be configured to cause the electronic device to generate a first stereoscopic audio signal from an audio signal using an audio processing unit set to a first gain based on identifying that the state of the display is a first state, and to output the generated first stereoscopic audio signal through the audio device.

[0008] In one embodiment, the instructions may be configured to cause the electronic device to change the gain of the audio processing unit from the first gain to the second gain based on identifying that the state of the display has changed from the first state to the second state.

[0009] According to one embodiment, the instructions may be configured to cause the electronic device to generate a second stereoscopic audio signal from the audio signal using the audio processing unit set to the second gain, and to output the generated second stereoscopic audio signal through the audio device or audio module.

[0010] According to one embodiment, a method for controlling audio output according to a display state in an electronic device includes an operation of identifying a state of a display of the electronic device by using at least one sensor of the electronic device in response to execution of an application.

[0011] According to one embodiment, the method includes generating a first stereoscopic audio signal from an audio signal using an audio processing unit set to a first gain based on identifying that the state of the display is a first state, and outputting the generated first stereoscopic audio signal through an audio device.

[0012] According to one embodiment, the method includes an operation of changing a gain of the audio processing unit from the first gain to the second gain based on identifying that the state of the display has changed from the first state to the second state.

[0013] According to one embodiment, the method includes generating a second stereoscopic audio signal from the audio signal using the audio processing unit set to the second gain, and outputting the generated second stereoscopic audio signal through the audio device.

[0014] According to one embodiment, a storage medium storing at least one computer-readable instruction, wherein the at least one instruction, when executed by at least one processor of an electronic device, causes the electronic device to perform at least one operation, the at least one operation including an operation of identifying a state of a display of the electronic device by using at least one sensor of the electronic device in response to execution of an application.

[0015] According to one embodiment, the at least one operation includes generating a first stereoscopic audio signal from an audio signal using an audio processing unit set to a first gain based on identifying that the state of the display is a first state, and outputting the generated first stereoscopic audio signal through an audio device.

[0016] In one embodiment, the at least one operation includes changing a gain of the audio processing unit from the first gain to the second gain based on identifying that the state of the display has changed from the first state to the second state.

[0017] According to one embodiment, the at least one operation includes generating a second stereoscopic audio signal from the audio signal using the audio processing unit set to the second gain, and outputting the generated second stereoscopic audio signal through the audio device.

[0018] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0019] FIG. 2 is a diagram illustrating an unfolded state and a folded state of an electronic device of a first type according to one embodiment.

[0020] FIG. 3A is an exemplary diagram showing the front and back of an electronic device in an unfolded state of a second type of electronic device according to one embodiment.

[0021] Fig. 3b is an exemplary diagram showing a folded state of the electronic device of Fig. 3a.

[0022] FIG. 4 is an exemplary diagram showing a slide-in state and a slide-out state of an electronic device of a third type according to one embodiment.

[0023] Figure 5 is an internal block diagram of an electronic device according to one embodiment.

[0024] Figure 6 is a block diagram for generating a stereoscopic audio signal according to one embodiment.

[0025] FIG. 7 is a flowchart illustrating an operation for controlling audio output according to a display state in an electronic device according to an embodiment.

[0026] Figure 8 is an exemplary diagram for explaining a head transfer function according to one embodiment.

[0027] Figure 9 is an exemplary diagram illustrating an artificial reverberator according to one embodiment.

[0028] Fig. 10 is an exemplary diagram illustrating a crosstalk remover according to one embodiment.

[0029] Fig. 11 is a flowchart of operations in an audio processing unit according to a display state according to one embodiment.

[0030] FIG. 12 is an exemplary diagram illustrating the characteristics of a stereoscopic audio signal output according to a display state centered on a user according to one embodiment.

[0031] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0032] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0033] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100), according to one embodiment. Referring to FIG. 1 , in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0034] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0035] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0036] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0037] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0038] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0039] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0040] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0041] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0042] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0043] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0044] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0045] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0046] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0047] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0048] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0049] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0050] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0051] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0052] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0053] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0054] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0055] In the detailed description below, reference numerals in the drawings may be used interchangeably or omitted for components that can be easily understood through the preceding embodiments, and their detailed descriptions may also be omitted. An electronic device according to an embodiment disclosed in this document may be implemented by selectively combining components of different embodiments, and components of one embodiment may be replaced by components of another embodiment. For example, it should be noted that the present invention is not limited to specific drawings or embodiments.

[0056] FIG. 2 is a drawing showing an unfolded state and a folded state of an electronic device (101) of a first type according to one embodiment.

[0057] Referring to FIG. 2, in one embodiment, the electronic device (101) may include a foldable housing and a foldable display (260) disposed within a space formed by the foldable housing. The foldable housing may have a shape that is substantially symmetrical around a folding axis (e.g., A-axis). According to one embodiment, a surface on which the first display (260) is disposed may be defined as a first surface (210a) of the electronic device (101) (or a front surface of the electronic device (101), and a surface opposite to the first surface (210a) may be defined as a second surface (220a) (or a back surface of the electronic device (101).

[0058] Referring to FIG. 2, the electronic device (101) may have a first form in which the vertical length is longer than the horizontal length based on the front surface. For example, the first form of the electronic device (101) may be referred to as a fold electronic device, and a state in which the lower sides of the main body of the first form of the electronic device (101) face downwards based on the folding axis (e.g., the A-axis) may be referred to as a portrait mode. In addition, a state in which one of the two sides of the main body of the first form of the electronic device (101) faces downwards based on the folding axis (e.g., the A-axis) may be referred to as a landscape mode.

[0059] As illustrated in FIG. 2(a), the first display (260) may be formed to occupy substantially the entire first surface (210a) of the electronic device (101). As illustrated in FIG. 2(b), the second display (261) may be formed to occupy at least a portion of the second surface (220a) or substantially the entire second surface (220a). In this case, the first display (260) may pivot via a separate hinge module, and the second display (261) may be fixed to the housing. For example, the hinge structure may be configured to fold or unfold inwardly or outwardly. For example, a free stop hinge may maintain the folded state of the electronic device (101) at various angles.

[0060] For example, a second display (261) may be arranged on one of a pair of housings arranged on either side of a folding axis (e.g., A-axis). According to one embodiment, the first display (260) may mean a flexible display in which at least a portion of the housing may be transformed into a flat or curved surface. The first display (260) may include a first area (or first display area) (260a) arranged on one side of the folding axis (e.g., A-axis) and a second area (or second display area) (260b) arranged on the other side.

[0061] For example, when the electronic device (101) is in an unfolded state (e.g., a flat state or an unfolded state), the surface of the first region (260a) and the surface of the second region (260b) form an angle of about 180 degrees with each other and may face the same direction (e.g., the front direction of the electronic device (101)).

[0062] As illustrated in FIG. 2(c), when the electronic device (101) is in a folded state, the surface of the first region (260a) of the first display (260) and the surface of the second region (260b) form a narrow angle (e.g., between 0 and 10 degrees) with each other and may face each other. According to one embodiment, when the electronic device (101) is in a folded state, the second display (261) may be disposed in one of a pair of housings that are disposed on either side of the folding axis (e.g., the A-axis), but this is exemplary, and the second display (261) may form most of the rear surface (220a) depending on the structure or function. For example, the electronic device (101) may include the second display (261) of which at least a portion is visually exposed through the rear cover. Therefore, it should be noted that the sizes and shapes of the first display (260) and the second display (261) are not limited thereto. In addition, the first display (260) area division shown in FIG. 2(a) is exemplary, and may be divided into multiple areas (e.g., 2, 4, or more than 4) depending on the structure or function.

[0063] According to one embodiment, the foldable housing of the electronic device (101) may include an opening for an audio module (e.g., a speaker) or an audio device as indicated by S1, S2. For example, the audio module may be arranged on the upper portion (210) and / or the lower portion (220) of the electronic device (101) as illustrated in FIG. 2, but the arrangement location of the audio module is not limited thereto.

[0064] FIG. 3a is an exemplary diagram showing the front and back of an electronic device in an unfolded state of a second type of electronic device according to one embodiment, and FIG. 3b is an exemplary diagram showing the electronic device of FIG. 3a in a folded state.

[0065] As illustrated in FIGS. 3A and 3B, the foldable housing may include a first housing structure (or structure) (310) and a second housing structure (320). The foldable housing of the electronic device (101) is not limited to the shape and mounting illustrated in FIGS. 3A and 3B, and may be implemented by other shapes or combinations of parts and / or mounting. According to one embodiment, the first housing structure (310) and the second housing structure (320) may be arranged on both sides with respect to the folding axis (A-A') and may have a shape that is substantially symmetrical with respect to the folding axis (A-A'). The angle or distance between the first housing structure (310) and the second housing structure (320) may vary depending on whether the electronic device (101) is in an unfolded state, a folded state, or a partially unfolded (or partially folded) intermediate state.

[0066] According to one embodiment, when the electronic device (101) is in an unfolded state, the first housing structure (310) and the second housing structure (320) may be arranged to form an angle of about 180 degrees and face substantially in the same direction. In the unfolded state, the surface of the first display area (301) and the surface of the second display area (302) of the display (300) may form an angle of about 180 degrees with each other and face in the same direction (e.g., toward the front of the electronic device (101).

[0067] In one embodiment, when the electronic device (101) is in a partially folded state, the first housing structure (310) and the second housing structure (320) may form an angle of about 90 degrees to about 180 degrees.

[0068] According to one embodiment, a sub-display (360) may be arranged on at least a portion of the rear surface (310a, 320a) of the electronic device (101). According to one embodiment, in a folded state, the electronic device (101) may provide content through the sub-display (360). For example, the content may include a standby screen or an application execution screen.

[0069] Based on the unfolded state, the display (300) arranged on the front of the electronic device (101) may be referred to as a front display (or main display), and the sub-display (360) arranged on the rear may be referred to as a rear display (or sub-display). According to one embodiment, the first housing structure (310) and the second housing structure (320) may include openings for an audio module (e.g., a speaker) or an audio device as indicated by S1 and S2. For example, the audio module may be arranged on the upper portion (315) and / or the lower portion (325) of the electronic device (101).

[0070] In this way, the electronic device (101) including the foldable display (or flexible display) can be folded (folded or bent) based on one axis. Here, one axis may be preset or may be arbitrary. A preset axis may mean that only a specific area (e.g., a part including the axis) of the foldable display of the electronic device (101) is bendable. On the other hand, a random axis may mean that the entire area of ​​the display of the electronic device is bendable. Although FIGS. 3A and 3B illustrate folding in half based on an axis passing through the center of the electronic device, those skilled in the art will readily understand that there is no limitation on the position of the axis.

[0071] FIG. 4 is an exemplary diagram showing a slide-in state and a slide-out state of an electronic device of a third type according to one embodiment.

[0072] Referring to FIG. 4, the electronic device (101) may include a housing (420) and a display positioned within a space formed by the housing (420). At least a portion of the display may have a slidable or rollable form.

[0073] FIG. 4(a) shows a state in which a part of the display (e.g., a first area (A1)) is exposed to the outside of the housing (420), and FIG. 4(b) shows a state in which another part of the display (e.g., a second area (A2)) is housed within the housing (420) and then completely exposed to the outside of the housing. The state illustrated in FIG. 4(a) may be referred to as a slide-in state of the electronic device (101), or a state in which the second area (A2) of the display is closed. The state illustrated in FIG. 4(b) may be referred to as a slide-out state of the electronic device (101), or a state in which the second area (A2) of the display is open.

[0074] According to one embodiment, the electronic device (101) may have an intermediate state between the slide-in state (e.g., a fully closed state) of FIG. 4(a) and the slide-out state (e.g., a fully opened state) of FIG. 4(b). According to one embodiment, in the intermediate state of the electronic device (101), as at least a portion of the display (460) slides, an area exposed to the outside may vary. Accordingly, another portion of the display of the electronic device (101) (e.g., a second area (A2)) may be only partially exposed to the outside of the housing (420).

[0075] Referring to FIG. 4, according to one embodiment, the electronic device (101) may include a housing (420) including a first housing portion (415) and a second housing portion (425), wherein the second housing portion (425) may be configured to be movably connected to the first housing portion (415) between a retracted (or retracted, or inserted) position (or state) (d1) and an extended (or expanded, or unfolded, or stretched) position (or state) (d2).

[0076] According to one embodiment, the electronic device (101) may include a flexible display (460) connected to the first housing portion (415) and the second housing portion (425) such that the size of a portion that can be seen from the front side of the housing (420) changes as the second housing portion (425) moves between the retracted position (or state) (d1) and the extended position (or state) (d2).

[0077] According to one embodiment, the electronic device (101) may include an actuator configured to move the second housing portion (425) relative to the first housing portion (415).

[0078] According to one embodiment, the housing (420) may include a first housing portion (415) and a second housing portion (425) that is arranged to be relatively movable with respect to the first housing portion (415). In one embodiment, the first housing portion (415) may be interpreted as a structure in which the electronic device (101) is arranged to be slidable with respect to the second housing portion (425). According to one embodiment, the second housing portion (425) may be arranged to be reciprocally movable for a predetermined distance in a direction shown with respect to the first housing portion (415), for example, in a direction indicated by arrow ①. When the electronic device (101) is in a slide-in state, the second housing portion (425) may be defined as a retracted position, and when the electronic device (101) is in a slide-out state, the second housing portion (425) may be defined as an extended position.

[0079] A housing (420) as illustrated in FIGS. 4(a) and 4(b) may include a first side wall (410) and a second side wall (430) on a first side (or long side) (e.g., side (440) or side (450)). In one embodiment, when the side walls are formed to surround the display (460), an opening for an audio module (e.g., a speaker) or an audio device may be formed in the first side wall (410) or the second side wall (430), or another wall of the housing (420), as indicated by S1 and S2. In FIG. 4, the audio module is illustrated as being arranged on both sides of the electronic device (101), for example, on at least one of the side side portions (440, 450), but the arrangement of the audio module is not limited thereto. According to one embodiment, in FIG. 4, the audio module is positioned adjacent to the side wall, but the present invention is not limited thereto, and the audio module may be formed as another separate structure and positioned on both edges of the combined or assembled structure or housing. In this way, the electronic device (101) may be designed so that the audio module is included in a structure that expands or contracts depending on the appearance and usage status. For example, in FIG. 4(b), the second housing portion (425) is exemplified as sliding in one direction (e.g., to the right) with respect to the first housing portion (415), but it may slide in another direction (e.g., to the left) or in both directions (e.g., to the right and left).

[0080] Although FIGS. 2 to 4 illustrate a case where one speaker is arranged on at least one side of each side of the housing, multiple speakers may be arranged. For example, two or more speakers may be arranged on a first side (e.g., the upper part (210) of the electronic device of FIG. 2, the upper part (315) of the electronic device of FIG. 3A, or the side (440) of FIG. 4), and two or more speakers may also be arranged on a second side (e.g., the lower part (220) of the electronic device of FIG. 2, the lower part (325) of the electronic device of FIG. 3A, or the side (450) of FIG. 4).

[0081] As illustrated in FIGS. 2 to 4, if the electronic device (101) includes a foldable display (e.g., the first display (260) of FIG. 2 or the display (300) of FIG. 3) or has a structure having a slidable or rollable display (e.g., the display (460) of FIG. 4), the position of the speaker (e.g., with respect to the user) may not be fixed but may vary depending on the state of the display. In such a case, a deviation in audio performance may occur when processing an audio signal, and when outputting a stereoscopic audio signal, it may be difficult to provide the realistic sound desired by the user. If a stereoscopic audio signal with a set stereoscopic sound effect applied is output regardless of the state of the display, for example, even though the content is played through a relatively small screen when the state of the display is folded or slid-in, a sound effect that seems to be projected far away in space is output, so the user may feel discomfort due to visual and auditory dissonance.

[0082] In one embodiment, an electronic device, an operating method thereof, and a storage medium for controlling audio output according to a display state can be provided, so as to output a stereoscopic audio signal by adaptively correcting an audio signal output through an audio module according to a display state. In one embodiment, when application of a stereoscopic sound effect is required according to a display state, the stereoscopic sound effect can be maximized, thereby providing realistic content and sound to a user.

[0083] Figure 5 is an internal block diagram of an electronic device according to one embodiment.

[0084] Referring to FIG. 5, the electronic device (101) (e.g., the electronic device (101) of FIGS. 1 to 4) may include a display (560) (e.g., the display (160) of FIG. 1, the first display (260) of FIG. 2, the display (300) of FIG. 3A, or the display (460) of FIG. 4), a sensor module (or sensor device) (576) (e.g., the sensor module (176) of FIG. 1), an audio processing unit (521), an audio module (or audio device) (570) (e.g., the audio module (170) of FIG. 1), and at least one processor (520) (e.g., the processor (120) of FIG. 1). The electronic device (101) may further include a memory (530) (e.g., the memory (130) of FIG. 1). According to one embodiment, the electronic device (101) may omit at least one of the components or additionally include other components.

[0085] According to one embodiment, the sensor module (576) may include at least one sensor. When the display (560) is a foldable display (e.g., the display (260) of FIG. 2 or the display (300) of FIG. 3A), the sensor module (576) may include a sensor (e.g., a digital Hall sensor, a 6-axis sensor) for determining the state of the display (560) (or the state of the electronic device (101)) (e.g., an unfolded state, a folded state, and / or a partially folded state) and the folded angle. In addition to the above, the sensor (576) may also include, for example, a strain sensor that outputs a strain value used to indirectly measure the folded angle of the electronic device (101).

[0086] In one embodiment, when the display (560) is a slidable or rollable display (e.g., display (460) of FIG. 4), the sensor module (576) may include a sensor for determining the state of the display (560) (or the state of the electronic device (101)) (e.g., slide-out state, slide-in state, and / or intermediate state) and the movement distance (or degree of extension).

[0087] In one embodiment, the processor (520) may identify the distance of movement (or degree of expansion, or expansion distance) based on at least one of a motor that moves the display (560), or a change in the capacitance of the display (560), in addition to at least one sensor.

[0088] According to one embodiment, the processor (520) may execute at least one application that requires output of an audio signal, such as a voice call function, an audio file playback function, or a video recording function. The processor (520) may visually output (or display) content corresponding to the application through a display (560). In addition, the processor (520) may audibly output an audio signal associated with the content through an audio module (570).

[0089] According to one embodiment, the audio processing unit (521) can input or output audio signals through the audio module (570), and process audio signals for the audio module (570). According to one embodiment, the audio processing unit (521) may be an audio digital signal processor (DSP). Although FIG. 5 illustrates a case where the processor (520) and the audio processing unit (521) are implemented separately, the audio processing unit (521) may be implemented to be included in the processor (520). For example, the processor (520) may include an audio DSP, and the audio DSP may be capable of driving a software (S / W) tuning solution.

[0090] According to one embodiment, the audio processing unit (521) may process an audio signal (hereinafter, “output audio signal”) to be output through a speaker (e.g., an output audio signal through an audio module (e.g., a speaker) of FIGS. 2 to 4) and an input audio signal to a microphone (not shown) that is generated due to the execution of an application that requires the output of an audio signal, such as a voice call function, an audio file playback function, or a video recording function. For example, when an earphone (not shown) is connected to the electronic device (101), the output audio signal may be output through the speaker of the earphone rather than the speaker (e.g., the audio module of FIGS. 2 to 4).

[0091] According to one embodiment, the audio processing unit (521) or the processor (520) may identify a gain to be applied to the output audio signal based on the state of the display (or the state of the electronic device (101)) (e.g., slide-out state, slide-in state, folded state, closed state, and / or intermediate state). The audio processing unit (521) or the processor (520) may generate a stereoscopic audio signal using software (or an algorithm) for generating a stereoscopic sound effect.

[0092] According to one embodiment, the audio processing unit (521) or the processor (520) may operate based on at least one of a head related transfer function (HRTF), an artificial reverberator, or a crosstalk canceller to generate a surround sound effect. According to one embodiment, the operations in the audio processing unit (521) or the processor (520) may be implemented as hardware and / or software components and may be referred to as a 'unit' or a 'module'. For example, an algorithm for generating a surround sound effect may be referred to as a surround sound filter, a surround sound generation unit, or a surround sound generation module. Here, the surround sound effect may be referred to as a spatial sound effect, which represents a surround audio signal (or surround sound) in a virtual space as if audio signals are output through speakers in various positions.

[0093] In one embodiment, examples of algorithms for generating a surround sound effect include a head-mounted transfer function, an artificial reverberation device, or a crosstalk canceller, but the types of the algorithms may not be limited thereto.

[0094] For example, the audio processing unit (521) may include a plurality of components for generating a three-dimensional sound effect as illustrated in FIG. 6. Here, FIG. 6 is a block diagram for generating a three-dimensional audio signal according to an embodiment. FIG. 6 illustrates a case where a head-related transfer function (610), an artificial reverberation unit (620), or a crosstalk remover (630) are implemented as individual components within the audio processing unit (521). However, the audio processing unit (521) may be implemented to be included in the processor (520), and instead of the audio processing unit (521), the head-related transfer function (610), the artificial reverberation unit (620), or the crosstalk remover (630) may be implemented as individual components within the processor (520).

[0095] Referring to FIG. 6, the audio processing unit (521) or the processor (520) may apply a gain (unit: dB) according to the state of the display (560) to at least one or a combination of a head-related transfer function (610), an artificial reverberation unit (620), or a crosstalk remover (630) when an audio signal corresponding to the execution of an application is input, to the audio signal, thereby generating a stereoscopic audio signal. The first gain may correspond to a first display state, for example, a closed display state or a folded display state, and the second gain may correspond to a second display state, for example, an open state or an unfolded display state. The audio processing unit (521) or the processor (520) may output the generated stereoscopic audio signal through an audio module (570) (e.g., a speaker), an external speaker, or an earphone.

[0096] According to one embodiment, the processor (520) may store in the memory (530) the gain to be applied to the head-mounted transfer function (610), the artificial reverberator (620), or the crosstalk remover (630). The processor (520) may store in the memory (530) different gains for each state of the display. The processor (520) may generate a stereoscopic audio signal using the audio processing unit (521) set to a gain according to the state of the display (560). For example, the different gains may be values ​​between 0 and 1.

[0097] According to one embodiment, the memory (530) can store different gains according to the state of the display (560) in a table format.

[0098] According to one embodiment, when the display (560) is a foldable display (e.g., the display (260) of FIG. 2 or the displays (300, 360) of FIGS. 3A and 3B) and the display (560) is in a folded state with respect to the folding axis, the processor (520) may control the display (560) to provide content through the display (e.g., the sub-display (360) of FIG. 3B) so that the stereoscopic sound effect is reduced compared to when the display (560) is in an unfolded state. For example, when the display (560) is in an unfolded state, the processor may control the display (560) to increase the stereoscopic sound effect compared to when the display (560) is in a folded state. The stereoscopic sound effect and / or gain may be controlled depending on the size of the currently displayed display area.

[0099] In one embodiment, the processor (520) may set the first gain when the display (560) is in a folded state, and may set the second gain to be greater than the first gain when the display (560) is in an unfolded state. For example, when the display (560) is in a folded state, the first gain may be set to 0 so as to minimize the stereoscopic sound effect, and when the display (560) is in an unfolded state, the second gain may be set to be greater than the first gain, for example, 1, so as to maximize the stereoscopic sound effect.

[0100] According to one embodiment, the state of the display (560) may be a partially folded state in addition to a folded state or an unfolded state. When the display (560) is in a partially folded state, for example, when the display (560) is folded at an angle between about 90 degrees and 180 degrees, different gains may be set according to the folded angle. For example, when the folded angle is between 90 degrees and 180 degrees, the gain may be set to a value between 0 and 1, which is about 0.5. In this way, when the range between 90 degrees and 180 degrees is divided into four equal parts, the gain corresponding to the range to which the folded angle belongs among the gains divided into four equal parts may be set as a value between 0 and 1. The method of determining the gain (or the value of the gain) according to the state of the display, for example, the folded angle of the display, may not be limited thereto.

[0101] Meanwhile, in the above description, the case where the gain to be applied to the head-mounted transfer function, the artificial reverberation unit, or the crosstalk eliminator is adjusted depending on whether the display (560) is in a folded, unfolded, or partially folded state has been described as an example. However, the gain value may be set differently depending on, for example, whether the main display (e.g., the main display (300) of FIG. 3A) and the sub-display (e.g., the sub-display (360) of FIG. 3B) are on / off, rather than the degree of folding. For example, when the main display (e.g., the main display (300) of FIG. 3A) is off and the sub-display (e.g., the sub-display (360) of FIG. 3B) is on, the gain to be applied to the head-mounted transfer function, the artificial reverberation unit, or the crosstalk eliminator may be set to the minimum. For example, the gain may be set to the minimum so that the three-dimensional sound effect is minimized. For example, when the main display (e.g., the main display (300) of FIG. 3a) is on and the sub-display (e.g., the sub-display (360) of FIG. 3b) is off, the gain to be applied to the head transfer function, artificial reverberation device, or crosstalk remover can be set to increase from the gain set to the minimum.

[0102] According to one embodiment, when the display (560) is a slidable or rollable display (e.g., the display (460) of FIG. 4), and the state of the display (560) is a slide-out state (or a fully opened state), the stereoscopic sound effect can be controlled to increase compared to the slide-in state (or a fully closed state).

[0103] Accordingly, when the state of the display (560) is such that the size of the visible portion of the display (560) from the front side of the housing is a first size (e.g., a slide-in state), the processor (520) may set the first gain to 0 so that the stereoscopic sound effect is reduced compared to when the display (560) is in a state (e.g., a slide-out state) in which the size of the visible portion of the display (560) from the front side of the housing is a second size larger than the first size, since the processor (520) provides content through the display (e.g., the first area (A1) of the display in FIG. 4(a)). Conversely, in order to increase the stereoscopic sound effect, the processor (520) may set the second gain to be larger than the first gain, for example, 1 or another appropriate value.

[0104] In one embodiment, the state of the display (560) may correspond to an intermediate state between a slide-in state and a slide-out state. When the display (560) is in an intermediate state, different gains may be set, for example, depending on the distance moved out of the housing.

[0105] According to one embodiment, the memory (530) may store an algorithm for generating a surround sound effect, such as a head-related transfer function, an artificial reverberation unit, or a crosstalk canceller. The processor (520) may apply a gain corresponding to the state of the display (560) to the algorithm for generating the surround sound effect. For example, the processor (520) may apply the same gain to each of the head-related transfer function (610), the artificial reverberation unit (620), or the crosstalk canceller (630). Additionally, the processor (520) may apply different gains to each of the head-related transfer function (610), the artificial reverberation unit (620), or the crosstalk canceller (630). For example, when the state of the display (560) is a fully expanded state or a slit-out state, the processor (520) may apply a gain set to 1 to the head transfer function (610), a gain set to 0.9 to the artificial reverberator (620), and a gain set to 1 to the crosstalk remover (630).

[0106] According to one embodiment, the processor (520) may identify whether the electronic device (101) is in speaker (e.g., stereo speaker) mode, and if the electronic device (101) is in speaker mode and the display (560) is in a fully unfolded state or a slid-out state, the processor (520) may apply a gain set to 1 to the crosstalk eliminator (630). On the other hand, if the electronic device (101) is not in speaker mode and the display (560) is in a fully unfolded state or a slid-out state, the processor (520) may apply a gain set to 1 to each of the head-mounted transfer function (610) and the artificial reverberator (620).

[0107] As described above, the audio processing unit (521) or the processor (520) can obtain a gain value (or multiple gain values) according to the state of the display (560) stored (or stored) in the memory (530) and apply the gain value to the audio signal using at least one or a combination of a head-mounted transfer function (610), an artificial reverberator (620), and a crosstalk remover (630), thereby generating a stereoscopic audio signal. In one embodiment, the stereoscopic sound effect can be reduced or increased according to the state of the display (560), thereby providing realistic content and sound to the user.

[0108] According to one embodiment, the electronic device (101) may include a display (160, 560), at least one sensor (176, 576), an audio device (or audio module) (570), at least one processor (120, 520), and a memory (530) for storing instructions.

[0109] According to one embodiment, the instructions, when executed by the at least one processor, may be configured to cause the electronic device to identify a state of the display by using the sensor in response to execution of an application.

[0110] According to one embodiment, the instructions may be configured to cause the electronic device to generate a first stereoscopic audio signal from an audio signal using an audio processing unit (521) set to a first gain based on identifying that the state of the display is a first state, and to output the generated first stereoscopic audio signal through the audio device.

[0111] In one embodiment, the instructions may be configured to cause the electronic device to change the gain of the audio processing unit from the first gain to the second gain based on identifying that the state of the display has changed from the first state to the second state.

[0112] According to one embodiment, the instructions may be configured to cause the electronic device to generate a second stereoscopic audio signal from the audio signal using the audio processing unit set to the second gain, and to output the generated second stereoscopic audio signal through the audio device.

[0113] According to one embodiment, the audio processing unit may be configured to operate based on at least one of a head related transfer function, an artificial reverberator, or a crosstalk canceller.

[0114] According to one embodiment, the instructions may be configured to cause the electronic device to store different gains for each state of the display, and to change the gain of the audio processing unit to a gain corresponding to the changed state of the display among the stored gains.

[0115] According to one embodiment, the different gains may be values ​​between 0 and 1, the first gain may be 0, and the second gain may be 1.

[0116] According to one embodiment, the display may be a foldable display, and a first state of the display may be a state in which the display is folded based on a folding axis, and a second state of the display may be a state in which the display is unfolded.

[0117] In one embodiment, the instructions may be configured to cause the electronic device to apply the second gain, which is greater than the first gain, to at least one of the head-mounted transfer function, the artificial reverberator, or the crosstalk canceller based on identifying that the state of the display has changed from the first state to the second state.

[0118] According to one embodiment, the display is a slidable or rollable display, and a first state of the display may be a state in which a size of a portion of the display that can be seen from the front side of the housing is a first size, and a second state of the display may be a state in which a size of a portion of the display that can be seen from the front side of the housing is a second size larger than the first size. The second size may be a maximum size of the display, and the first size may be a minimum size of the display.

[0119] In one embodiment, the instructions may be configured to cause the electronic device to apply the second gain to the crosstalk eliminator, which is greater than the first gain, based on identifying that the electronic device is in speaker mode and, in response to identifying that the electronic device is in speaker mode, identifying that the state of the display changes from the first state to the second state.

[0120] In one embodiment, the instructions may be configured to cause the electronic device to apply the second gain to the head-mounted transfer function or the artificial reverberator, which is greater than the first gain, based on identifying that the state of the display changes from the first state to the second state in response to identifying that the electronic device is not in the speaker mode.

[0121] FIG. 7 is a flowchart illustrating an operation for controlling audio output according to a display state in an electronic device according to an embodiment.

[0122] Referring to FIG. 7, the operating method may include operations 705 to 720. Each operation of the operating method of FIG. 7 may be performed by at least one of an electronic device (e.g., the electronic device (101) of FIGS. 1 to 5), at least one processor of the electronic device (e.g., the processor (120) of FIG. 1 or the processor (520) of FIG. 5). In one embodiment, at least one of operations 705 to 720 may be omitted, the order of some operations may be changed, or another operation may be added.

[0123] In operation 705, the electronic device (101) can identify the state of the display (160, 560) of the electronic device (101) in response to the execution of the application. The electronic device (101) can also identify the state of the display (160, 560) using a sensor or other appropriate method or device.

[0124] In operation 710, the electronic device (101) can generate a first stereoscopic audio signal from an audio signal using an audio processing unit (521) set to a first gain based on identifying that the state of the display is a first state, and output the generated first stereoscopic audio signal through an audio module (or audio device) (570).

[0125] In one embodiment, the audio processing unit may operate based on at least one of a head-related transfer function, an artificial reverberation device, or a crosstalk canceller.

[0126] In operation 715, the electronic device (101) may change the gain of the audio processing unit from the first gain to the second gain based on identifying that the state of the display has changed from the first state to the second state.

[0127] In operation 720, the electronic device (101) can generate a second stereoscopic audio signal from the audio signal using the audio processing unit set to the second gain, and output the generated second stereoscopic audio signal through the audio module (or audio device).

[0128] According to one embodiment, the electronic device (101) may apply the second gain, which is greater than (or less than) the first gain, to at least one of the head-mounted transfer function, the artificial reverberation device, or the crosstalk canceller based on identifying that the state of the display has changed from the first state to the second state.

[0129] According to one embodiment, the electronic device (101) can store different gains for each state of the display, and change the gain of the audio processing unit to a gain corresponding to the changed state of the display among the stored gains.

[0130] According to one embodiment, the different gains may be values ​​between 0 and 1, the first gain may be 0, and the second gain may be 1. A gain value of 1 may mean a maximum gain, and a gain value of 0 may mean a minimum gain or no gain.

[0131] According to one embodiment, the display is a foldable display, and a first state of the display may be a state in which the display is folded based on a folding axis, and a second state of the display may be a state in which the display is unfolded.

[0132] According to one embodiment, the display may be a slidable or rollable display, and a first state of the display may be a state in which the size of a portion visible from the front side of the housing is changed to a minimum, and a second state of the display may be a state in which the size of a portion visible from the front side of the housing is changed to a maximum.

[0133] According to one embodiment, the electronic device (101) can identify whether the electronic device (101) is in speaker mode. The electronic device (101) can apply a second gain greater than the first gain to the crosstalk eliminator based on identifying that the state of the display has changed from the first state to the second state in response to identifying that the electronic device (101) is in the speaker mode.

[0134] According to one embodiment, the electronic device (101) may apply a second gain greater than the first gain to the head-mounted transfer function or the artificial reverberator based on identifying that the state of the display has changed from the first state to the second state in response to identifying that the electronic device (101) is not in the speaker mode.

[0135] Figure 8 is an exemplary diagram for explaining a head transfer function according to one embodiment.

[0136] The head-related transfer function (610) (HRTF, head related transfer filter) may include path information from the spatial location of a sound generated from a sound source to the user's ears, for example, frequency transfer characteristics. The HRTF (610) may include information on reflection and diffraction from the head, torso, or ears that exist on the path from the sound source to the eardrums, which serve as clues for the user to determine the location of the sound source, along with information on time differences and level differences between channels that can perceive a sense of direction. Therefore, a three-dimensional audio signal can be generated by applying a head related transfer function that adds spatial information corresponding to a three-dimensional location to a simple sound that is not stereoscopic.

[0137] Referring to FIG. 8, when an audio signal is input, the input audio signal may be transferred to a head-related transfer function (610) to which a specified gain (612) is applied. Here, the gain (612) may be determined according to the state of the display. For example, when the state of the display is a first state (e.g., a folded state or a slide-in state), the gain (612) may be set to 0 or a value close to 0. On the other hand, when the state of the display is a second state (e.g., an unfolded state or a slide-out state), the gain (612) may be set to 1 or a value close to 1. Here, the closer the gain (612) is set to 1, the higher the three-dimensional sound effect may be. Therefore, assuming that the audio signal input to the head-mounted transfer function (610) is an S signal, referring to the drawing reference numeral 611, a first stereoscopic audio signal (e.g., S' signal) can be generated and output by adding (614) a signal to which a gain (612) set to 1 is applied to the S' signal indicating directionality through the head-mounted transfer function (610) (e.g., S' signal + gain (e.g., 1 dB)) and a signal to which a 1-gain (613) is applied to the original audio signal (e.g., S signal + 1-gain (e.g., 0 dB)). In other words, the S signal is input to the head-mounted transfer function (610), modified by the head-mounted transfer function (610), and then further modified with a first gain (612). The same S signal is modified with a second gain (613) (e.g., 1-first gain). The two differently modified signals are added together to form a sum from the signal modified by the head transfer function (610) and the signal modified by the first gain (612) and the signal modified by the second gain.

[0138] On the other hand, in the case of the gain (612) set to 0 or a value close to 0, a second stereoscopic audio signal (e.g., S signal) can be generated and output by adding (614) a signal to which the gain (612) set to 0 or a value close to 0 is applied to the S' signal indicating directionality through the head-mounted transfer function (610) (e.g., S' signal + gain (e.g., 0 dB)) and a signal to which a 1-gain (613) is applied to the original audio signal (e.g., S signal + 1-gain (e.g., 1 dB)). In other words, the S signal is input to the head-mounted transfer function (610), modified by the head-mounted transfer function (610), and then further modified by the first gain (612). The same S signal is modified by the second gain (613) (e.g., 1-first gain). The two differently modified signals are added together to form a sum from the signal modified by the head-related transfer function (610) and the first gain (612) and the signal modified by the second gain. Thus, for example, if the first gain is 1, the second gain can be 0. If the first gain is 0, the second gain can be 1. The sum of the first gain and the second gain can be 1. In this way, a portion of the sound modified by the head-related transfer function can be adjusted.

[0139] The first stereoscopic audio signal may be a signal with a maximally increased stereoscopic effect compared to the second stereoscopic audio signal.

[0140] Figure 9 is an exemplary diagram illustrating an artificial reverberator according to one embodiment.

[0141] An artificial reverberator may be a function that provides virtual directionality by applying a function that simulates reflections (or reverberation) to an input audio signal.

[0142] Referring to FIG. 9, assuming that the audio signal input to the artificial reverberator (620) is an S signal, when the state of the display is a second state (e.g., an unfolded state or a slide-out state), the gain (622) can be set to 1 or close to 1. Referring to the drawing reference numeral 621, a signal (e.g., S' signal + gain (e.g., 1 dB)) to which the gain (622) set to 1 is applied to the S' signal representing the reflection sound for the audio signal through the artificial reverberator (620) and the original audio signal (e.g., S signal) is added (623) to generate and output a first stereoscopic audio signal (e.g., S signal + (S' signal + gain)). In other words, the original signal and the signal modified by the artificial reverberator can be mixed, and the ratio of the signal modified by the artificial reverberator in the final signal can vary depending on the current display state.

[0143] On the other hand, when the state of the display is the first state (e.g., folded state or slide-in state), the gain (622) can be set to 0 or a value close to 0. Accordingly, a signal (e.g., S' signal + gain (e.g., 0 dB)) in which the gain (622) set to 0 is applied to the S' signal representing the reflection sound for the audio signal through the artificial reverberator (620) and the original audio signal (e.g., S signal) is added (623) to generate and output a second stereoscopic audio signal (e.g., S signal + (S' signal + gain)). The second stereoscopic audio signal may be a signal in which the stereoscopic sound effect is reduced compared to the first stereoscopic audio signal.

[0144] Fig. 10 is an exemplary diagram illustrating a crosstalk remover according to one embodiment.

[0145] When listening to a three-dimensional audio signal through stereo speakers, the audio signals generated from the left and right speakers may be mixed due to interference in space as they travel to the user's ears, making it difficult to perceive a three-dimensional effect at the listening position. The mixed signal added to the original audio signal, for example, a two-channel audio signal, at the listening position is called crosstalk and can be removed using a crosstalk remover (630).

[0146] Referring to FIG. 10, a crosstalk remover (630) can remove crosstalk from a two-channel audio signal (e.g., left and right audio signals) using four channels (e.g., H_LL (631), H_RL (632), H_LR (633), or H_RR (634)). The four channels can include a Left Left channel (631), a Right Left channel (632), a Left Right channel (633), and / or a Right Right (634) channel.

[0147] Referring to drawing number 635, when the gains (636, 637) are set to 1 or close to 1, the signal of H_LL (631) and the signal of H_RL (632) are added (638) to output a left stereoscopic audio signal, and the signal of H_LR (633) and the signal of H_RR (634) are added (639) to output a right stereoscopic audio signal. For example, when the gains (636, 637) are set to 1 or close to 1, crosstalk may occur, reducing the stereoscopic sound effect. On the other hand, when the gain (636, 637) is set to 0 or close to 0, the crosstalk of the H_RL (632) channel and the H_LR (633) channel among the H_LL (631) channel, the H_RL (632) channel, the H_LR (633) channel, or the H_RR (634) channel can be eliminated, thereby increasing the surround sound effect. Accordingly, in the case of a speaker mode that outputs sound through a stereo speaker included in the electronic device (101), the user can experience the surround sound effect by eliminating crosstalk in a form corresponding to the speaker.

[0148] Fig. 11 is a flowchart of operations in an audio processing unit according to a display state according to one embodiment.

[0149] Referring to FIG. 11, the operating method may include operations 1105 to 1140. Each operation of the operating method of FIG. 11 may be performed by at least one of an electronic device (e.g., the electronic device (101) of FIGS. 1 to 5), at least one processor of the electronic device (e.g., the processor (120) of FIG. 1 or the processor (520) of FIG. 5). In one embodiment, at least one of operations 1105 to 1140 may be omitted, the order of some operations may be changed, or another operation may be added.

[0150] In operation 1105, the electronic device (101) can identify the state of the display (560) in operation 1110 in response to the execution of an application or when an audio signal is input, for example, using a sensor.

[0151] In operation 1115, the electronic device (101) can identify a gain corresponding to the state of the display (560).

[0152] In operation 1120, the electronic device (101) can identify whether it is in speaker mode (or whether the speaker mode is activated). For example, the electronic device (101) can identify whether it is in speaker mode (or whether the speaker mode is activated) using a speaker (e.g., a stereo speaker) within the electronic device (101).

[0153] In response to identifying that the speaker mode is active (or that the speaker mode is enabled), in operation 1125, the electronic device (101) may apply the gain to at least one of the head-related transfer function, the artificial reverberation unit, or the crosstalk canceller of the audio processing unit (521).

[0154] In operation 1135, the electronic device (101) can generate a stereoscopic audio signal from the audio signal using the audio processing unit (521) set to the gain, and output the stereoscopic audio signal in operation 1140. For example, the electronic device (101) can control the stereoscopic audio signal to be output through a speaker (e.g., a stereo speaker) included in the electronic device (101).

[0155] On the other hand, in operation 1120, in response to identifying that it is not in speaker mode (or that the speaker mode is not activated), in operation 1130, the electronic device (101) may apply the gain to the head-related transfer function or artificial reverberation unit of the audio processing unit (521). In operation 1135, the electronic device (101) may generate a stereoscopic audio signal from the audio signal using the audio processing unit (521) set with the gain, and output the stereoscopic audio signal in operation 1140. For example, the electronic device (101) may control the stereoscopic audio signal to be output through the audio module (570) (e.g., the audio module of FIGS. 2 to 4).

[0156] Fig. 12 is an exemplary diagram illustrating the characteristics of a stereoscopic audio signal output according to a display state centered on a user according to one embodiment. Fig. 12 illustrates an electronic device (101) having a display structure that can be slidable or rolled up.

[0157] A user may execute at least one application requiring audio signal output, such as a voice call function, an audio file playback function, or a video recording function, using an electronic device (101) within a space (1210). The electronic device (101) may identify the state of the display before generating a stereoscopic audio signal from the audio signal in response to the execution of the application.

[0158] As illustrated in 1200a of FIG. 12, when the state of the display is a first state (e.g., slide-in state), the electronic device (101) can generate a stereoscopic audio signal from the input audio signal by applying a first gain (e.g., 0 dB) to at least one of a head-mounted transfer function, an artificial reverberation device, or a crosstalk remover when an audio signal is input, and output the stereoscopic audio signal through a speaker (e.g., 1270a, 1270b).

[0159] As illustrated in 1200b of FIG. 12, when the state of the display changes from a first state (e.g., a slide-in state) to a second state (e.g., a slide-out state), the electronic device (101) can generate a stereoscopic audio signal from the input audio signal and output it through speakers (e.g., 1270a, 1270b) by applying a second gain (e.g., 1 dB) greater than a first gain (e.g., 0 dB) to at least one of a head-mounted transfer function, an artificial reverberator, or a crosstalk remover when inputting an audio signal. If the electronic device (101) operates in a speaker mode that outputs sound through stereo speakers, the electronic device (101) can generate a stereoscopic audio signal from the input audio signal and output it through speakers (e.g., 1270a, 1270b) by applying a second gain (e.g., 1 dB) greater than the first gain (e.g., 0 dB) to the crosstalk remover. When operating in the speaker mode, the electronic device (101) can generate a three-dimensional audio signal by eliminating crosstalk. For example, the sound is reflected off the wall of the space (1210) surrounding the user and output to the location where the user is located, allowing the user to hear a three-dimensional sound similar to that projected from a speaker located at the listening location.

[0160] Additionally, when the state of the display changes from a second state (e.g., a slide-out state) to a first state (e.g., a slide-in state), the gain to be applied to at least one of the head-mounted transfer function, the artificial reverberation device, or the crosstalk canceller may be reduced to reduce the surround sound effect.

[0161] According to one embodiment, by adjusting the gain of an algorithm for generating a surround sound effect as the state of the display changes, the surround sound effect can be maximized when the state of the display is in an unfolded state or a slide-out state, thereby providing a natural user experience by linking the visual experience and the auditory experience.

[0162] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0163] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0164] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0165] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0166] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0167] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0168] According to one embodiment, a storage medium storing at least one computer-readable instruction, wherein the at least one instruction, when executed by at least one processor (120, 520) of an electronic device (101), causes the electronic device to perform at least one operation, wherein the at least one operation may include an operation of identifying a state of a display (160, 560) of the electronic device in response to execution of an application.

[0169] According to one embodiment, the at least one operation may include generating a first stereoscopic audio signal from an audio signal using an audio processing unit (521) set to a first gain based on identifying that the state of the display is a first state, and outputting the generated first stereoscopic audio signal through an audio device (or audio module) (570).

[0170] In one embodiment, the at least one operation may include changing a gain of the audio processing unit from the first gain to the second gain based on identifying that the state of the display has changed from the first state to the second state.

[0171] According to one embodiment, the at least one operation may include generating a second stereoscopic audio signal from the audio signal using the audio processing unit set to the second gain, and outputting the generated second stereoscopic audio signal through the audio device.

[0172] In one embodiment, the display may be a sliding or rollable display, wherein a first state of the display is a state in which the size of a portion visible from the front of the housing is changed to a minimum, and a second state of the display is a state in which the size of a portion visible from the front of the housing is changed to a maximum.

Claims

1. In an electronic device (101), display(160, 560); At least one sensor (176, 576); audio device (570); and At least one processor (120, 520); and A memory (530) for storing instructions, wherein the instructions cause the electronic device (101) to perform, by the at least one processor, In response to the execution of the application, the state of the display (160, 560) is identified by using the sensor (176, 576), Based on identifying that the state of the above display (160, 560) is the first state, a first stereoscopic audio signal is generated from the audio signal using an audio processing unit (521) set to the first gain, and the generated first stereoscopic audio signal is output through the audio device (570). Based on identifying that the state of the display (160, 560) has changed from the first state to the second state, the gain of the audio processing unit (521) is changed from the first gain to the second gain, An electronic device configured to generate a second stereoscopic audio signal from the audio signal using the audio processing unit (521) set to the second gain, and to output the generated second stereoscopic audio signal through the audio device (570).

2. In the first paragraph, the audio processing unit (521) An electronic device that operates based on at least one of a head related transfer function, an artificial reverberator, or a crosstalk canceller.

3. In the first or second paragraph, the instructions cause the electronic device (101) to: Store different gains for each state of the above display (160, 560), An electronic device, wherein the gain of the audio processing unit (521) is set to be changed to a gain corresponding to the changed state of the display (160, 560) among the stored gains.

4. An electronic device according to any one of claims 1 to 3, wherein the different gains have values ​​between 0 and 1, the first gain is 0, and the second gain is 1.

5. In any one of claims 1 to 4, the display is a foldable display, The first state of the above display is a state in which the display is folded based on the folding axis, An electronic device wherein the second state of the display is that the display is in an unfolded state.

6. In any one of paragraphs 1 to 5, the instructions cause the electronic device (101) to: An electronic device set to apply the second gain, which is greater than the first gain, to at least one of the head-mounted transfer function, the artificial reverberation device, or the crosstalk canceller based on identifying that the state of the display (160, 560) has changed from the first state to the second state.

7. In any one of clauses 1 to 6, the display (160, 560) is a slidable or rollable display, The first state of the above display (160, 560) is a state in which the size of the visible portion of the display (160, 560) from the front side of the housing is the first size, An electronic device in which the second state of the display (160, 560) is a state in which the size of a portion of the display (160, 560) that can be seen from the front side of the housing is a second size larger than the first size.

8. In any one of paragraphs 1 to 7, the instructions cause the electronic device (101) to: Identify whether the above electronic device (101) is in speaker mode, An electronic device, wherein, in response to identifying that the electronic device (101) is in the speaker mode, the electronic device is set to apply the second gain greater than the first gain to the crosstalk eliminator based on identifying that the state of the display (160, 560) has changed from the first state to the second state.

9. In any one of paragraphs 1 to 8, the instructions cause the electronic device (101) to: An electronic device, wherein the electronic device (101) is set to apply the second gain, which is greater than the first gain, to the head-mounted transfer function or the artificial reverberator based on identifying that the state of the display (160, 560) has changed from the first state to the second state in response to identifying that the electronic device (101) is not in the speaker mode.

10. A method for controlling audio output according to the status of a display (160, 560) in an electronic device (101), An operation of identifying a state of a display (160, 560) of the electronic device (101) by using at least one sensor (176, 576) of the electronic device (101) in response to execution of the application; An operation of generating a first stereoscopic audio signal from an audio signal using an audio processing unit (521) set to a first gain based on identifying that the state of the above display (160, 560) is a first state, and outputting the generated first stereoscopic audio signal through an audio device (570); An operation of changing the gain of the audio processing unit (521) from the first gain to the second gain based on identifying that the state of the display (160, 560) has changed from the first state to the second state; and A method for controlling audio output according to a display state, comprising an operation of generating a second stereoscopic audio signal from the audio signal using the audio processing unit (521) set to the second gain, and outputting the generated second stereoscopic audio signal through the audio device (570).

11. In the 10th paragraph, the audio processing unit (521) A method for controlling audio output depending on a display state, the method comprising: operating based on at least one of a head related transfer function, an artificial reverberator, or a crosstalk canceller.

12. In clause 10 or 11, An operation of storing different gains for each state of the above display (160, 560); and A method for controlling audio output according to a display state, including an operation of changing the gain of the audio processing unit (521) to a gain corresponding to a changed state of the display (160, 560) among the stored gains.

13. In any one of claims 10 to 12, the display (160, 560) is a foldable display, The first state of the above display (160, 560) is a state in which the display (160, 560) is folded based on the folding axis. A method for controlling audio output according to a display state, wherein the second state of the display (160, 560) is an unfolded state.

14. In any one of paragraphs 10 to 13, A method for controlling audio output according to a display state, comprising: applying a second gain greater than the first gain to at least one of the head-mounted transfer function, the artificial reverberation device, or the crosstalk canceller, based on identifying that the state of the display (160, 560) has changed from the first state to the second state.

15. A storage medium storing at least one computer-readable instruction, wherein the at least one instruction, when executed by at least one processor (120, 520) of an electronic device (101), causes the electronic device to perform at least one operation, wherein the at least one operation is: An operation of identifying a state of a display (160, 560) of the electronic device by using at least one sensor (176, 576) of the electronic device in response to execution of the application; An operation of generating a first stereoscopic audio signal from an audio signal using an audio processing unit (521) set to a first gain based on identifying that the state of the above display (160, 560) is a first state, and outputting the generated first stereoscopic audio signal through an audio device (570); An operation of changing the gain of the audio processing unit (521) from the first gain to the second gain based on identifying that the state of the display (160, 560) has changed from the first state to the second state; and A storage medium including an operation of generating a second stereoscopic audio signal from the audio signal using the audio processing unit (521) set to the second gain, and outputting the generated second stereoscopic audio signal through the audio device (570).

Images