Electronic device and method for providing user interface thereof

Abstract

The electronic device may include memory, a display, a haptic module, a sound module, and a processor. The processor may detect an event for outputting vibration and sound, delay the output of vibration, and change the sound output mode from a first mode to a second mode for high-speed output while the output of vibration is delayed. In the second mode, the processor may output sound using the sound module and output delayed vibration through the haptic module so that the output of sound and delayed vibration are synchronized. As the sound output mode is changed to the second mode, the sound output path may be opened. Alternatively, the sound output path may be changed from the first path for normal output to the second path for high-speed output. In addition to this, various other embodiments are possible.

Description

Technology Field

[0001] This document relates to an electronic device and a method for providing a user interface thereof. Background Technology

[0002] Electronic devices (e.g., smartphones, mobile terminals, laptops, tablets, or wearable devices) can provide various functions such as voice communication functions, short-range wireless communication functions, mobile communication functions, shooting functions, content playback functions, and navigation functions.

[0003] In addition, the use of flexible electronic devices (e.g., smartphones) in which the physical shape or size of the display can be varied has recently become widespread. In the case of sliderable or foldable electronic devices, the demand for a large screen and portability can be simultaneously satisfied by adopting a structure that allows the display to be expanded or contracted. For example, portability can be maintained by reducing the display in the slide-in or folded state of the electronic device. A large screen can be provided by expanding the display in the slide-out or unfolded state of the electronic device.

[0004] As the performance of electronic devices improves, their utility increases, and their functions and structures become more diverse, the importance of user interfaces is also growing day by day. The problem to be solved

[0005] When electronic devices use different types of user interfaces (e.g., vibration and sound), it may be difficult to implement harmonious interaction between them.

[0006] For example, synchronization can be difficult because the hardware outputting vibration and sound is different, and the software logic for processing, handling, or outputting these different types of data—vibration and sound data—also differs. Due to a lack of synchronization, vibration and sound may not harmonize and may be output separately.

[0007] In the event of synchronization failure, electronic devices may not be able to deliver useful information to the user in an efficient manner at the correct timing.

[0008] Furthermore, unnatural user interfaces that do not harmonize with one another can be detached from the user's senses or actions in the real world or the actual situation of the electronic device, thereby reducing user interest, immersion, or attention.

[0009] Various embodiments of this document can implement harmonious interaction between different types of user interfaces to deliver a natural and immersive emotional experience or improve operability.

[0010] Various embodiments of this document can improve synchronization performance between different types of user interfaces and deliver useful information to users in a highly engaging and intuitive manner.

[0011] Various embodiments of this document can deliver a vivid and realistic emotional experience to the user by responding in real time to changes in the mechanical state of a flexible type electronic device or to user actions using said electronic device. means of solving the problem

[0012] An electronic device according to various embodiments may include a memory, a display, a haptic module, a sound module, and at least one processor. The at least one processor may be operatively connected to the memory, the display, the haptic module, and the sound module. The memory may store instructions in which, at execution, the at least one processor detects an event for outputting vibration and sound, delays the output of the vibration, changes the sound output mode from a first mode to a second mode for high-speed output while the output of the vibration is delayed, outputs the sound using the sound module in the second mode, and outputs the delayed vibration through the haptic module so that the outputs of the sound and the delayed vibration are synchronized.

[0013] A method for providing a user interface of an electronic device according to various embodiments may include an operation of detecting an event for outputting vibration and sound, an operation of delaying the output of the vibration, an operation of changing the sound output mode from a first mode to a second mode for high-speed output while the output of the vibration is delayed, and an operation of outputting the sound in the second mode and outputting the delayed vibration so that the output of the sound and the output of the delayed vibration are synchronized. Effects of the invention

[0014] According to various embodiments, harmonious interaction between different types of user interfaces can be implemented to deliver a natural and immersive emotional experience or improve operability.

[0015] According to various embodiments, synchronization performance between different types of user interfaces can be improved to deliver useful information to users in a highly engaging and intuitive manner.

[0016] According to various embodiments, a flexible type electronic device can respond in real time to changes in the mechanical state of the device or user actions using the device, thereby delivering a vivid and realistic emotional experience to the user.

[0017] In addition, various effects that can be identified directly or indirectly through this document may be provided. Brief explanation of the drawing

[0018] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments. FIG. 2 is a block diagram of an electronic device according to one embodiment. FIG. 3 is a block diagram illustrating the hierarchical structure of an electronic device according to one embodiment. FIG. 4 is a flowchart illustrating a method for providing a user interface of an electronic device according to one embodiment. FIG. 5 is a flowchart illustrating a method for providing a user interface of an electronic device according to another embodiment. FIG. 6 is a timing diagram illustrating vibration and sound output results in an electronic device according to one embodiment. FIG. 7 is a timing diagram illustrating the output results of vibration, sound, and visual elements in an electronic device according to one embodiment. FIG. 8 is a flowchart illustrating the operation of switching sound output modes in an electronic device according to one embodiment. FIG. 9 is a flowchart illustrating the operation of opening a sound output path in an electronic device according to one embodiment. FIG. 10 is a flowchart illustrating the operation of delaying vibration in an electronic device according to one embodiment. FIGS. 11a to 11e are drawings for explaining a method of providing a user interface of an electronic device according to one embodiment. FIGS. 12a and FIGS. 12b are drawings for explaining a method of providing a user interface of an electronic device according to another embodiment. FIGS. 13a to 13c are drawings for explaining a method of providing a user interface for electronic devices according to another embodiment. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Specific details for implementing the invention

[0019] Various embodiments are described below with reference to the attached drawings.

[0020] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0021] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0022] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0023] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0024] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0025] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0026] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0027] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0028] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0029] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0030] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0031] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0032] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0033] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0034] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0035] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0036] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0037] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0038] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0039] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0040] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0041] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0043] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0044] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0045] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0046] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0047] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0048] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0050] FIG. 2 is a block diagram of an electronic device according to one embodiment.

[0051] In one embodiment, the electronic device (200) may include at least a portion of the electronic device (101) shown in FIG. 1.

[0052] Referring to FIG. 2, the electronic device (200) may include a processor (210), memory (220), display (230), haptic module (240), and sound module (250). For example, the processor (210), memory (220), display (230), and haptic module (240) of FIG. 2 may correspond to the processor (120), memory (130), display module (160), and haptic module (179) of FIG. 1, respectively. The sound module (250) of FIG. 2 may include at least some of the audio module (170) and sound output module (155) of FIG. 1.

[0053] The electronic device (200) may include additional components in addition to the components shown in FIG. 2. Alternatively, at least one of the components shown in FIG. 2 may be omitted.

[0054] The processor (210), memory (220), display (230), haptic module (240), and sound module (250) included in the electronic device (200) are electrically and / or operationally connected to exchange signals (e.g., commands or data) with one another.

[0055] In one embodiment, the processor (210) may include at least one processor. For example, the processor (210) may include at least one of an application processor (AP) (e.g., the main processor (121) of FIG. 1) and a communication processor (CP) (e.g., an auxiliary processor (123)).

[0056] The processor (210) can execute and / or control various functions supported by the electronic device (200). The processor (210) can control at least some of the memory (220), display (230), haptic module (240), and sound module (250). Functions (or logic) specified by said control can be performed.

[0057] The processor (210) can execute an application and control various hardware by executing code written in a programming language stored in the memory (220) of the electronic device (200).

[0058] In one embodiment, the operation of the processor (210) may be performed as instructions stored in memory (220) are executed. Alternatively, the processor (210) may execute instructions stored in memory (220) to perform a specified function (or logic).

[0059] In one embodiment, the display (230), the haptic module (240), and / or the sound module (250) may each be an output module for providing a user interface (UI) of a specified type. Different types of user interfaces may be output through the display (230), the haptic module (240), and / or the sound module (250).

[0060] In one embodiment, sound, which is an auditory type user interface, can be output (or played) through the sound module (250). Vibration, which is a tactile type user interface, can be output (or played) through the haptic module (240).

[0061] In one embodiment, a hybrid type user interface can be output through a combination of a sound module (250) and a haptic module (240).

[0062] In some embodiments, visual elements, which are visual types of user interfaces, may be additionally output (e.g., displayed) through the display (230). The visual elements may include one or more of graphic elements (e.g., animations) displayed through the display (230) and dimming elements expressed by the lighting of the display (230). For example, graphic elements may include all kinds of elements that can be output (or displayed) through the display (230), such as animations, icons, menus, message windows, images, and screens. Dimming elements may be elements that can be expressed through dimming control of the lighting of the display (230). For example, the dimming elements may be elements that implement screen turn-on, screen brightness level-up, or screen flickering of the display (230).

[0063] In some embodiments, a hybrid type user interface may be output through a combination of a display (230), a sound module (250), and a haptic module (240).

[0064] In one embodiment, the processor (210) can synchronize the output timing by eliminating output deviations between different types of user interfaces (e.g., between vibration and sound, or between vibration, sound and visual elements).

[0065] For example, the processor (210) can synchronize the output of two types of user interfaces, vibration and sound. For another example, the processor (210) can synchronize the output of three types of user interfaces, vibration, sound, and visual elements.

[0066] In one embodiment, the processor (210) can detect an event for outputting vibration and sound. The processor (210) can delay the output of vibration. For example, the processor (210) can delay the vibration output time (or vibration start time) by a specified offset (e.g., 70ms to 80ms) so that the vibration starts at the sound output time (or sound start time).

[0067] The processor (210) can change the sound output mode from a first mode to a second mode for high-speed output while the output of vibration is delayed.

[0068] For example, as the sound output mode is changed to a second mode, the sound output path may be opened and the sound module (250) may be activated. As another example, as the sound output mode is changed to a second mode, the sound output path may be changed from a first path for normal output to a second path for high-speed output.

[0069] The point at which the sound output mode changes from the first mode to the second mode may vary depending on the embodiment. For example, the point in time may vary depending on the operation scenario of the electronic device (200).

[0070] For example, the processor (210) may change the sound output mode to a second mode for high-speed output in response to an event (or immediately upon the occurrence of the event). In this case, the timing of the change to the second high-speed mode may correspond to the timing of the occurrence of the event (e.g., within a certain offset (e.g., ±30ms)). As the sound output mode is changed to the second mode in response to the event, the sound output path may be changed from a first path for normal output to a second path for high-speed output.

[0071] As another example, the processor (210) may change the sound output mode to a second mode for high-speed output in response to a pre-event of the event. In this case, the timing of the change to the second high-speed mode may correspond to the timing of the occurrence of the pre-event (e.g., within a certain offset (e.g., ±30ms)), and the timing of the sound output may correspond to the timing of the occurrence of the event (e.g., within a certain offset (e.g., ±30ms)). As the sound output mode is changed to the second mode in response to the pre-event, the sound output path is opened so that the sound module (250) is activated first, and then sound can be output through the sound module (250) activated in response to the event.

[0072] If the opening of the sound output path and / or the activation of the sound module (250), which is a hardware element for sound output, is performed in advance, the time required for sound output may be reduced by more than a certain value (e.g., 100ms), thereby improving synchronization performance.

[0073] In one embodiment, the sound output progress state of the electronic device (200) may sequentially transition from a standby state to a ready state, an idle state, and a play state depending on the progress of the sound output. In this case, as the sound output progress state transitions to the idle state immediately before sound playback, a signal requesting the output of delayed vibration may be transmitted from the sound output path to the vibration output path.

[0074] The sound output path can continue the sound output after transmitting the signal. The vibration output path can delay the vibration and output the delayed vibration in response to the signal. Accordingly, the synchronization performance between vibration and sound can be improved.

[0075] In one embodiment, when the sound output mode is a first mode, which is a general output mode, a first buffer (a buffer for general output) may be used. When the sound output mode is a second mode, a second buffer (a buffer for high-speed output) may be used for high-speed output. The second buffer may have a smaller size or latency value compared to the first buffer.

[0076] In one embodiment, the processor (210) can output sound using the sound module (250) in a second mode and output delayed vibration through the haptic module (240) so that the output of the sound and the delayed vibration are synchronized.

[0077] In some embodiments, the electronic device (200) may detect an event for the output of vibration, sound, and visual elements. In this case, the electronic device (200) may further perform an action to synchronize the output of the visual elements with the output of vibration and sound. For example, the electronic device (200) may shift the output timing of the visual elements (e.g., animation). Additionally, the electronic device (200) may delay the output of vibration, and while the output of vibration is delayed, open at least a portion of the sound output path or change the sound output mode from a non-fast output mode (e.g., a first mode) to a fast output mode (e.g., a second mode). The electronic device (200) may output sound through the sound module (250) and output delayed vibration through the haptic module (240) using at least one of the open sound output path and the fast output mode, and output the visual elements through the display (230) in response to the shifted output timing, so that the output of sound, delayed vibration, and visual elements is synchronized.

[0078] According to one embodiment, the electronic device (200) can implement harmonious interaction through synchronization in various situations where physical feedback (e.g., vibration as tactile feedback, sound as auditory feedback, and / or visual elements as visual feedback) is required.

[0079] FIG. 3 is a block diagram illustrating the hierarchical structure of an electronic device according to one embodiment.

[0080] Referring to FIG. 3, the electronic device (200) may include a hardware layer (310), a kernel layer (320), an abstraction layer (330) (hardware abstraction layer, HAL), a framework layer (340), and an application layer (350).

[0081] In one embodiment, the hardware layer (310) may include a speaker (251), a motor (241), and a display (230). Each component of the hardware layer (310) may be implemented as physically distinct hardware. For example, the speaker (251) may be included in the sound module (250) of FIG. 2. The motor (241) may be included in the haptic module (240) of FIG. 2. The display (230) may correspond to the display (230) of FIG. 2.

[0082] Each component of the kernel layer (320), abstraction layer (330), framework layer (340) and / or application layer (350) may be implemented in firmware and / or software and stored in memory (220) of the electronic device (200). For example, the processor (210) of the electronic device (200) may operate each component by executing instructions stored in memory (220). Alternatively, each component may operate as instructions stored in memory (220) are executed.

[0083] Components of the kernel layer (320), abstraction layer (330), framework layer (340) and / or application layer (350) can control components of the hardware layer (310) (e.g., at least one of a speaker (251), a motor (241), and a display (230)) or perform specified functions (or logic) through said components.

[0084] The kernel layer (320) may include an audio driver (321), a vibration driver (323), and a display driver (325).

[0085] The abstraction layer (330) may include an audio HAL (331), a vibration HAL (333), and a display HAL (335).

[0086] The framework layer (340) may include an audio service (341), an audio manager (342), a vibration service (343), a system vibrator (344), a graphics framework (345), and a graphics surface (346).

[0087] An interaction module (300) may be configured between the abstraction layer (330) and the framework layer (340). For example, instructions constituting the interaction module (300) may be stored in memory (220) and executed by the processor (210). The interaction module (300) may be for synchronizing different types of user interfaces, for example, vibration, sound, and graphics. The interaction module (300) may interact with the abstraction layer (330) and the framework layer (340) for synchronizing operations. For example, a part of the interaction module (300) may be implemented in the abstraction layer (330) and interact with the audio HAL (331), vibration HAL (333), and display HAL (335) within the abstraction layer (330). Other parts of the interaction module (300) are implemented in the framework layer (340) and can interact with the audio service (341), vibration service (343), and graphics framework (345) within the framework layer (340).

[0088] The application layer (350) may include one or more applications or one or more application components. For example, the application layer (350) may include an application (351) including a Power UI module (352) and a keyboard module (353), and an application (355) including a time picker module (356) and a scroll UI module (357).

[0089] Reference numeral 365 may represent the flow of original data (e.g., commands, signals, and / or data). The original data may be unsynchronized data. For example, the original data may not pass through the interaction module (300). For example, if the output of vibration, sound, and graphics is not synchronized, the synchronization operation for the output between the audio service (341), the vibration service (343), and / or the graphics framework (345) may be omitted. In this case, the first data output from the audio service (341), the second data output from the vibration service (343), and the third data output from the graphics framework (345) may be processed individually. The first data, the second data, and the third data may be transmitted to the audio HAL (331), the vibration HAL (333), and the display HAL (335), respectively, without a synchronization process.

[0090] Reference numeral 361 may represent a flow of synchronized data (e.g., commands, signals, and / or data). For example, the synchronized data may pass through an interaction module (300). For example, when synchronizing the output of vibration, sound, and graphics, a synchronization operation for the output between the audio service (341), the vibration service (343), and / or the graphics framework (345) may be executed. In this case, the first data output from the audio service (341), the second data output from the vibration service (343), and the third data output from the graphics framework (345) may be interactively processed. The first data, the second data, and the third data may be synchronized and transmitted to the audio HAL (331), the vibration HAL (333), and the display HAL (335) at substantially the same time.

[0091] This allows for synchronization between the timing of sound output through the speaker (251), the timing of vibration output through the motor (241), and / or the timing of visual element output through the display (230).

[0092] FIG. 4 is a flowchart illustrating a method for providing a user interface of an electronic device according to one embodiment. The embodiment of FIG. 4 may correspond to a method for synchronizing between vibration and sound.

[0093] Referring to FIG. 4, a method for providing a user interface of an electronic device according to one embodiment may include operations 410, 420, 430, and 440. The method may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, or the processor (210)). For convenience, each operation of the method is assumed to be performed by the electronic device (200) of FIG. 2. However, it is not limited thereto. In various embodiments, at least one of the illustrated operations may be omitted, the order of some operations may be changed, or other operations may be added.

[0094] In operation 410, the electronic device (200) can detect an event.

[0095] In one embodiment, the event may be an event requiring different types of user interfaces. The event may be an event requiring physical feedback (tactile feedback and auditory feedback) in real time. The event may be an event for the output of vibration and sound.

[0096] For example, the above event may be a user input event. A user input event may be an event that occurs in response to a user's physical operation action. For example, a user input event may be a gesture interaction event. For example, a user input event may be any one of the scroll event of FIG. 11d, the touch event of FIG. 11e, the touch event of FIG. 12a and FIG. 12b, the handwriting event of FIG. 13a, the hinge event of FIG. 13b, the folding or unfolding event of FIG. 13a and FIG. 13b, the sliding event of FIG. 13c, a control event, a keyboard event, or a switching (on, off) event.

[0097] As another example, the above event may be a cue event. A cue event may be a signaling event for conveying information. For example, a cue event may be any one of a state transition event (e.g., charger connection event of FIG. 11a), an event notifying one of a warning, error, failure, or success (e.g., authentication error event of FIG. 11b), a notification event, a guidance event (e.g., guidance event of FIG. 11c), an emotional event (e.g., celebration), an alarm event, or an incoming call event.

[0098] In one embodiment, the electronic device (200) can adjust the output timing of vibration and sound for synchronization between sound and vibration with different output durations. The electronic device (200) can adjust one or more of the two elements of sound output timing and vibration output timing.

[0099] Operation 420 and operation 430 may be for synchronizing the output of vibration and sound.

[0100] The time required for sound output in the electronic device (200) (e.g., 200 to 300 ms) may be longer than the time required for vibration output (e.g., 50 ms). The vibration output procedure may be completed within a short time (e.g., 50 ms) compared to the sound output procedure. Therefore, the electronic device (200) can synchronize the output of vibration and sound by delaying the vibration and outputting the vibration in response to the sound output time.

[0101] In operation 420, the electronic device (200) may delay the output of vibration. Due to the delay, the timing of the vibration output may be shifted. For example, the electronic device (200) may control the vibration output timing (or vibration start timing) to be delayed (or shifted) by the amount of the offset by applying a specified offset (e.g., 70ms to 80ms) so that the vibration is output together with the sound output timing (or sound start timing). Accordingly, the timing of the vibration output is delayed, and the synchronization performance between the vibration and the sound may be improved.

[0102] In operation 430, the electronic device (200) can change the sound output mode from a first mode for normal (non-fast) output to a second mode for fast output while the output of vibration is delayed. Accordingly, the timing of the sound output is advanced, and the synchronization performance between vibration and sound can be improved.

[0103] In the first mode, sound can be output through a general output path (e.g., the first path (810) of FIG. 8). The general output path may be a default path designated for sound output. The general output path may include hardware components for sound output (e.g., the sound module (250) of FIG. 2, the speaker (251) of FIG. 3).

[0104] In the second mode, the sound output path may be opened in advance before the event occurs, or the sound output path may be changed from a normal output path to a high-speed output path (e.g., the second path (820) of FIG. 8).

[0105] For example, the electronic device (200) can activate the sound module (250) by opening the sound output path as the sound output mode changes to a second mode. The time at which the sound output path is opened may correspond to a time before the event occurs (e.g., the time of occurrence of a prior event).

[0106] As another example, the electronic device (200) may change the sound output path from a first path for normal output (e.g., the first path (810) in FIG. 8) to a second path for high-speed output (e.g., the second path (820) in FIG. 8) as the sound output mode is changed to a second mode. The second path may be a path that skips at least a portion of the first path or shortens the first path. Alternatively, for high-speed output, the second path may include a component different from the first path (e.g., the second buffer (825) in FIG. 8).

[0107] In one embodiment, a method of synchronizing vibration and sound by opening a sound output path to activate a sound module (250) is illustrated in more detail as follows.

[0108] The sound output procedure on the electronic device (200) may include a plurality of steps. As the sound output procedure proceeds step by step, the sound output progress state of the electronic device (200) may be switched sequentially. As the sound output procedure proceeds step by step, sound data may be sequentially transmitted from the starting end of the sound output path (e.g., application (351, 355) of FIG. 3) to the end end (e.g., sound module (250) of FIG. 2, speaker (251) of FIG. 3).

[0109] For example, the sound output progress state of the electronic device (200) can be sequentially switched to four states, namely, standby state, ready state, idle state, and playing state.

[0110] The standby state may be a state in which sound data (or sound stream) is provided and hardware elements for sound output (e.g., sound module (250), speaker (251)) are disabled.

[0111] The ready state may be a state in which sound data is prepared and hardware elements for sound output are disabled, but any operation is started in the electronic device (200) (e.g., event occurrence, power saving mode disabled).

[0112] The idle state may be a state where sound data is prepared and hardware elements for sound output are activated.

[0113] The playback state may be a state in which sound data is prepared, a hardware element for sound output is activated, and the electronic device (200) is working (e.g., a state in which sound is being played).

[0114] In this way, if the sound output procedure of the electronic device (200) proceeds in stages, it may take a long time to activate the hardware elements for sound output (e.g., sound module (250), speaker (251)). As a result, the time required for sound output (e.g., 200–300 ms) may be relatively longer than the time required for vibration output (e.g., 50 ms). The timing of sound output may be relatively delayed compared to the timing of vibration output.

[0115] The electronic device (200) can enable sound to be output immediately without the activation process for the hardware elements when necessary by activating the hardware elements (e.g., sound module (250), speaker (251)) in advance at a time earlier than the sound output time.

[0116] In operation 440, the electronic device (200) can output sound using the sound module (250) in the second mode and output delayed vibration through the haptic module (240) so that the output of sound and delayed vibration is synchronized.

[0117] As the sound output mode of the electronic device (200) changes from a first mode for general output to a second mode for high-speed output, the sound output path may be opened, or the sound output path may be changed (e.g., skipped or shortened) from a first path for general output (e.g., the first path (810) in FIG. 8) to a second path for high-speed output (e.g., the second path (820) in FIG. 8). For example, when the sound output path is opened, the sound output time may be shortened by about 100ms, making it possible to synchronize vibration and sound. For another example, when the sound output path is changed (e.g., skipped or shortened), the sound output time may be shortened by about 90ms, making it possible to synchronize vibration and sound.

[0118] In one embodiment, the electronic device (200) may change the sound output mode to a second mode for high-speed output in response to an event (or immediately upon the occurrence of the event). As the sound output mode is changed to the second mode, the sound output path is opened to activate the sound module (250), and sound may be output through the activated sound module (250). For example, the electronic device (200) may perform the operation of opening the sound output path and the operation of outputting sound through the opened sound output path in response to a point in time when the event occurs.

[0119] In another embodiment, the electronic device (200) may change the sound output mode to a second mode for high-speed output in response to a pre-event of the event. As the sound output mode is changed to the second mode in response to the pre-event, the sound output path may be opened and the sound module (250) may be activated. Subsequently, the electronic device (200) may output sound through the sound module (250) activated in response to the event. For example, the electronic device (200) may open the sound output path in response to a first time when the pre-event occurs, and then output sound through the opened sound output path in response to a second time when the event occurs.

[0120] In one embodiment, the sound output progress state in the electronic device (200) may sequentially transition from a standby state to a ready state, an idle state, and a playback state. As the sound output progress state transitions to an idle state, a signal requesting the output of delayed vibration from the sound output path to the vibration output path may be transmitted.

[0121] The sound output path transmits a signal requesting the output of delayed vibration to the vibration output path, then immediately switches to a playback state to output sound, and the vibration output path can output delayed vibration upon receiving the signal.

[0122] In this way, the electronic device (200) can achieve synchronization between vibration and sound by performing a sound output procedure that takes a relatively long time at least partially (e.g., up to an idle state) and then performing a vibration output procedure that takes a relatively short time.

[0123] In one embodiment, when the sound output mode is a first mode for normal output, a first buffer (e.g., the first buffer (815) of FIG. 8) may be used. When the sound output mode is a second mode for high-speed output, a second buffer (e.g., the second buffer (825) of FIG. 8) may be used. The second buffer may have a smaller size or latency value compared to the first buffer.

[0124] FIG. 5 is a flowchart illustrating a method for providing a user interface of an electronic device according to another embodiment. The embodiment of FIG. 5 may correspond to a method for synchronizing between vibration, sound, and visual elements.

[0125] Referring to FIG. 5, a method for providing a user interface of an electronic device according to one embodiment may include operations 510, 520, and 530. The method may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, or a processor (210)). For convenience, each operation of the method is assumed to be performed by the electronic device (200) of FIG. 2, provided that it is not limited thereto. In various embodiments, at least one of the illustrated operations may be omitted, the order of some operations may be changed, or other operations may be added.

[0126] Some of the operations of FIG. 5 may correspond to the operations of FIG. 4. For example, operation 510 may correspond to operation 410 of FIG. 4. Operation 521 may correspond to operation 420. Operations 522 and / or operation 523 may correspond to operation 430. Operation 530 may correspond to operation 440.

[0127] In operation 510, the electronic device (200) can detect an event. For example, the event may be an event for the output of vibration, sound, and visual elements.

[0128] Operation 520 may be for synchronizing the output of vibration, sound, and visual elements. Operation 520 may include at least one of operation 521, operation 522, operation 523, and operation 524.

[0129] The electronic device (200) can execute at least one of operation 521, operation 522, operation 523 and operation 524 to synchronize the output of vibration, sound and visual elements.

[0130] In operation 521, the electronic device (200) can delay the output of vibration. Due to the delay, the timing of the vibration output may be shifted.

[0131] Operations 522 and 523 may be for changing the sound output mode from a first mode for normal (non-fast) output to a second mode for fast output. For example, a sound output path may be opened as the second mode is changed. For another example, as the second mode is changed, the sound output path may be changed from a first path for normal output to a second path for fast output. The second path may be a path that skips at least a portion of the first path or shortens the first path. Alternatively, the second path may include components different from the first path.

[0132] As the sound output mode changes from the first mode to the second mode, the sound output timing is advanced, allowing for synchronization with the vibration output timing and / or the output timing of visual elements.

[0133] In operation 522, the electronic device (200) can open a sound output path to enable a hardware element for sound output (e.g., the sound module (250) of FIG. 2, or the speaker (251) of FIG. 3) for high-speed output of sound.

[0134] In operation 523, the electronic device (200) may be configured to change the sound output path from a first path for normal output to a second path for high-speed output for high-speed output of sound. For example, the second path may be a path that skips at least a portion of the first path or shortens the first path. For another example, the second path may include components different from the first path.

[0135] In operation 524, the electronic device (200) can move the timing of the output of the visual element to synchronize with vibration and sound.

[0136] In one embodiment, the visual element may include one or more of a graphic element (e.g., animation) and a dimming element. For example, the graphic element may include any kind of element that can be output (or displayed) through the display (230), such as an animation, an icon, a menu, a message window, an image, or a screen. The dimming element may be an element that can be expressed through dimming control of the lighting of the display (230). For example, the dimming element may be an element that implements screen turn-on, screen brightness level-up, or screen flickering of the display (230).

[0137] For example, the electronic device (200) can shift (e.g., delay or advance) the output timing of the visual element to correspond to the vibration output timing and / or sound output timing.

[0138] In one embodiment, the timing of the output of a visual element may correspond to the timing of the generation of a triggering signal for the output of the visual element. The electronic device (200) may generate a triggering signal for the output of a visual element at the timing of the vibration output and / or the sound output. For example, a visual element (e.g., an animation) may be displayed through a display (230) in response to the triggering signal.

[0139] In one embodiment, the electronic device (200) can control brightness (or screen brightness) by controlling the dimming of the display (230) in addition to displaying visual elements through the display (230). For example, in response to a triggering signal, the display of visual elements through the display (230) and the control of the brightness of the display (230) can be performed simultaneously.

[0140] In one embodiment, the electronic device (200) delays the output of vibration through operation 521, and while the output of vibration is delayed, it may execute at least one of operation 522, operation 523 and operation 524 to synchronize the output time of sound output and the output time of visual elements with the output time of vibration output.

[0141] In operation 530, the electronic device (200) can output sound using the sound module (250) in a second mode for high-speed output, output delayed vibration through the haptic module (240), and output visual elements through the display (230) in response to the shifted output time so that the output of sound, delayed vibration, and visual elements is synchronized.

[0142] FIG. 6 is a timing diagram illustrating the output results of vibration and sound in an electronic device according to one embodiment.

[0143] Reference numeral 610 is a timing diagram illustrating the output results of vibration and sound in an electronic device according to a comparative example. Reference numeral 611 is the output waveform of vibration. Reference numeral 612 is the output waveform of sound.

[0144] In the timing diagram of 610, TA1 is the first time point, which is the time when the event occurs. TA2 is the second time point, which is the time when the vibration starts. TA3 or TA3' is the third time point, which is the time when the sound starts.

[0145] According to the comparative example, the vibration start time (TA2) and the sound start time (TA3 or TA3') are different, so a deviation may occur between the two times, and as a result, vibration and sound may be output asynchronously. The sound output time (e.g., 200ms to 300ms) may correspond to the time interval from the first time point (TA1), which is the time of event occurrence, to the third time point (TA3 or TA3'), which is the time of sound start. The vibration output time (e.g., 50ms) may correspond to the time interval from the first time point (TA1), which is the time of event occurrence, to the second time point (TA2), which is the time of vibration start. The vibration output time may be shorter than the sound output time.

[0146] There may also be a deviation (e.g., 100ms) between sound output times (TA3, TA3'). For example, when the electronic device is in a first state where music is not playing, the sound output time may be TA3. When the electronic device is in a second state where music is playing, the sound output time may be TA3', which is faster than the first state.

[0147] The sound output progress state of the electronic device can be divided into a standby state (621), a ready state (622), an idle state (623), and a playback state (624). As illustrated, the output of vibration and sound can be controlled sequentially when an event occurs. In this case, the intervals in the ready state (622) and the idle state (623) prior to sound output may be extended, resulting in a deviation in output between vibration and sound.

[0148] According to the comparative example, when an event occurs, vibration and sound may be output separately or controlled individually without mutual interaction. The time taken for vibration output (e.g., 50ms) may be shorter than the time taken for sound output (e.g., 200~300ms) by more than a certain value, so a large output deviation may occur. A deviation may occur between the time of vibration output (TA2) and the time of sound output (TA3 or TA3').

[0149] As described, if an event occurs at the first time point (TA1), vibration may be output at the second time point (TA2) (e.g., within TA1+50ms), and then sound may be started at the third time point (TA3 or TA3'). There may be a certain deviation between the vibration end time and the sound start time (TA3), or vibration and sound may be output asynchronously.

[0150] Reference numeral 650 is a timing diagram illustrating the output results of vibration and sound in an electronic device (200) according to one embodiment. Reference numeral 661 is the output waveform of vibration. Reference numeral 662 is the output waveform of sound.

[0151] In the timing diagram of reference numeral 650, TB1 is the first time point, which is the time point of event occurrence. TB2 is the second time point, which is the time point of vibration output (or vibration start). TB3 is the third time point, which is the time point of sound output (or sound start).

[0152] When an event occurs at the first time point (TB1), the vibration may be delayed by an offset (e.g., 70ms to 80ms) specified at the second time point (TA2) of reference numeral 610 and output starting from the second time point (TB2). Sound may be output simultaneously with the vibration. The output of sound may begin at the third time point (TB3) synchronized with the second time point (TB2). The third time point (TB3) may be substantially the same time point as the second time point (TB2) (e.g., within TB2 ± 30ms).

[0153] According to one embodiment, vibration and sound can be controlled to interact with each other when an event occurs. The electronic device (200) can be controlled so that there is no substantial difference between the vibration output time (TB2) and the sound output time (TB3).

[0154] The sound output progress status of the electronic device (200) can be divided into a standby state (671), a ready state (672), an idle state (673), and a playback state (674).

[0155] For example, the standby state (671) may be a state prior to the occurrence of an event. The ready state (672) and the idle state (673) may be states that are in preparation for the output of sound. The idle state (673) may exist for a certain period of time (e.g., 10ms) or may not exist substantially. The playback state (674) may be a state in which sound is output.

[0156] For example, the standby state (671) may be a state in which sound data (or sound stream) is provided and hardware elements for sound output (e.g., sound module (250), speaker (251)) are disabled. The ready state (672) may be a state in which sound data is provided and hardware elements for sound output are disabled, but any operation is started in the electronic device (200) (e.g., event occurrence, wake-up). The idle state (673) may be a state in which sound data is provided and hardware elements for sound output are enabled. The playback state (674) may be a state in which sound data is provided and hardware elements for sound output are enabled, and the electronic device (200) is in operation (e.g., playing sound).

[0157] According to one embodiment, the output of vibration and sound can be controlled in parallel when an event occurs. The interval corresponding to the ready state (672) and / or idle state (673) can be shortened so that the output difference between vibration and sound can be eliminated.

[0158] According to one embodiment, the electronic device (200) may delay the timing of the vibration output by allocating a delay time (e.g., 70ms to 80ms) for the vibration output for synchronization. For example, the timing of the vibration output may be moved (delayed) from the second timing point (TA2) of reference numeral 610 to the second timing point (TB2) of reference numeral 650.

[0159] Alternatively, the electronic device (200) can reduce the time required for sound output by advancing the sound output time (TB3). For example, the sound output time can be moved (advanced) from the third time point (TA3 or TA3') of reference numeral 610 to the third time point (TB3) of reference numeral 650.

[0160] According to one embodiment, when an event occurs at a first time point (TB1), the sound may be output at a time point advanced by a specified offset. For example, it may be output from a third time point (TB3) of reference numeral 650 by an offset (e.g., 90ms to 100ms) advanced from a third time point (TA3 or TA3') of reference numeral 610. For example, to advance the sound output time point (TB3), the electronic device (200) may open a sound output path (e.g., operation 430 of FIG. 4, or operation 522 of FIG. 5), or change the sound output path from a first path for general sound output to a second path for high-speed sound output (e.g., operation 430 of FIG. 4, or operation 523 of FIG. 5). For example, the sound output timing may be shortened by about 100ms through operation 522 of FIG. 5, which opens the sound output path, or the sound output timing may be shortened by about 90ms through operation 523 of FIG. 5, which changes the sound output path. As an example, the second path for high-speed output may be a path that skips at least a part of the first path for normal output or shortens the first path. As another example, the second path may include components different from the first path.

[0161] Accordingly, vibration and sound, which are different types of interfaces, can be synchronized when an event occurs.

[0162] FIG. 7 is a timing diagram illustrating the output results of vibration, sound, and visual elements in an electronic device according to one embodiment.

[0163] Reference numeral 710 is a timing diagram illustrating the output results of vibration, sound, and visual elements in an electronic device according to a comparative example. Reference numeral 711 is the output waveform of vibration. Reference numeral 712 is the output waveform of sound. Reference numeral 713 is the output waveform of a signal for a visual element.

[0164] In the timing diagram of reference numeral 710, TA1 is the first time point, which is the time point of event occurrence. TA2 is the second time point, which is the time point of vibration output (or vibration start). TA3 or TA3' is the third time point, which is the time point of sound output (or sound start). TA4 is the time point of visual element output.

[0165] According to the comparative example, the timing of vibration output (TA2), the timing of sound output (TA3 or TA3'), and the timing of visual element output (TA4) are different from each other, so there may be a deviation between the timings, and as a result, vibration, sound, and visual elements may be output asynchronously.

[0166] The electronic device may require a longer time relative to vibration for processing and / or outputting sound data. The sound output time (e.g., 200–300 ms) may be longer than the vibration output time (e.g., 50 ms). In the timing diagram of 710, the sound output time (e.g., 200–300 ms) may correspond to the time interval from the first time point (TA1) to the third time point (TA3 or TA3’).

[0167] There may also be variations between sound output times (TA3, TA3') depending on the operating state of the electronic device (e.g., music playing state or non-playing state).

[0168] The electronic device can sequentially control the output of sound and visual elements when an event occurs. The electronic device can output visual elements at a fourth time point (TA4) that has a certain deviation from the third time point (TA3), which is the time point for sound output.

[0169] According to the comparative example, as illustrated, there may be a significant deviation between the vibration output time (TA2), the sound output time (TA3), and the visual element output time (TA4).

[0170] As described, if an event occurs at the first time point (TA1), vibration may start at the second time point (TA2) (within TA1+50ms) and then sound may start at the third time point (TA3 or TA3'). Additionally, after sound starts, visual elements may be output at the fourth time point (TA4) (within TA3+100ms). There may be a discrepancy between the vibration output time point (TA2), the sound output time point (TA3 or TA3'), and the visual element output time point (TA4).

[0171] Reference numeral 750 is a timing diagram illustrating the output results of vibration, sound, and visual elements in an electronic device (200) according to one embodiment. Reference numeral 761 is the output waveform of vibration. Reference numeral 762 is the output waveform of sound. Reference numeral 763 is the output waveform of a visual element (or visual signal).

[0172] The sound output progress state of the electronic device (200) can be divided into a standby state (771), a ready state (772), an idle state (773), and a playback state (774). For example, the standby state (771), ready state (772), idle state (773), and playback state (774) of FIG. 7 may correspond to the standby state (671), ready state (672), idle state (673), and playback state (674) of FIG. 6, respectively.

[0173] According to one embodiment, when an event occurs at a first time point (TB1), the vibration may be delayed by an offset (e.g., 70ms to 80ms) specified at a second time point (TA2) of reference numeral 710 and output from the second time point (TB2). Sound and visual elements may be output simultaneously with the vibration. Synchronized with the second time point (TB2), the output of sound may begin at a third time point (TB3) and the output of visual elements may begin at a fourth time point (TB4). The third time point (TB3) and the fourth time point (TB4) may be substantially the same time point as the second time point (TB2) (e.g., within TB2 ± 30ms).

[0174] The time required for vibration output may correspond to the time interval from the first time point (TB1) to the second time point (TB2). The time required for sound output may correspond to the time interval from the first time point (TB1) to the third time point (TB3). The time required for visual element output may correspond to the time interval from the first time point (TB1) to the fourth time point (TB4).

[0175] The electronic device (200) can control the output time of vibration, sound, and visual elements (TB1~TB2, TB1~TB3, TB1~TB4) to be substantially the same.

[0176] According to one embodiment, the output of vibration, sound, and visual elements can be controlled in parallel when an event occurs. The interval during which the electronic device (200) is in a ready state (772) and / or an idle state (773) is shortened so that the output deviation between the vibration, sound, and visual elements can be eliminated.

[0177] According to one embodiment, the electronic device (200) may delay the timing of the vibration output by allocating a delay time (e.g., 70ms to 80ms) for the vibration output. For example, the electronic device (200) may move (e.g., delay) the vibration output timing (TB2) from the second timing (TA2) of reference numeral 710 to the second timing (TB2) of reference numeral 750 to synchronize the vibration output timing (TB2) with the sound output timing (TB3) and the visual element output timing (TB4).

[0178] Additionally, the electronic device (200) may advance the sound output time (TB3) to correspond to the vibration output time (TB2). To advance the sound output time (TB3), the electronic device (200) may open a sound output path or change the sound output path from a first path for general sound output to a second path for high-speed sound output. For example, the electronic device (200) may move (e.g., advance) the sound output time (TB3) from the third time (TA3) of reference numeral 710 to the third time (TB3) of reference numeral 750 to synchronize the sound output time (TB3) with the vibration output time (TB2) and the visual element output time (TB4).

[0179] Additionally, the electronic device (200) may shift the output time of the visual element (TB4) to correspond to the sound output time (TB3). The sound output time (TB3) and the output time of the visual element (TB4) may be substantially the same or within a certain offset (e.g., ±30ms). For example, the output time of the visual element (TB4) may correspond to the time of occurrence of the triggering signal for the output of the visual element. The electronic device (200) may shift (e.g., advance) the time of occurrence of the triggering signal for the output of the visual element from the fourth time (TA4) of reference numeral 710 to the fourth time (TB4) of reference numeral 750.

[0180] Accordingly, as described, the deviation between the timing of vibration output, sound output, and visual element output can be improved. When an event occurs, synchronization between different types of interfaces, such as vibration, sound, and visual elements, can be implemented.

[0181] Synchronization operations between vibration, sound, and visual elements, which are different types of interfaces in an electronic device, are not limited to the illustrated vibration output timing, sound output timing, and visual element output timing. According to another embodiment, if the vibration output timing occurs later than the sound output timing, the sound output timing may be shifted (e.g., delayed) to match the vibration output timing. For example, if the vibration output timing is delayed due to the preparation operation of the haptic module (240), the interval in the idle state (773) may be increased to achieve synchronization between the vibration output timing and the sound output timing. In one embodiment, the preparation operation of the haptic module (240) may include delaying signal processing to the interaction module (300) from at least some of the system vibrator (344) or vibration service (343) of FIG. 3.

[0182] FIG. 8 is a flowchart illustrating an operation to switch a sound output mode in an electronic device according to one embodiment. For example, the operation of FIG. 8 may correspond to operation 430 of FIG. 4 or operation 523 of FIG. 5.

[0183] In one embodiment, the electronic device (200) can output sound using either a first path (810) or a second path (820).

[0184] The first path (810) may be a non-fast output path. The second path (820) may be a fast output path for sound. For example, the second path (820) may skip at least a portion of the first path (810) (e.g., the audio plunger (802) in FIG. 8) or shorten the non-fast output path. For another example, the second path (820) may include a component different from the first path (810) for fast output of sound (e.g., the second buffer (825) in FIG. 8, which is a buffer for fast output).

[0185] The electronic device (200) can reduce the time required for sound output by changing the first path (810) to the second path (820) to synchronize the sound with vibration and / or visual elements.

[0186] In the example of FIG. 8, the first path (810) may include a media decoder (801), an audio flinger (802), an audio HAL (803), and an audio object (804). For example, the audio object (804) may correspond to a hardware element for sound output (e.g., the sound module (250) of FIG. 2 or the speaker (251) of FIG. 3). The media decoder (801), the audio flinger (802), and the audio HAL (803) are software and / or firmware elements that can process sound and output it through the audio object (804). The first path (810) may include a first buffer (815). The first buffer (815) may be a buffer for general output.

[0187] The second path (820) may be a path that skips at least a part of the first path (810) (e.g., audio plunger (802)) or a path that shortens the first path (810).

[0188] The second path (820) may include a second buffer (825). The second buffer (825) may be a buffer for high-speed output.

[0189] In the case of the first path (810), the logic of the resampler (811) and the effector (812) is performed at the audio plunger (802) stage, and time may be required to perform said logic. Additionally, the size of the first buffer (815) that stores sound data within the first path (810) (e.g., 3,840 frames (48 kHz × 20 ms × 4)) may be larger than the size of the second buffer (825) used in the second path (820) (e.g., 384 frames (48 kHz × 4 ms × 2)). As a result, in the case of a normal output using the first path (810), a long time may be required for processing and / or outputting sound data compared to a high-speed output using the second path (820).

[0190] When changing the sound output path from the first path (810) to the second path (820), the logic execution time of the resampler (811) and effector (812) and / or the logic execution time according to the buffer size may be shortened.

[0191] The audio plunger (802) may include a resampler (811) and an effector (812). The audio HAL (803) may correspond to the audio HAL (331) of FIG. 2. The audio object (804) may correspond to the sound module (250) of FIG. 2 or the speaker (251) of FIG. 3.

[0192] In one embodiment, an audio object (804) may include a first buffer (815) and a second buffer (825). The second buffer (825) may have a smaller size and / or latency compared to the first buffer (815). For example, the size and / or latency of the first buffer (815) may be between 2 and 10 times the size and / or latency of the second buffer (825). If the size of the second buffer (825) is less than 2 times the size of the first buffer (815), the time reduction effect may be reduced. If the size of the second buffer (825) exceeds 10 times the size of the first buffer (815), resource consumption may be excessive.

[0193] In the example of FIG. 8, the size of the first buffer (815) may be 3,840 frames (48 kHz × 20 ms × 4). When the first buffer (815) is empty, one cell (20 ms × 1) must be filled to be output. In this case, the latency of the first buffer (815) used in the first path (810) for normal output may be 20 ms. The size of the second buffer (825) may be 384 frames (48 kHz × 4 ms × 2). When the second buffer (825) is empty, one or two cells (4 ms × 2) must be filled to be output. In this case, the latency of the second buffer (825) used in the second path (820) for high-speed output may be 4 ms or 8 ms.

[0194] When performing a normal output, sound may be output through a first path (810). For example, the first path may be a path including an audio plunger (802). The audio plunger (802) may be for resampling sound data and / or effect processing. Sound data may be transmitted from a media decoder (801) to a resampler (811). The resampler (811) may resample sound data of a first frequency (e.g., 44.1 kHz) transmitted from the media decoder (801) to a second frequency (e.g., 48 kHz) and transmit it to an effector (812). The effector (812) may process the received sound data with effects (e.g., sound alive, sound booster) and store it in a first buffer (815). For example, if the first buffer (815) is empty, sound data may be output as one cell of the first buffer (815) is filled. Accordingly, it may take more than 20ms for sound output.

[0195] When high-speed output is used, sound can be output through a second path (820). For example, the high-speed sound output path may be a path that skips the audio plunger (802). Sound data can be transmitted directly from the media decoder (801) to the second buffer (825) without passing through the audio plunger (802). For example, if the second buffer (825) is empty, sound data can be output as one or two cells of the second buffer (825) are filled. Accordingly, it may take 4ms or 8ms or more of time for sound output.

[0196] When using the second path (820), the time required for sound output is shortened compared to the first path (810), so the sound output time can be brought forward.

[0197] For example, if the sound output path is changed from a first path (810) for normal output to a second path (820) for high-speed output, the sound output time may be reduced by a certain value (e.g., 90ms).

[0198] In one embodiment, the electronic device (200) may change the sound output mode from a first mode to a second mode in response to an event. For example, the second mode may be for changing the sound output path from a first path for general output to a second path for high-speed output. As the sound output mode is changed to the second mode, the sound output path may be changed from a first path (810) for general output to a second path (820) for high-speed output.

[0199] FIG. 9 is a flowchart illustrating an operation of opening a sound output path in an electronic device according to one embodiment. For example, the operation of FIG. 8 may correspond to operation 430 of FIG. 4 or operation 522 of FIG. 5.

[0200] In one embodiment, the electronic device (200) (or the sound output path of the electronic device (200)) may include an application (806), an audio service (807), a media decoder (801), an audio plunger (802), an audio HAL (803), an audio driver (808), and an audio object (804).

[0201] For example, an application (806) may be included in the application layer (350) of FIG. 3. The application (806) may correspond to the applications (351, 355) of FIG. 3. An audio service (807) may correspond to the audio service (341) of FIG. 3. A media decoder (801) and an audio plunger (802) may be included in the interaction module (300) of FIG. 3. An audio HAL (803), an audio driver (808), and an audio object (804) may correspond to the audio HAL (331), an audio driver (321), and a speaker (251) of FIG. 3, respectively.

[0202] The electronic device (200) can perform operation 901 and operation 905.

[0203] Operation 901 may be a sound playback preliminary operation. For example, the electronic device (200) may perform operation 901 in a state prior to sound playback (e.g., the ready state (672) and / or idle state (673) of FIG. 6).

[0204] Operation 905 may be a sound playback operation. For example, the electronic device (200) may perform operation 905 in a playback state (e.g., playback state (674) of FIG. 6).

[0205] If the sound playback preliminary operation (901) is performed in advance, the sound playback operation (905) can be performed immediately. According to this, the time (e.g., 100ms) for the sound playback preliminary operation (901) is unnecessary, and the time required for sound output is reduced, so the synchronization performance between the sound and other interfaces (vibration and / or visual elements) can be improved.

[0206] Operation 901 may include operations 910 and 930. Operation 910 may be an operation that opens a sound output path for sound output. Operation 930 may be an operation that activates an audio object (804), which is a hardware element, for sound output. As operation 930 is performed, the audio object (804) may be switched from an inactive state to an active state.

[0207] According to one embodiment, by performing operation 910 and operation 930 in parallel, both operations can be completed within a set time (e.g., within 50ms).

[0208] Referring to FIG. 9, in operation 911, the application (806) can transmit a command (setFastAudioOpenMode) to the audio service (807) to change the sound output mode from a first mode for normal output to a second mode for high-speed output. The command (setFastAudioOpenMode) may be an open command for the sound output path.

[0209] In operation 912, the audio service (807) can send a play command (play) to the media decoder (801).

[0210] In operation 913, the media decoder (801) can send a dispatch command to the audio plunger (802).

[0211] Operation 910 can be performed in response to a dispatch command.

[0212] Operation 910 may include operations 914 to 923.

[0213] In operation 914, the audio plunger (802) can send a write command to the audio HAL (803). In operation 915, the audio HAL (803) can send a path open command (pcm_open) to the audio driver (808). In operation 916, the audio driver (808) can send a load command (load_dsp) to the audio object (804). In operation 920, the audio object (804) can respond to the load command (load_dsp) by setting the buffer to be used as a high-speed output buffer (e.g., the second buffer (825) of FIG. 8) and then send a return response to the audio driver (808). In operations 921 and 922, the audio driver (808) may perform a loading operation to load firmware functions (or functions) required for sound output according to the return response, and then transmit the return response to the audio HAL (803). In operation 923, the audio HAL (803) may transmit the return response to the audio plunger (802).

[0214] The electronic device (200) can perform operation 930.

[0215] Operation 930 may include operations 931 to 945.

[0216] In operation 931, the audio plunger (802) may send an init_output_device command to the audio HAL (803). In operation 932, the audio HAL (803) may send an init_device command to the audio object (804). In operation 941, the audio object (804) may send a return response to the audio HAL (803) after transitioning from an inactive state to an active state. In operation 942, the audio HAL (803) may perform a dispatch operation, and in operation 943, may send a config_codec command to the audio driver (808). In operation 944, the audio driver (808) may send a return response to the audio HAL (803). In operation 945, the audio HAL (803) can transmit a return response to the audio plunger (802).

[0217] The electronic device (200) can perform operation 905 when an event occurs. Operation 905 may be a sound playback operation.

[0218] In operations 951 and 952, a play command may be sequentially transmitted from the application (806) to the audio service (807) and the media decoder (801). In operation 953, the media decoder (801) may transmit a dispatch command to the audio plunger (802). In operation 954, the audio plunger (802) may transmit an output write command (out_write) to the audio HAL (803) to read sound data stored in the buffer. In operation 955, the audio HAL (803) may transmit a write command (pcm_write) to the audio driver (808). In operation 956, the audio driver (808) may transmit an output command (output) to the audio object (804). The audio object (804) may output (or play) sound in response to the output command (output).

[0219] For example, the electronic device (200) can perform a sound playback preliminary operation (901) in advance during a pre-event stage before the event occurs for synchronization, thereby reducing the time required for the operation (e.g., about 100ms) when the event occurs. The electronic device (200) can also perform an operation (910) to open the sound output path and / or an activation operation (930) of the audio object (804) in advance during the interval before the sound output point. By performing the opening of the sound output path and / or hardware activation logic, which are essential processes for playing the sound, in advance, the operation can be omitted during actual sound playback.

[0220] FIG. 10 is a flowchart illustrating an operation to delay vibration in an electronic device according to one embodiment. For example, the operation of FIG. 10 may correspond to operation 420 of FIG. 4 or operation 521 of FIG. 5.

[0221] In the example of FIG. 10, operation 1010 can be performed through an application of the electronic device (200) (e.g., application (351, 355) of FIG. 3). Operation 1020 can be performed through a sound output path. For example, the sound output path may be a path including at least some of the audio manager (342), audio service (341), interaction module (300), audio HAL (331), audio driver (321), and speaker (251) of FIG. 3, or the second path (820) of FIG. 8. Operation 1030 can be performed through a vibration output path. For example, the vibration output path may be a path including at least some of the system vibrator (344), vibration service (343), interaction module (300), vibration HAL (333), vibration driver (323), and motor (241) of FIG. 3.

[0222] The first time point (TB1) may be the time when the event occurs.

[0223] In operation 1011, the electronic device (200) can check whether an event has occurred through an application. For example, the event may be an event for outputting vibration and sound. For example, the event may be any one of an event based on user input (e.g., touch event, gesture event, scroll event), an event based on a state transition of the electronic device (200) (e.g., charger connection event), or a cues event (e.g., error event, notification event). In operation 1011, if the application of the electronic device (200) detects an event, operation 1013 may be performed. In one embodiment, in operation 1011, if the application of the electronic device (200) does not detect an event, operation 1011 may be repeated or the occurrence of an event may be waited for.

[0224] In operation 1013, the application of the electronic device (200) can transmit a playback command to a sound output path.

[0225] Operation 1020 may include operations 1021, operations 1023, operations 1025 and operations 1027.

[0226] In operation 1021, the sound output path of the electronic device (200) can proceed with the sound output procedure in response to a playback command of the application. For example, depending on the progress of the sound output procedure, the sound output progress state can be sequentially switched to a standby state, a ready state, an idle state, and a playback state.

[0227] In operation 1023, the electronic device (200) can check whether the sound output progress state has switched to an idle state (e.g., the idle state (673) of FIG. 6, or the idle state (773) of FIG. 7) through the sound output path.

[0228] For example, the idle state may be a state immediately preceding the playback state, corresponding to a state where a sound output preliminary operation (e.g., operation 901 (sound playback preliminary operation) of FIG. 9) is completed. For another example, the idle state may be a state where sound data is prepared (e.g., stored in the second buffer (825) of FIG. 8) and a hardware element for sound output (e.g., sound module (250) or speaker (251)) is activated.

[0229] When transitioning to an idle state, the sound output path can notify that the sound output preliminary operation is complete by passing a callback from operation 1025 to the vibration output path.

[0230] In one embodiment, when transitioning to an idle state, operation 1027 may be performed. In one embodiment, if the device does not transition to an idle state, the electronic device (200) may repeat operation 1023 or wait for transition to an idle state. For example, the electronic device (200) may check for stream status and check whether it has transitioned to an idle state. For example, when the sound output progress state transitions to an idle state, a callback may be passed from a sound output path (e.g., audio manager (342)) responsible for sound output (or playback) to a vibration output path (e.g., vibration service (343)) responsible for vibration output. The vibration output path (e.g., vibration service (343)) may synchronize the timing of sound and vibration output by playing vibration when the sound output progress state transitions to an idle state in response to the callback.

[0231] In operation 1027, the sound output path can play (or output) a sound corresponding to the event.

[0232] In operation 1031, as a callback is passed from the sound output path, the vibration output path of the electronic device (200) can play (or output) a vibration corresponding to the event.

[0233] The second time point (TB2) may correspond to the sound output time point. The third time point (TB3) may correspond to the vibration output time point.

[0234] The electronic device (200) can control the second time point (TB2) and the third time point (TB3) to substantially the same time point (e.g., within a specified offset (e.g., ±30ms)) by passing a callback according to the sound output progress state from the sound output path to the vibration output path. This can improve the synchronization performance between sound and vibration.

[0235] FIGS. 11a to 11e are drawings for explaining a method of providing a user interface of an electronic device according to one embodiment. FIGS. 11a to 11e may correspond to a method of providing a user interface of a smartphone-type electronic device. The events exemplified in FIGS. 11a to 11e may correspond to user input events or cues events.

[0236] Figure 11a illustrates a method of providing a user interface in response to the occurrence of cues events (e.g., charger connection events).

[0237] An electronic device (200) according to one embodiment may be a smartphone-type electronic device (201).

[0238] In FIG. 11a, a smartphone-type electronic device (201) can detect the occurrence of a charger connection event as a charger cable (1112) is connected to a charging port (1110). The charger connection event may be an event for the output of vibration, sound, and visual elements. The charger connection event may be a wireless connection method, not just the wired connection method shown.

[0239] The electronic device (201) can synchronize and output vibration (1116), sound (1115), and visual element (1117) indicating the occurrence of the charger connection event when the charger connection event occurs.

[0240] The electronic device (201) can change the sound output mode from a first mode to a second mode for high-speed output in response to a charger connection event.

[0241] For example, the electronic device (201) can open a sound output path at a first time when a charger connection event occurs to activate a sound module (e.g., the sound module (250) of FIG. 2 or the speaker (251) of FIG. 3) and then output sound (1115) through the activated sound module. The electronic device (201) can output vibration (1116) and visual elements (1117) together in response to the sound output time.

[0242] Figure 11b illustrates a method of providing a user interface based on a queue event (e.g., an authentication error event).

[0243] In FIG. 11b, a smartphone-type electronic device (201) can display a lock screen (1120). The lock screen (1120) may include a biometric authentication window (1121).

[0244] When the user performs an action for authentication (e.g., fingerprint input), a window (1126) indicating authentication failure may be displayed on the lock screen (1120).

[0245] The electronic device (201) can provide a witty user interface so that the user does not find this situation uncomfortable. The electronic device (201) can implement a harmonious interaction by synchronizing and outputting as a whole a vibration (e.g., a vibration pattern output simultaneously with a sound effect pattern), a sound (e.g., a fun sound effect pattern), and a visual element (e.g., an animation matching the sound effect pattern, a window (1126) on the lock screen (1120)) that indicates authentication failure in response to an authentication error event.

[0246] FIG. 11c illustrates a method of providing a user interface based on queue events (e.g., guidance events).

[0247] In FIG. 11c, a smartphone-type electronic device (201) can display a lock screen (1130). The lock screen (1130) may include a biometric authentication window (1131) (e.g., a fingerprint authentication menu and a face authentication menu).

[0248] When the user performs an action for authentication (e.g., fingerprint input), a window (1136) may be displayed on the lock screen (1130) indicating that authentication failed and prompting for another method of authentication (e.g., face authentication).

[0249] The electronic device (201) can provide a user interface that conveys a message in a witty and accurate manner through a window (1136) on the lock screen (1130) so that the user does not feel uncomfortable with the situation and can take an appropriate attitude in the event of authentication failure. The electronic device (201) can implement harmonious interaction by outputting synchronized vibration, sound, and visual elements (e.g., a window (1136) on the lock screen (1130)) as a whole.

[0250] Figure 11d illustrates a method of providing a user interface based on user input events (e.g., scroll events).

[0251] An electronic device (200) according to one embodiment may be a smartphone-type electronic device (201) or a smart watch-type electronic device (202).

[0252] In FIG. 11d, a smartphone-type electronic device (201) can display a first screen (1140). The first screen (1140) may include a timepicker area (1141). The electronic device (201) can detect the occurrence of a scroll event based on a user's scrolling input to the timepicker area (1141).

[0253] Alternatively, a smart watch type electronic device (202) may display a second screen (1145) including a time picker area (1146).

[0254] The occurrence of a scroll event can be detected based on the user's scrolling input to the time picker area (1141, 1146). The scroll event may be an event for outputting vibration, sound, and visual elements.

[0255] For example, the electronic device (201, 202) may open a sound output path in advance for high-speed sound output before the occurrence of a scroll event. The electronic device (201, 202) may open a sound output path in advance in response to a prior event of the scroll event. For example, the prior event may be an event (e.g., an alarm app launch event) for displaying a screen (1140, 1145) (e.g., an alarm app launch screen) containing a time picker area (1141, 1146).

[0256] As the sound output path is opened in advance before the occurrence of a scroll event, changes resulting from the user's scroll input (e.g., scroll direction, scroll speed, scroll distance, and / or at least some of scroll pressure) can be reflected in the output sound in real time.

[0257] In addition, through synchronization between vibration, sound, and visual elements during the user's scrolling action, realistic physical properties (acceleration, friction, sound, dynamic tactile sensation) can be expressed or an interesting emotional experience can be delivered.

[0258] FIG. 11e illustrates a method of providing a user interface based on user input events (e.g., touch events).

[0259] In FIG. 11e, a smartphone-type electronic device (201) can display a screen (1150) including a refresh menu (1151).

[0260] When there is user input (e.g., touch, tap) for the refresh menu (1151), the electronic device (201) may respond to said user input to perform loading preparation and then provide a clear user interface that indicates that loading preparation is complete. The electronic device (201) may implement harmonious interaction by outputting synchronized vibration, sound, and visual elements as a single unit.

[0261] FIGS. 12a and 12b are drawings for explaining a method of providing a user interface of an electronic device according to another embodiment. FIGS. 12a and 12b may correspond to a method of providing a user interface of a foldable electronic device. The events exemplified in FIGS. 12a and 12b may correspond to user input events.

[0262] An electronic device (200) according to one embodiment may be a foldable electronic device (205).

[0263] In FIG. 12a, the foldable electronic device (205) can display a home screen (1210) containing icons and / or widgets.

[0264] The foldable electronic device (205) can detect user input (1211) for an icon (e.g., tap and long press) or user input (1215) for a widget (e.g., tap and long press).

[0265] The foldable electronic device (205) can output synchronized vibration, sound, and visual elements in real time in response to user input (1211, 1215) to create the effect of icons or widgets actually (physically) falling off on the home screen (1210). Accordingly, a vivid and realistic emotional experience can be delivered to the user.

[0266] In FIG. 12b, the foldable electronic device (205) can display a home screen (1220) that includes a split view.

[0267] The foldable electronic device (205) can detect user input (1221, 1225) (e.g., drag) for the split view.

[0268] The foldable electronic device (205) can output synchronized vibration, sound, and visual elements in real time in response to user input (1221, 1225) to create the effect of the split view actually moving on the home screen (1220).

[0269] FIGS. 13a to 13c are drawings for explaining a method of providing a user interface for electronic devices according to another embodiment. FIGS. 13a to 13c may correspond to a method of providing a user interface for an electronic device of the type of flexible smartphone having a variable form factor (e.g., a foldable electronic device (206, 207) or a sliderable electronic device (208)). The events exemplified in FIGS. 13a to 13c may be user input events. For example, the event may be an event for a change in the mechanical state (or usage state) of the foldable electronic device (206, 207) or the sliderable electronic device (208).

[0270] An electronic device (200) according to one embodiment may be a foldable electronic device (206, 207) or a sliderable electronic device (208).

[0271] In FIG. 13a, the foldable electronic device (206) may be capable of changing its mechanical state according to the user's folding motion. For example, the foldable electronic device (206) may have any one of a folded state, a partially unfolded state, or an unfolded state as the folding angle varies.

[0272] The foldable electronic device (206) may include a flexible display (1310). The flexible display (1310) may include a first region (1311) and a second region (1312).

[0273] The foldable electronic device (206) may be in a situation where it must provide different types of user interfaces in response to the interaction of the stylus pen (1320).

[0274] When a user performs a handwriting action using a stylus pen (1320) in a partially unfolded state of a foldable electronic device (206) as exemplified in FIG. 13a, a user input event (e.g., a writing event) can be detected.

[0275] A foldable electronic device (206) can display a visual element (1325) corresponding to a user input event (e.g., a write event) through a flexible display (1310). The foldable electronic device (206) can output synchronized vibration, sound, and visual elements (1325) in real time in response to the user input event.

[0276] For example, the foldable electronic device (206) may open a sound output path in advance for high-speed sound output before the occurrence of a write event. The foldable electronic device (206) may open a sound output path in advance in response to a prior event of the write event (or in response to a first time point in time when the prior event occurred).

[0277] For example, the stylus pen (1320) may include a push-pull button for turning on and turning off. For example, the stylus pen (1320) may be turned on when the push-pull button is pressed to the push state. The stylus pen (1320) may be turned off when the push-pull button is pressed again to the pull state.

[0278] For example, a preliminary event may be an event that turns on the stylus pen (1320). When the user switches the stylus pen (1320) from the off state to the on state, it may be an intention to use the handwriting function. Therefore, the foldable electronic device (206) may open a sound output path in advance in response to the turn-on event of the stylus pen (1320) (or in response to a first time point when the turn-on event occurs) to switch a hardware element for sound output (e.g., the speaker (251) of FIG. 3) to an active state. Accordingly, the time required for sound output is shortened, making high-speed sound output possible. Subsequently, the foldable electronic device (206) may output vibration, sound, and visual elements synchronized with the user's writing input in response to a writing event (or in response to a second time point when the writing event occurs). As the sound output path is opened in advance before the writing event actually occurs, changes resulting from the user's handwriting input (e.g., gestures) are reflected in real-time in the output vibration, sound, and visual elements, enabling realistic expression.

[0279] FIG. 13b may correspond to a method of providing a user interface for a foldable electronic device (207) capable of multi-folding (or Z-folding). The foldable electronic device (207) may include a flexible display (1340). The flexible display (1340) may include a first region (1341), a second region (1342), and a third region (1343).

[0280] In FIG. 13b, the foldable electronic device (207) may be capable of changing its mechanical state according to the user's folding motion. For example, the foldable electronic device (207) may have any one of a full folded state, a partial folded state, and a full unfolded state as the folding angle varies.

[0281] In a folded state resulting from an out-folding operation, the first region (1341) and the second region (1342) of the flexible display (1340) can be folded so that they face in opposite directions. In a folded state resulting from an in-folding operation, the second region (1342) and the third region (1343) of the flexible display (1340) can be folded so that they face each other.

[0282] The mechanical state of the foldable electronic device (207) may be changed in correspondence with the folding state of the first folding unit (1331) and the second folding unit (1332). For example, the mechanical state of the foldable electronic device (207) may be a fully folded state, i.e., having the narrowest width, which is folded around the first folding unit (1331) and the second folding unit (1332). For another example, the mechanical state of the foldable electronic device (207) may be a partially folded state according to the intermediate folding state of the first folding unit (1331) and the second folding unit (1332), as illustrated in FIG. 13b. For yet another example, the mechanical state of the foldable electronic device (207) may be a fully unfolded state, i.e., having the widest width, which is when the first folding unit (1331) and the second folding unit (1332) are unfolded.

[0283] The foldable electronic device (207) can provide different types of user interfaces in response to user actions that change the mechanical state of the foldable electronic device (207).

[0284] The foldable electronic device (207) can detect a first event that changes the mechanical state (e.g., a hinge event for folding and / or unfolding) and output synchronized vibration, sound, and visual elements in real time in response to the first event.

[0285] For example, a foldable electronic device (207) may open a sound output path in response to a first event (e.g., a hinge event) and adjust the timing of the vibration output and / or the output of a visual element to correspond to the timing of the sound output. Accordingly, the output of vibration, sound and / or visual elements may be synchronized.

[0286] As another example, the foldable electronic device (207) may open a sound output path in response to a second event, which is a prior event to a first event (e.g., a hinge event), and output synchronized vibration, sound, and / or visual elements in response to the second event. For example, the prior event to the first event (e.g., the second event) may be a grip event resulting from a user grasping both sides of the foldable electronic device (207). For example, the action of a user grasping both sides of the foldable electronic device (207) may be an intention to fold and / or unfold the foldable electronic device (207). Therefore, the foldable electronic device (207) may open a sound output path in advance in response to the grip event (or in response to a first time when the grip event occurs) to keep hardware for sound output (e.g., a speaker (251)) in an active state. Accordingly, the time required for sound output may be shortened, enabling high-speed sound output. Subsequently, the foldable electronic device (207) can output vibrations, sound, and / or visual elements synchronized with the user's folding and / or unfolding actions in response to a hinge event (or in response to a second time point when the hinge event occurs). Since the sound output path is opened in advance before the hinge event actually occurs, changes resulting from the user's folding and / or unfolding actions for the hinge event can be reflected in real time in the output vibrations, sound, and / or visual elements, making realistic expression possible.

[0287] FIG. 13c may correspond to a method of providing a user interface for a slideable electronic device (205).

[0288] In FIG. 13c, the slideable electronic device (205) may be capable of changing its mechanical state according to the sliding action of the user. Reference numeral 1350 indicates the basic state (or reduced state, slide-in state) of the slideable electronic device (205). Reference numeral 1360 indicates the extended state (or slide-out state) of the slideable electronic device (205).

[0289] The slideable electronic device (205) may include a flexible display (1370). The flexible display (1370) may include a first region (1371) and a second region (1372). The first region (1371) may always be exposed to the outside of the electronic device (205). The second region (1372) may be an area extending from the first region (1371) that can be partially retracted into the interior of the slideable electronic device (205) by a sliding-in motion and partially withdrawn to the outside of the slideable electronic device (205) by a sliding-out motion.

[0290] The slideable electronic device (205) can output synchronized vibration, sound, and visual elements as a single unit to enable harmonious interaction in real time according to the user's sliding action.

[0292] An electronic device according to various embodiments (e.g., the electronic device (200) of FIG. 2) may include a memory (e.g., the memory (220) of FIG. 2), a display (e.g., the display (230) of FIG. 2), a haptic module (e.g., the haptic module (240) of FIG. 2), a sound module (e.g., the sound module (250) of FIG. 2), and at least one processor (e.g., the processor (210) of FIG. 2). The at least one processor may be operatively connected to the memory, the display, the haptic module, and the sound module. The memory may store instructions such that, at the time of execution, at least one processor detects an event for outputting vibration and sound, delays the output of the vibration, changes the sound output mode from a first mode to a second mode for high-speed output while the output of the vibration is delayed, outputs the sound using the sound module in the second mode, and outputs the delayed vibration through the haptic module so that the output of the sound and the output of the delayed vibration are synchronized.

[0293] According to various embodiments, as the sound output mode is changed to the second mode, the sound output path is opened and the sound module can be activated.

[0294] According to various embodiments, as the sound output mode is changed to the second mode, the sound output path may be changed from a first path for normal output to a second path for high-speed output.

[0295] According to various embodiments, the memory may additionally store instructions in which, at the time of execution, the at least one processor outputs a visual element through the display, and shifts the timing of the output of the visual element so that the sound, the delayed vibration, and the output of the visual element are synchronized.

[0296] According to various embodiments, the sound output mode may be changed to the second mode in response to the event.

[0297] According to various embodiments, as the sound output mode is changed to the second mode, the sound output path may be changed from a first path for normal output to a second path for high-speed output.

[0298] According to various embodiments, the sound output mode may be changed to the second mode in response to a pre-event of the event.

[0299] According to various embodiments, as the sound output mode is changed to the second mode in response to the prior event, the sound output path may be opened and the sound module may be activated. In response to the event, the sound may be output through the activated sound module.

[0300] According to various embodiments, the sound output progress state may sequentially transition from a standby state to a ready state, an idle state, and a playback state. As the sound output progress state transitions to the idle state, a signal requesting the output of the delayed vibration may be transmitted from the sound output path to the vibration output path.

[0301] According to various embodiments, a first buffer may be used when the sound output mode is the first mode, and a second buffer may be used for high-speed output when the sound output mode is the second mode. The second buffer may have a smaller size or latency value compared to the first buffer.

[0302] A method for providing a user interface of an electronic device according to various embodiments may include an operation of detecting an event for outputting vibration and sound, an operation of delaying the output of the vibration, an operation of changing the sound output mode from a first mode to a second mode for high-speed output while the output of the vibration is delayed, and an operation of outputting the sound in the second mode and outputting the delayed vibration so that the output of the sound and the output of the delayed vibration are synchronized.

[0303] According to various embodiments, as the sound output mode is changed to the second mode, the sound output path is opened and the sound module can be activated.

[0304] According to various embodiments, as the sound output mode is changed to the second mode, the sound output path may be changed from a first path for normal output to a second path for high-speed output.

[0305] According to various embodiments, the method may further include an operation of outputting a visual element. The method may shift the timing of the output of the visual element so that the sound, the delayed vibration, and the output of the visual element are synchronized.

[0306] According to various embodiments, the sound output mode may be changed to the second mode in response to the event.

[0307] According to various embodiments, as the sound output mode is changed to the second mode, the sound output path may be changed from a first path for normal output to a second path for high-speed output.

[0308] According to various embodiments, the sound output mode may be changed to the second mode in response to a pre-event of the event.

[0309] According to various embodiments, as the sound output mode is changed to the second mode in response to the prior event, the sound output path may be opened and the sound module may be activated. In response to the event, the sound may be output through the activated sound module.

[0310] According to various embodiments, the sound output progress state may sequentially transition from a standby state to a ready state, an idle state, and a playback state. As the sound output progress state transitions to the idle state, a signal requesting the output of the delayed vibration may be transmitted from the sound output path to the vibration output path.

[0311] According to various embodiments, a first buffer may be used when the sound output mode is the first mode, and a second buffer may be used for high-speed output when the sound output mode is the second mode. The second buffer may have a smaller size or latency value compared to the first buffer.

Claims

Claim 1 An electronic device comprising: a memory; a display; a haptic module; a sound module; and at least one processor operatively connected to the memory, the display, the haptic module, and the sound module, wherein the memory is executed by the at least one processor and stores instructions such that the electronic device detects an event for outputting vibration and sound, delays the output of the vibration, and while the output of the vibration is delayed, changes the sound output mode from a first mode to a second mode in which the time required for sound output is reduced, wherein in the first mode, the sound output path is designated as a first path, and as the sound output mode is changed to the second mode, the sound output path is changed from the first path to a second path, and the second path is a path that skips at least a part of the first path or shortens the first path, and in the second mode, outputs the sound using the sound module and outputs the delayed vibration through the haptic module so that the output of the sound and the delayed vibration are synchronized. Claim 2 An electronic device according to claim 1, wherein the instructions are executed by the at least one processor, such that the electronic device performs at least one of opening the sound output path or activating the sound module as the sound output mode is changed to the second mode. Claim 3 The electronic device according to claim 1, wherein the instructions are executed by the at least one processor so as to cause the electronic device to perform a sound output preliminary operation in response to a pre-event of the event, wherein the sound output preliminary operation comprises at least one of opening the second path or activating the sound module. Claim 4 An electronic device according to claim 1, wherein the instructions are executed by the at least one processor, such that the electronic device outputs a visual element through the display, and shifts the timing of the output of the visual element so that the sound, the delayed vibration, and the output of the visual element are synchronized. Claim 5 In paragraph 2, the instructions are executed by the at least one processor, thereby enabling the electronic device to perform the opening of the sound output path and the activation of the sound module in parallel. Claim 6 In paragraph 3, the instructions are executed by the at least one processor, thereby enabling the electronic device to perform the opening of the second path and the activation of the sound module in parallel. Claim 7 delete Claim 8 In paragraph 3, the instructions are executed by the at least one processor, so that the electronic device activates the sound module by opening the second path in response to the prior event, and outputs the sound through the activated sound module in response to the event. Claim 9 An electronic device according to claim 1, wherein the instructions are executed by the at least one processor, such that the electronic device sequentially switches the sound output progress state from a standby state to a ready state, an idle state, and a playback state, and transmits a signal requesting the output of the delayed vibration from the sound output path to the vibration output path as the sound output progress state switches to the idle state. Claim 10 An electronic device according to claim 1, wherein the first path includes a first buffer, and the second path includes a second buffer having a value of smaller size or latency compared to the first buffer. Claim 11 A method for providing a user interface of an electronic device, comprising: an operation of detecting an event for outputting vibration and sound; an operation of delaying the output of the vibration; an operation of changing the sound output mode from a first mode to a second mode in which the time required for sound output is reduced while the output of the vibration is delayed, wherein in the first mode, the sound output path is designated as a first path, and as the sound output mode is changed to the second mode, the sound output path is changed from the first path to a second path, and the second path is a path that skips at least a part of the first path or shortens the first path; and an operation of outputting the sound and the delayed vibration in the second mode so that the output of the sound and the delayed vibration are synchronized. Claim 12 A method according to claim 11, wherein the operation of changing the sound output mode from the first mode to the second mode comprises an operation of performing at least one of opening the sound output path or activating the sound module of the electronic device as the sound output mode is changed to the second mode. Claim 13 A method according to claim 11, further comprising an operation to perform a sound output preliminary operation in response to a pre-event of the above event, wherein the sound output preliminary operation comprises at least one of opening the second path or activating the sound module of the electronic device. Claim 14 A method according to claim 11, further comprising an operation of outputting a visual element, wherein the timing of the output of the visual element is shifted so that the sound, the delayed vibration, and the output of the visual element are synchronized. Claim 15 In claim 12, a method of performing the opening of the sound output path and the activation of the sound module in parallel. Claim 16 In paragraph 13, a method of performing the opening of the second path and the activation of the sound module in parallel. Claim 17 delete Claim 18 In paragraph 13, a method in which the second path is opened in response to the prior event to activate the sound module, and the sound is output through the activated sound module in response to the event. Claim 19 A method according to claim 11, wherein the sound output progress state is sequentially switched from a standby state to a ready state, an idle state, and a playback state, and as the sound output progress state is switched to the idle state, a signal requesting the output of the delayed vibration is transmitted from the sound output path to the vibration output path. Claim 20 A method according to claim 11, wherein the first path includes a first buffer, and the second path includes a second buffer having a value of smaller size or latency compared to the first buffer.

Images