Electronic device and method for improving force touch operation thereof

Abstract

An electronic device according to an embodiment may include: a display; a touch sensor; a memory; and a processor, in which the memory includes instructions configured to cause the processor to: determine, based on touch data received from the touch sensor, whether a force touch that gradually increases an amount of change in skin pressure according to a touch area is recognized within a long touch detection time; executing a force touch action function mapped to the force touch based on the force touch recognition being maintained for a predetermined period of time; learning a misoperation condition of the force touch; and displaying a sensitivity correction UI on the display when sensitivity adjustment of the force touch is required.

Description

Technical Field

[0001] Various embodiments relate to an electronic device and a method for improving a force touch operation, and more particularly, to an electronic device and a method for improving a force touch operation based on artificial intelligence. Background Art

[0002] Electronic devices including touch screens are widely used, and these electronic devices can accurately recognize touch inputs through the touch screens, thereby providing users with a satisfactory experience.

[0003] Such touch input may be classified into various types of input such as force touch, long touch, tap touch (or short touch), and multi-touch. A force touch may indicate a touch by which a user applies pressure of a predetermined intensity or more with a finger, and a long touch may indicate a touch in which pressure is maintained for a predetermined time or longer regardless of a specific pressure intensity.

[0004] There are 3D touch and haptic touch as touch input technologies for distinguishing force touch from long touch. 3D touch is a technology for recognizing pressure intensity by using a pressure sensor (eg, a hardware component), and haptic touch may be a technology for recognizing how long a user touches with the help of software. Summary of the invention

[0005] Technical issues However, in the haptic touch method, since software is used to identify how long the user has been touching, the force applied to the screen cannot be identified, and therefore it may be difficult to provide a function of taking a two-step action by pressing the screen harder. For 3D touch, the gap between the light beams and the change in the length of the display are detected to detect the force with which the user presses the screen. In this case, there should be no error in the linear display gap, the sensor required to detect the linear display gap is expensive, and adding additional sensors may cause design challenges due to the characteristics of the display (e.g., OLED). In addition, unless the user presses the exact part of the screen hard enough, it is difficult to detect 3D touch, and the user may need to rely on sensory judgment to perform force touch input and long touch input, which may be a disadvantage.

[0006] In addition, although the intensity and time of touch may vary for each user, electronic devices generally determine the pressure intensity or touch time with uniform pressure and / or fixed values, which may cause relative recognition errors in operation. In addition, no user interface (UI) is provided that can distinguish between long touch and force touch.

[0007] The embodiments of this document aim to provide a method for analyzing changes in a user's skin area (or skin pressure based on a touch area) to determine whether a force touch is recognized without a separate pressure sensor, so as to improve the recognition rate of the force touch.

[0008] The embodiments of this document aim to provide a method for dynamically learning a touch recognition area according to an appearance state and / or a holding state of an electronic device to reflect user characteristics, so as to improve the recognition rate of force touch.

[0009] The embodiments herein aim to provide a method for providing intuitive state changes of a long touch and a force touch to resolve the ambiguous distinction between the long touch and the force touch.

[0010] The embodiments herein are intended to provide a layout method of reconfiguring a screen based on application characteristics when a force touch action based on a force touch is performed to improve usability.

[0011] The technical objectives that this document aims to achieve are not limited to the above objectives, and other technical objectives that are not clearly defined herein will be clearly understood by those skilled in the art to which the present invention belongs based on the description provided below.

[0012] Solution to the problem An electronic device according to an example embodiment includes a display. An electronic device according to an example embodiment includes a touch sensor. An electronic device according to an example embodiment includes a memory. An electronic device according to an example embodiment includes a processor. The memory according to an example embodiment includes instructions configured to cause the processor to perform the following operations: determine whether a force touch is recognized based on touch data received from the touch sensor, the force touch gradually increasing the amount of change according to the skin pressure of the touch area within a long touch detection time. The memory according to an example embodiment includes instructions configured to cause the processor to perform the following operations: based on force touch recognition being maintained for a predetermined period of time, execute a force touch action function mapped to the force touch. The memory according to an example embodiment includes instructions configured to cause the processor to learn a misoperation situation of the force touch. The memory according to an example embodiment includes instructions configured to cause the processor to perform the following operations: display a sensitivity correction UI on the display when sensitivity adjustment of the force touch is required.

[0013] A method for improving a force touch operation according to an example embodiment includes receiving touch data received from a touch sensor. The method according to an example embodiment includes: based on the touch data, determining whether a force touch with a gradually increasing change in skin pressure according to a touch area is recognized within a long touch detection time. The method according to an example embodiment includes: based on recognizing the force touch, executing a force touch action function mapped to the force touch. The method according to an example embodiment includes: when recognizing the force touch, learning a misoperation situation of the force touch, and displaying a sensitivity correction UI on a display if sensitivity adjustment of the force touch is required.

[0014] Advantageous Effects of the Invention Through the electronic device and method according to the embodiments, the touch recognition area according to the shape state (e.g., shape change) and / or holding state of the electronic device can be learned, and the force touch recognition algorithm (e.g., reference area or threshold) can be dynamically changed to improve the recognition rate and usability of force touch.

[0015] Through the electronic device and method according to the embodiment, intuitive situation feedback for recognition of a long touch and a force touch may be provided to resolve ambiguous distinction between the long touch and the force touch.

[0016] Through the electronic device and method according to the embodiment, it may be determined that a force touch operation is recognized differently from the user's intention, and user feedback related to touch sensitivity adjustment may be induced to resolve erroneous operation recognition in order to improve force touch recognition strength (sensitivity).

[0017] Effects obtained from the present disclosure are not limited to those mentioned above, and other effects that are not explicitly stated herein will be clearly understood by those skilled in the art to which the present disclosure belongs based on the description provided below. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram illustrating example electronic devices in a network environment according to various embodiments.

[0019] Figure 2 (a) to Figure 2 (h) shows shapes of various external forms of electronic devices according to various embodiments.

[0020] Figure 3 A platform structure of an electronic device according to various embodiments is shown.

[0021] Figures 4a to 4c is a diagram illustrating an artificial intelligence-based force touch recognition operation according to an embodiment.

[0022] Figure 5is a diagram briefly showing a configuration of an electronic device according to an embodiment.

[0023] Figure 6 is a diagram illustrating force touch and long touch determination times according to an embodiment.

[0024] Figure 7 A method of improving an artificial intelligence-based force touch operation of an electronic device according to an embodiment is shown.

[0025] Figure 8 A screen for configuring a force touch motion function according to an embodiment is shown.

[0026] Fig. 9 A force touch management screen of an electronic device according to an embodiment is shown.

[0027] Fig.10 A force touch management screen of an electronic device according to an embodiment is shown.

[0028] Fig.11 A method of improving an artificial intelligence-based force touch operation of an electronic device according to an embodiment is shown.

[0029] Fig.12 A touch gesture processing operation according to an embodiment is shown.

[0030] Fig.13 A force touch learning method of an electronic device according to an embodiment is shown.

[0031] Fig.14 A screen showing touch sensitivity adjustment caused by force touch learning in force touch management by an electronic device according to an embodiment is shown.

[0032] Fig.15 A manual learning configuration screen for touch area learning at the time of force touch management by an electronic device according to an embodiment is shown.

[0033] Figures 16 to 20 A force touch management invention configured for each force touch action is shown according to various embodiments.

[0034] Fig.21 A screen showing an on / off configuration of supporting a force touch motion function in an electronic device according to an embodiment. DETAILED DESCRIPTION

[0035] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.

[0036] Figure 1 is a block diagram illustrating electronic devices in a network environment according to various embodiments.

[0037] Reference Figure 1 , the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the above components (e.g., the connection terminal 178) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the above-described components (eg, sensor module 176, camera module 180, or antenna module 197) may be implemented as a single integrated component (eg, display module 160.

[0038] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware component or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 120 may store a command or data received from another component (e.g., sensor module 176 or communication module 190) in the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. For example, when the electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121, or as part of the main processor 121.

[0039] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (rather than the main processor 121) may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., a camera module 180 or a communication module 190) that is functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 123 (e.g., a neural processing unit) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model may be generated through machine learning. For example, such learning may be performed by the electronic device 101 where the artificial intelligence is executed or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, for example. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q network or a combination of two or more thereof, but is not limited thereto. Additionally or optionally, the artificial intelligence model may include a software structure in addition to a hardware structure.

[0040] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0041] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0042] The input module 150 may receive commands or data to be used by other components (e.g., the processor 120) of the electronic device 101 from outside (e.g., a user) of the electronic device 101. The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0043] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive incoming calls. Depending on the embodiment, the receiver can be implemented as a separate part from the speaker, or as part of the speaker.

[0044] The display module 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display module 160 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of a force caused by a touch.

[0045] The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 may obtain sound via the input module 150, or output sound via the sound output module 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0046] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) outside the electronic device 101, and then generate an electrical signal or a data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0047] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly with an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0048] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0049] The haptic module 179 may convert the electric signal into mechanical stimulation (eg, vibration or motion) or electric stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electric stimulator.

[0050] The camera module 180 may capture still images or moving images. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0051] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0052] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0053] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multiple components (e.g., multiple chips) separated from each other. The wireless communication module 192 may identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0054] The wireless communication module 192 may support 5G networks after 4G networks and next generation communication technologies (e.g., new radio (NR) access technologies). NR access technologies may support enhanced mobile broadband (eMBB), massive machine type communications (mMTC), or ultra-reliable low latency communications (URLLC). The wireless communication module 192 may support high frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 192 may support various technologies for ensuring performance on high frequency bands, such as, for example, beamforming, massive multiple input multiple output (massive MIMO), full dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., a second network 199). According to an embodiment, the wireless communication module 192 may support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane delay for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less for a round trip).

[0055] The antenna module 197 may transmit or receive a signal or power to or from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiation element formed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module 197 may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). A signal or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiation element may be additionally formed as a part of the antenna module 197.

[0056] According to various embodiments, the antenna module 197 may form a millimeter wave antenna module. According to an embodiment, the millimeter wave antenna module may include a printed circuit board, a radio frequency integrated circuit (RFIC), and a plurality of antennas (e.g., array antennas), wherein the RFIC is disposed on a first surface (e.g., bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a designated high frequency band (e.g., millimeter wave band), and the plurality of antennas are disposed on a second surface (e.g., top surface or side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the designated high frequency band.

[0057] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transfer signals (e.g., commands or data) therebetween.

[0058] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic device 102 or the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be executed in the electronic device 101 may be executed in one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices receiving the request may execute at least part of the requested function or service, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology, or client-server computing technology, for example, may be used. The electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology or IoT-related technologies.

[0059] As used herein, each of phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B or C” may include any one or all possible combinations of items listed together with a corresponding one of the plurality of phrases. As used herein, terms such as “first” and “second” or “first” and “second” may be used to simply distinguish a corresponding component from another component and do not limit the components in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “coupled with another element (e.g., the second element)”, “coupled to another element (e.g., the second element)”, “connected with another element (e.g., the second element)”, or “connected to another element (e.g., the second element)” with or without the terms “operably” or “communicatively” being used, it means that the element may be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0060] As used in connection with various embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0061] The various embodiments described herein may be implemented as software (e.g., program 140) including one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0062] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be released in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be downloaded via an application store (e.g., Play Store TM ) the computer program product may be published (e.g., downloaded or uploaded) online, or the computer program product may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, a server of an application store, or a forwarding server).

[0063] According to various embodiments, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities, and some of the multiple entities may be separately arranged in different components. According to various embodiments, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., a module or a program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as a corresponding one of the multiple components performing one or more functions before integration. According to various embodiments, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more of the operations may be run or omitted in a different order, or one or more other operations may be added.

[0064] Figure 2 (a) to Figure 2 (h) shows shapes of various external forms of electronic devices according to various embodiments.

[0065] refer to Figure 2 (a) to Figure 2 (h), an electronic device according to various embodiments (eg, Figure 1 The electronic device 101 in the embodiment may include various types of housing structures so that a display (eg, Figure 1 The display area of ​​the display module 160 in the embodiment is variable. According to an embodiment, the electronic device 101 may include Figure 2 A strip-type or panel-type electronic device other than the type shown in FIG.

[0066] Can be changed according to the appearance Figure 2 (a) to Figure 2 The size or ratio of the display or display area of ​​the electronic device 101 shown in (h). The display area may indicate an area of ​​the display area that is exposed to the outside or activated to display visual information.

[0067] For example, Figure 2 (a) to Figure 2 As shown in (d), the electronic device 101 may be implemented as a foldable electronic device 101 that operates in a manner of folding inward, folding outward, or folding inward / outward according to the rotation of the first housing 2110 and the second housing 2115 relative to each other. In the foldable electronic device, the display area of ​​the display 2120 may be changed according to the appearance state. For example, according to the angle or distance between the first housing 2110 and the second housing 2115, the foldable electronic device may be changed to an unfolded state (or a first state), a folded state (e.g., a second state), or an intermediate state (e.g., a third state).

[0068] For another example, Figure 2 (e) and Figure 2 As shown in FIG. 2( f), the electronic device 101 may be implemented as a slidable electronic device that operates in such a manner that the second housing 2215 slides relative to the first housing 2210 to slide into or out of the first housing. In the slidable electronic device, the display area of ​​the display 2220 may be expanded or reduced according to the shape state.

[0069] For another example, Figure 2 (g) and Figure 2 As shown in (h), the electronic device 101 can be implemented as a scrollable electronic device that operates in a manner in which the second housing 2315 is scrolled relative to the first housing 2310 so as to be scrolled into or out of the first housing. In the scrollable electronic device, the display area of ​​the display 2320 can be expanded or reduced according to the form factor state.

[0070] In the case of a slidable electronic device or a rollable electronic device, there may be a closed state in which the display is not extended, an intermediate state, or an open state in which the display 2220 or 2320 is extended, depending on the direction in which the second housing 2215 or 2315 moves relative to the first housing 2210 or 2310. The open state may be defined as a state in which the display area is extended compared to the closed state, and display areas of various areas may be provided according to the movement position of the second housing 2215 or 2315.

[0071] Hereinafter, the electronic device 101 according to various embodiments of the present disclosure is shown as an example of an electronic device having a foldable type outer shape (e.g., a foldable electronic device), but the electronic device 101 according to various embodiments and operations thereof are not limited thereto. For example, the electronic device 101 may have various outer shapes (such as a bar type or a plate type, a rollable type, and / or a slidable type), and may also operate according thereto.

[0072] According to an embodiment, the electronic device 101 of various shapes may provide an artificial intelligence-based force touch management function. For example, the electronic device 101 may determine whether to recognize a force touch based on a change in skin area (or skin area / touch area according to skin pressure) based on artificial intelligence software without a physical pressure sensor.

[0073] According to an embodiment, the electronic device 101 of various forms may provide a force touch situation user interface (UI) based on force touch recognition.

[0074] According to an embodiment, the electronic device 101 of various forms may provide a function (e.g., a force touch action function) that maps actions that can be performed by force touch according to user configuration. The force touch action function may include an action function that can be specified for each application and / or a global action function applicable to the electronic device system.

[0075] According to an embodiment, the electronic device 101 of various shapes can learn the touch recognition area according to the shape state (e.g., the first state, the second state, or the intermediate state) and the holding state (e.g., held with one hand or held with two hands), and dynamically change (or adjust) the force touch recognition algorithm (e.g., the reference touch area or threshold).

[0076] According to various embodiments, electronic devices 101 of various shapes can learn whether to recognize the user's intention. Figure 1 The system can realize consistent force touch and provide a sensitivity correction user interface (UI) when the force touch sensitivity needs to be adjusted based on the learning situation.

[0077] Figure 3 A platform structure of an electronic device according to various embodiments is shown.

[0078] refer to Figure 3 , according to the electronic device of the embodiment (eg, Figure 1 The electronic device 101 in the embodiment may be based on Figure 3 The platform structure shown in the figure is used to handle the AI-based force touch management function. Figure 3 The described modules may be understood as being executed by a processor of the electronic device 101 (eg, Figure 1 The processor 120 may execute functions stored in the memory (e.g., Figure 1The instructions in the memory 130 in the memory 130 are used to implement the software modules and control the hardware associated with the functions (for example, Figure 1 1760 in the sensor module). Some modules may be implemented in hardware, while other modules may be implemented in software. Modules may be classified into an application layer, a framework layer, a hardware abstraction layer (HAL), a kernel driver layer, and / or a hardware (HW) layer, and at least some of the elements shown may be changed.

[0079] The application layer may include applications 310 .

[0080] The application 310 may be an application 310 stored in the memory 130, executable by the processor 120 or installed. The application 310 may include, for example, a force touch action application 315, application 1, application 2, application 3, a system user interface (UI), and / or various applications executable in the electronic device 101, and its type may not be limited.

[0081] The force touch action application 315 according to the embodiment may be an application that manages a force touch function based on artificial intelligence and provides interaction with a user to configure a force touch action. For example, the force touch action application 315 may be an application that can apply a force touch action configuration function, a force touch learning function (e.g., a force touch sensitivity adjustment function and a touch area adjustment function), and a function of changing a screen layout according to a force touch action. The force touch action application 315 and its management operation will be described with reference to the accompanying drawings described later.

[0082] A system user interface (UI) may manage a system of the electronic device 101 , such as a screen related to a notification bar or a quick view.

[0083] The framework layer may provide various functions to the application 310 so that the application 310 uses a function or information provided from at least one resource of the electronic device 101 .

[0084] The framework layer may include, for example, an input device manager 320 , a sensor manager 323 , a view system 325 , and an activity manager 327 , but is not limited thereto.

[0085] The input device manager 320 may determine whether a force touch is recognized based on a signal generated from the touch sensor (or touch data transmitted through the sensor manager 323), and transmit force touch recognition information (or a force touch event) to the force touch action application 315. For example, the input device manager 320 may observe whether a force touch occurs during a period (e.g., 500 ms) configured for determining a long touch, a force touch observation interval (e.g., 300 ms), and then perform pressure calculation (e.g., skin area change) on the force touch to determine whether a force touch is recognized.

[0086] The sensor manager 323 may control the sensor based on the configuration of the application 310. The sensor manager 323 may collect and control sensor information based on the availability of the sensor module. For example, when a user's touch input occurs, the sensor manager 323 may control the touch sensor to instruct the touch sensor to generate touch data. For example, the sensor manager 323 may generate touch data corresponding to the user's touch and transmit it to the input device manager 320.

[0087] The view system 325 may include a set of extensible views for creating an application user interface. According to an embodiment, the view system 325 may be a program for drawing at least one layer based on a display area resolution of a display. According to an embodiment, the application 310 may use a view (e.g., a drawing library) to draw at least one layer based on a display area resolution of a display.

[0088] The activity manager 327 may manage the life cycle of the activity. According to an embodiment, the activity manager 327 may manage the execution and termination of the application 310.

[0089] The hardware abstraction layer is a collection of hardware modules (e.g. Figure 1 An abstraction layer between the display module 160 and the sensor module 176 in the electronic device 101 and the software of the electronic device 101 , and may include an event center 330 and a surface deliverer 335 .

[0090] The event center 330 may be an interface in which events occurring in the touch circuit and the sensor circuit are standardized. The event center 330 may be included in an abstraction layer (Hardware Abstraction Layer, HAL) between a plurality of hardware modules included in the hardware layer and software of the electronic device.

[0091] The surface deliverer 335 may synthesize a plurality of layers. For example, the surface deliverer 335 may provide data indicating the synthesized plurality of layers to the display controller 345. The display controller (display driver IC, DDI) 345 may refer to a graphic display controller or a display driving circuit controller.

[0092] The kernel layer may include various drivers for controlling various hardware modules (e.g., the display module 160 and the sensor module 176) included in the electronic device 101. For example, the kernel layer may include a sensor driver 340 including an interface module for controlling a sensor controller 350 connected to the sensor module 176 and a display controller (display driver IC, DDI) 345, wherein the sensor driver 340 includes an interface module for controlling a sensor controller 350 connected to the sensor module 176, and the display controller 345 controls a display panel 355 connected to the display module 160.

[0093] The sensor driver 340 may connect the operating system to the sensor and include information about the driving method, characteristics and / or functions of the sensor. The sensor driver 340 may include an interface module that controls a sensor controller connected to the sensor. The display controller 345 may receive data representing multiple layers of the synthesis from the surface deliverer 335 and may correspond to a display driving circuit.

[0094] The hardware layer may include hardware modules or elements (eg, the sensor controller 350 and the display panel 355) included in the electronic device 101, but is not limited thereto, and may include Figure 1 The components shown in .

[0095] Figures 4a to 4c is a diagram illustrating an artificial intelligence-based force touch recognition operation according to an embodiment.

[0096] refer to Figures 4a to 4c , according to the electronic device of the embodiment (eg, Figure 1 The electronic device 101 in the present disclosure may identify force touch based on changes in skin area or changes in touch area caused by skin pressure based on artificial intelligence software. The electronic device 101 of the present disclosure may not include a physical pressure sensor.

[0097] The electronic device 101 may distinguish between a long touch and a force touch according to a change in skin pressure based on a touch area.

[0098] For example, as shown in the reference numeral <401> As indicated, it can be noted that when the user performs a long touch 410 on the touch screen, the change in skin pressure based on the touch area during the touch time (e.g., 300ms) maintains the same size after a predetermined time, and on the contrary, in the case of a force touch 420 in which the touch screen is pressed hard, the change in skin pressure based on the touch area has a gradually increasing size.

[0099] As shown in the figure <402> As indicated, when a change in skin pressure based on a touch area of ​​a touch occurring within a predetermined time increases, the electronic device may classify the touch as a force touch 420, and when the change in skin pressure based on the touch area is constant, the electronic device may classify the touch as a long touch 410.

[0100] In this case, the long touch 410 and the force touch 420 may have the same initial touch time. The electronic device 101 may observe whether the force touch 420 is generated within the long touch detection time (e.g., LT_a) used as a standard for determining the long touch 410, and then determine the force touch recognition in the force touch determination interval (FT_b). For example, the electronic device 101 may be divided into a force touch observation interval (FT_a), a force touch determination interval (FT_b), and a long touch determination interval (LT_a), and distinguish whether the touch is a long touch 410 or a force touch 420 according to the touch characteristics. As shown in FIG. <403> As indicated, the electronic device 101 may determine the second touch input as a force touch 420 because its touch pressure exceeds a reference value of touch pressure (eg, 300 gf / cm 2 ). In contrast, the electronic device 101 may not recognize the first touch input as a force touch because its touch pressure is lower than the reference value in the force touch observation interval, and may recognize the first touch input as a long touch 410 in the long touch determination interval.

[0101] Figure 5 is a diagram briefly showing a configuration of an electronic device according to an embodiment, and Figure 6 is a diagram illustrating force touch and long touch determination times according to an embodiment.

[0102] refer to Figure 5 , according to the electronic device of the embodiment (eg, Figure 1 The electronic device 101 in the embodiment may include a touch screen display 510 (eg, Figure 1 , the display module 160 in the processor 520 (for example, Figure 1 processor 120) and memory 530 (e.g., Figure 1 Memory 130 in the memory). Figure 5 The electronic device 101 may also include Figure 1 At least some of the elements and / or functions of the electronic device 101 in.

[0103] The touch screen display 510 may include a display 5110 , a touch sensor 5113 , and a touch sensor IC 5115 .

[0104] According to an embodiment, the display 5110 may display various images under the control of the processor 520. The display 5110 may be implemented as one of a liquid crystal display (LCD), a light emitting diode (LED) display, a micro LED display, a quantum dot (QD) display, or an organic light emitting diode (OLED) display, but is not limited thereto.

[0105] According to an embodiment, the touch screen display 510 may be at least partially flexible and may be implemented as a foldable display, a rollable display, a slidable display, or a stretchable display.

[0106] The touch screen display 510 may detect a touch and / or near-touch (or hovering) input by using a part of a user's body (eg, a finger) or an input device (eg, a stylus).

[0107] The touch sensor 5113 may convert a touch input through a user's finger or an input device into a touch signal and transmit the touch signal to the touch sensor IC 5115. The touch sensor may be implemented as, for example, one of an electrode sensor (conductivity sensor), a capacitive touch sensor, a resistive touch sensor, a surface touch sensor, a projection capture (PCAP) touch sensor, or an ultrasonic touch sensor (surface acoustic wave touch sensor), but is not limited thereto.

[0108] The touch sensor IC5115 can control the touch sensor to detect a touch input at at least one position. The touch sensor IC5115 can generate touch data (e.g., touch position, touch area, touch video data, or touch time) for the touch input detected based on a change in a detected signal (e.g., voltage, light amount, resistance, or charge amount).

[0109] The memory 530 may store various instructions that may be executed by the processor 520. Such instructions may include control commands that may be recognized by the processor 520, such as arithmetic and logical operations, data transfer, or input / output. The memory 530 may include a volatile memory (e.g., Figure 1 The volatile memory 132 and the non-volatile memory (eg, Figure 1 The non-volatile memory 134 in the memory is used to temporarily or permanently store various data.

[0110] The processor 520 may be an element that is operably, functionally, and / or electrically connected to the elements of the electronic device 101 (e.g., the touch screen display 510 and the memory 530) to perform calculations or data processing related to the control and / or communication of the elements. The processor 520 may include Figure 1 The operations performed by the processor 520 may be performed by instructions stored in the memory 130 and causing the processor 120 to operate when executed.

[0111] According to an embodiment, the processor 520 may detect various touch gestures based on the touch signal and perform a touch action corresponding to the touch gesture. For example, when the touch is maintained for a long touch detection time (LT_a) (eg, about 500 ms), the processor 520 may detect a long touch.

[0112] According to an embodiment, the processor 520 may process (or execute) operations related to a force touch management function based on artificial intelligence and a function of mapping actions that can be performed by force touch according to user configuration (e.g., a force touch action function). The processor 520 may store touch data (e.g., touch position, touch area, touch video data, or touch time) received from the touch sensor IC5115 in a memory (e.g., a queue or a buffer register). The processor 520 may arrange the touch data in a queue and determine whether a force touch is recognized based on the touch data arranged in the queue. There may be no limitation on the calculation and data processing functions that can be implemented by the processor 520 on the electronic device 101. However, in the following, operations related to force touch will be described.

[0113] According to an embodiment, in relation to the force touch operation, the processor 520 may include a calculation condition determination module 5210 , a force touch learning module 5230 , a force touch determination module 5220 , and an execution control module 5240 , and each module may operate under the control of the processor 520 .

[0114] When a finger (or stylus) touches at least one point of the display for a configured reference time (e.g., about 300 ms) (e.g., force touch observation interval (FT_a)) or longer, the calculation condition determination module 5210 may observe whether there is a force touch based on the touch data. For example, the calculation condition determination module 5210 may be configured as follows: Figure 6 As shown in , after the touch by the finger 610 starts, when the touch of the finger is maintained for a reference time (eg, approximately 300 ms) or longer within a force touch observation interval (FT_a), it is determined that the calculation condition related to the force touch 620 is satisfied.

[0115] The force touch determination module 5220 may determine Figure 6 6. The force touch determination module 5220 determines whether there is a force touch 620 within the long touch detection period (LT_a) shown in FIG. 6. When no force touch is observed and the touch is maintained for the long touch detection period (LT_a), the force touch determination module 5220 may recognize the touch as a long touch 630.

[0116] When the calculation conditions related to force touch are met, the force touch determination module 5220 may use the touch data in the pressure calculation interval (e.g., 300ms to 350ms) to determine whether a force touch is recognized based on the artificial intelligence network. For example, the force touch determination module 5220 may read the touch data arranged in the queue for the finger touch with a high priority. The force touch determination module 5220 may perform pressure calculation for the touch being maintained in the pressure calculation interval (FT_b) after the force touch observation interval (FT_a) to determine whether a force touch is recognized. The pressure calculation may be a process of receiving multiple touch data (or input data) and outputting multiple force data (or output data) based on the artificial intelligence network. The touch data may indicate touch video data, and the force data may indicate virtual force data.

[0117] The force touch determination module 5220 may analyze the change of the user's skin area obtained based on the touch data (or the skin pressure based on the touch area) to distinguish between the force touch and the long touch. For example, when the result of the force touch observation indicates that the change of the skin area obtained based on the touch data gradually increases, the force touch determination module 5220 may identify the touch as a force touch. When the result of the force touch observation indicates that the change of the skin area does not gradually increase and the skin area is maintained at a constant size, the force touch determination module 5220 may determine the touch as a long touch.

[0118] When the touch is recognized as a force touch, the execution control module 5240 may display a force touch situation UI on the display 5110. The force touch situation UI may be omitted according to the display configuration. When the force touch is recognized to be maintained for a configured time or maintained while the force touch situation UI is displayed, the execution control module 5240 may execute a force touch action function mapped to (or configured for) the force touch.

[0119] When executing the force touch action function, the execution control module 5240 may reconfigure the screen layout to correspond to the force touch action function and / or application characteristics to execute the force touch action function. The force touch action function may include an action function that can be specified for each application and / or a global action function applicable to the electronic device system. For example, the force touch action function may include, but is not limited to, a pop-up window execution function, a volume panel control function, a quick memo function, a function of creating and then sharing a screenshot, a clipboard function, a flashlight function, an automatic rotation function, and / or a mute control function.

[0120] According to an embodiment, the force touch learning module 5230 may control touch sensitivity learning and / or touch area learning operations related to the force touch.

[0121] For example, when the force touch is released while the force touch situation UI is displayed, or when the force touch is released after the force touch is recognized, the force touch learning module 5230 may record an erroneous operation situation in which a force touch different from the user's intention is recognized. For example, in a situation in which the force touch situation UI is displayed for a configured time (e.g., n seconds), based on the input within the force touch determination interval (e.g., 300-350ms) being recognized as a force touch, when a touch input from the user on a different area is received before the configured time ends, the force touch learning module 5230 may deem that an unexpected force touch has occurred and record the situation as an erroneous operation situation.

[0122] The force touch learning module 5230 may learn the erroneous operation situation of the force touch to perform force touch sensitivity correction. For example, when the situation where the force touch is released is repeatedly configured N times or more, the force touch learning module 5230 may display a sensitivity correction UI on the display 5110 to cause sensitivity correction. For another example, when entering a touch sensitivity configuration mode (or configuration screen) according to a user request (e.g., manual adjustment), the force touch learning module 5230 may learn the user's touch data through the touch sensitivity configuration mode to adjust the sensitivity of the force touch.

[0123] For another example, the force touch learning module 5230 may learn a touch recognition area (or force touch recognition area) according to the shape state and / or holding state of the electronic device, and dynamically change the force touch recognition algorithm (e.g., reference area or threshold). For example, when a finger touch is performed according to various angles, the force touch learning module 5230 may store the dynamic area of ​​the touch in a queue, and train the force touch recognition model with the dynamic area of ​​the touch to update (or customize) the model.

[0124] The intensity of the touch or the area of ​​the finger that makes the touch may be different for each user. In addition, the touch area may vary according to the user's thumb, middle finger, index finger, ring finger, or little finger, and the touch area may be different according to the angle of the left hand and the right hand. For example, the touch area of ​​the thumb may be dynamically changed within the rotation radius of the thumb.

[0125] The force touch learning module 5230 can learn the recognition area (hereinafter, touch recognition area) where force touch is performed and its changes according to the appearance state (e.g., the first state, the second state, or the intermediate state) and the holding state (e.g., held with one hand or held with two hands) of the electronic device 101.

[0126] For example, when a foldable electronic device is used as an example, the force touch learning module 5230 may store in a queue the touch recognition area within the rotation radius of the thumb of the right hand or left hand when the device in the folded state is held with one hand. Optionally, the force touch learning module 5230 may store in a queue the touch recognition area of ​​the left hand or right hand in a two-hand holding state. Optionally, the force touch learning module 5230 may store in a queue the touch recognition area of ​​the index finger of the right hand when the foldable electronic device is held by the left hand. The touch recognition area stored in the queue may be learned (e.g., machine learning or deep learning) based on an artificial intelligence network to improve the recognition rate.

[0127] When the touch recognition area decreases due to rotation or tilt of the finger angle, the electronic device applied to this article may have difficulty recognizing force touch, but learns the touch area and changes to update the force touch recognition algorithm (eg, force touch recognition model) to improve the recognition rate of force touch.

[0128] According to an electronic device (eg, Figure 1 and Figure 2 The electronic device 101 in the embodiment may include a display (eg, Figure 1 The display module 160 or Figure 5 The electronic device 101 according to an embodiment may include a touch sensor (eg, Figure 1 Input module 150 or Figure 5 The electronic device 101 according to an embodiment may include a memory (eg, Figure 1 Memory 130 or Figure 3 The electronic device 101 according to an embodiment may include a processor (eg, Figure 1 The processor 120 or Figure 5 The memory 530 according to the embodiment may include instructions configured to cause the processor 520 to perform the following operations: determine whether a force touch with a gradually increasing change in skin pressure according to the touch area is recognized within the long touch detection time based on the touch data received from the touch sensor 5113. The memory 530 according to the embodiment may include instructions configured to cause the processor 520 to perform the following operations: based on the recognition of the force touch being maintained for a predetermined time, execute a force touch action function mapped to the force touch. The memory 530 according to the embodiment may include instructions configured to cause the processor 520 to perform the following operations: learn the erroneous operation situation of the force touch, and display a sensitivity correction UI on the display 5110 when the sensitivity adjustment of the force touch is required.

[0129] According to an embodiment, the display 5110 may include a flexible display having a variable display area, wherein visual information is displayed in the variable display area.

[0130] According to an embodiment, the processor 520 may be configured to: receive the shape state and / or holding state of the electronic device from the sensor module, and learn the touch recognition area based on the shape state and / or holding state of the electronic device to dynamically adjust the threshold of the force touch recognition model configured to determine skin pressure.

[0131] According to an embodiment, the force touch recognition model may be configured to be based on an artificial intelligence network, receive touch data as input data, and output force touch data.

[0132] According to an embodiment, the processor 520 may be configured to display a force touch situation UI on the display 5110 based on the recognition of the force touch.

[0133] According to an embodiment, the processor 520 may be configured to: record a force touch malfunction situation when a user's touch release occurs while the force touch situation UI is displayed or after the force touch is recognized, and identify that a force touch sensitivity adjustment is required when the force touch malfunction situation is repeatedly configured N times or more.

[0134] According to an embodiment, the processor 520 may be configured to observe whether a force touch occurs in a force touch observation interval before the long touch detection period, and perform pressure calculation in a force touch determination interval to determine whether a force touch is recognized.

[0135] According to an embodiment, the processor 520 may be configured to: observe that the corresponding touch is a force touch when the change in skin pressure based on the touch area gradually increases over time, and observe that the corresponding touch is a long touch when the change in skin pressure based on the touch area remains the same size over time.

[0136] According to an embodiment, the force touch motion function may include a motion function that can be specified for each application and / or a global motion function applicable to the electronic device system.

[0137] According to an embodiment, the processor 520 may be configured to receive information about an application being executed, and when a force touch is recognized while the application is being executed, reconfigure the screen layout of the force touch action function based on the information about the application to dynamically change and display the screen.

[0138] According to an embodiment, the sensitivity correction UI may include a touch controller, and may be configured to enable touch sensitivity adjustment according to a position of the touch controller. Figure 7A method of improving an artificial intelligence-based force touch operation of an electronic device according to an embodiment is shown.

[0139] In the following embodiments, the operations may be performed sequentially, but they do not necessarily need to be performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0140] refer to Figure 7 At operation 710, the processor (eg, Figure 1 The processor 120 or Figure 5 The processor 520 in the embodiment may receive application information based on the execution of the application.

[0141] The application information may include at least one of application type, package name, description information, application status information (e.g., background information, PIP information based on screen switching, attribute information (e.g., screen window information (e.g., pop-up, split status, rotation, position or size information))), and landscape / portrait mode support information.

[0142] According to an embodiment, the processor 520 may receive the form state information and the grip information from a sensor (eg, a gyroscope, an acceleration, a hinge angle (Hall IC) sensor, or a grip sensor) based on the change of the form state.

[0143] According to an embodiment, operation 710 may be omitted.

[0144] In operation 720 , the processor 520 may determine whether a force touch is recognized based on the reception of the touch data.

[0145] The processor 520 may analyze the features of the touch data from the time point when the touch starts to observe whether a force touch is generated and determine whether the force touch is recognized before the long touch determination. For example, the processor 520 may observe whether a force touch is generated during a force observation interval within a configured long touch detection period (e.g., 500ms), and perform pressure calculation for the touch after the force observation interval to determine whether a force touch exists. The processor 520 may perform a long touch determination for the touch after the force touch determination.

[0146] According to an embodiment, in the process of determining force touch, the electronic device 101 may learn multiple touch data to adjust the touch sensitivity and adjust the threshold of the touch area to improve the touch recognition rate. For example, the processor 520 may receive the shape state information and the holding information of the electronic device, and learn the touch recognition area of ​​the user according to the shape state information and the holding information to dynamically adjust the threshold of the touch area for determining the force touch (or update the force touch recognition model).

[0147] According to an embodiment, the processor 520 may learn to repeatedly recognize situations where the user's unintended force touch is detected, and when it is determined that sensitivity adjustment is required, sensitivity adjustment may be induced by providing a touch correction UI. Fig.13 Describes force touch learning operations.

[0148] At operation 730 , the processor 520 may display a force touch situation UI on the display based on the recognition of the force touch.

[0149] The force touch situation UI may include at least one of a message and an animation object notifying the user of the occurrence of the force touch, and its type and shape are not limited. When the user recognizes that the desired force touch has occurred through the force touch situation UI, the force touch may be maintained, and when the user recognizes that the undesired force touch is recognized, the touch release may be performed.

[0150] According to an embodiment, operation 730 may be omitted.

[0151] At operation 740, the processor 520 may execute the mapped (or configured) force touch action function based on the force touch recognition being maintained. For example, when the force touch is maintained in a situation where the force touch situation UI is displayed, or when the force touch recognition is maintained, the processor 520 may execute the force touch action function mapped to the force touch.

[0152] At operation 750, the processor 520 may dynamically change the screen layout of the force touch motion function and display it on the display. The processor 520 may reconfigure the screen layout based on at least one of the form factor state of the electronic device and / or information about the application being executed, and display the screen for the force touch motion function on the display.

[0153] For example, when the pop-up window execution function is mapped to the force touch action function, the processor 520 can execute the force touch action function of switching the full screen currently being displayed to the pop-up window screen based on the recognition of the force touch, and reconfigure the screen layout and determine the size of the pop-up window based on the characteristics of the application (e.g., foreground application) currently being executed on the display (e.g., watching YouTube or reproducing a video on the Internet) to display the pop-up window on the display. In this case, the hidden function for screen optimization can be used to process the icons (e.g., menu icons) included in the screen layout.

[0154] Figure 8 A screen for configuring a force touch motion function according to an embodiment is shown.

[0155] refer to Figure 8 According to an embodiment, an electronic device (e.g., Figure 1 A processor (eg, Figure 1 The processor 120 or Figure 5 The processor 520 in the embodiment may be configured to display a display (e.g., Figure 1 The display module 160 or Figure 5 A screen 810 for configuring a force touch action function is displayed on a display 5110 in FIG.

[0156] The screen 810 for configuring the force touch motion function may include an on / off switch item 820 enabling selection of use / non-use of the force touch motion, a touch sensitivity adjustment bar 830 for adjusting touch sensitivity, and a force touch motion function item 840 .

[0157] exist Figure 8 In the example, the force touch action function item 840 is shown to include a pop-up window execution function, a volume panel control function, a quick memo function, a function for creating and sharing screenshots, a clipboard viewing function, a flashlight on / off function, an automatic rotation on / off function, and a full mute on / off function, but is not limited thereto and may change according to the configuration of the electronic device.

[0158] When the user selects one of the force touch motion function items, the electronic device 101 may execute the force touch motion function selected (or mapped or configured) by the user based on the recognition of the force touch.

[0159] Fig. 9 A force touch management screen of an electronic device according to an embodiment is shown.

[0160] refer to Fig. 9 , in an electronic device according to an embodiment (eg, Figure 1 In the electronic device 101 ), the volume panel control function can be mapped or configured as a force touch action function. Fig. 9 The screen may be an example of applying the volume panel control function in a media application.

[0161] The user may input a force touch 920 while viewing an image through an image reproduction screen (eg, application execution screen) 910 , as shown in a screen 901 .

[0162] Despite Fig. 9 Although not shown in the example, the electronic device 101 may provide a force touch situation UI (not shown) to the display based on the recognition of the force touch. For another example, the electronic device 101 may flash an LED light or a screen, provide a tactile effect, or output a sound to notify the user of the recognition of the force touch based on the recognition of the force touch. Depending on the configuration, the force touch situation UI may be displayed or not. The force touch situation UI may provide an environment for identifying whether the force touch is a force touch desired by the user or causing a touch release when the force touch is not a force touch desired by the user.

[0163] As shown in screen 902, the electronic device 101 may provide a volume panel adjustment bar 930 mapped to the force touch to the image reproduction screen 910 based on recognition of the force touch. When the user keeps dragging 925 in the second direction while maintaining the force touch 920, the electronic device 101 may adjust the volume of the video according to the dragging direction. Fig. 9 In the example of FIG. 1 , the electronic device 101 may provide a 2-step touch gesture function of detecting a drag input after force touch recognition without a separate pressure sensor to improve the user experience.

[0164] Fig.10 A force touch management screen of an electronic device according to an embodiment is shown.

[0165] refer to Fig.10 , in an electronic device according to an embodiment (eg, Figure 1 In the electronic device 101 ), the pop-up window function can be mapped or configured as a force touch action function. Fig.10 The screen may show an example of a state in which a pop-up window function is applied in a media content application.

[0166] Screen 1001 may illustrate an example of a state in which a home screen 1010 is displayed on a display. A user may select a media content application icon on the home screen 1010 to execute the media content application.

[0167] While the electronic device is displaying a media stream reproduction screen (eg, application execution screen) 1020 on the display, the user may input a force touch 1030 as shown in screen 1002. Before long touch recognition, the electronic device 101 may analyze a plurality of touch data to determine whether a force touch is recognized.

[0168] As shown in screen 1003, the electronic device 101 may display a force touch situation UI 1040 based on the recognition of the force touch. The force touch situation UI 1040 may be omitted. Alternatively, the electronic device 101 may notify the user of the recognition of the force touch by using an effect such as screen flickering or a haptic effect.

[0169] Screen 1004 shows an example of a state in which a pop-up window function is executed as a force touch action function when it is determined that a force touch is recognized in the stream reproduction screen displayed on the display. The electronic device 101 can display the stream reproduction screen as overlapping with the main screen 1010 by using a pop-up window 1025 in response to the force touch.

[0170] Screen 1005 may be an example in which the electronic device 101 reconfigures the screen layout of the pop-up window 1025 in consideration of the characteristics of the media content application and displays the reconfigured pop-up window 1027 on the display. For example, the electronic device 101 may process the top layout and the bottom layout of the stream reproduction screen by using a hiding function and process only the stream image to be displayed.

[0171] Fig.11 A method of improving an artificial intelligence-based force touch operation of an electronic device according to an embodiment is shown.

[0172] In the following embodiments, the operations may be performed sequentially, but they do not necessarily need to be performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0173] refer to Fig.11 In operation 1110, an electronic device according to an embodiment (eg, Figure 1 A processor (eg, Figure 1 The processor 120 or Figure 5 The processor 520 in the embodiment may determine whether a force touch with a gradually increasing touch area is observed within the long touch detection time.

[0174] For example, the processor 520 may observe whether a force touch is generated during a force observation interval within a configured long touch detection period (e.g., 500 ms), and perform pressure calculation on the touch after the force observation interval to determine whether a force touch exists. The pressure calculation may be a process based on an artificial intelligence network receiving multiple touch data (or input data) and outputting multiple force data (or output data). The touch data may indicate touch video data, and the force data may indicate virtual force data.

[0175] The processor 520 may determine whether a force touch is recognized through pressure calculation, or perform a long touch determination on the touch after determining whether a force touch exists.

[0176] At operation 1120 , when the force touch is not recognized and the touch is maintained for the configured long touch detection time, the processor 520 may recognize a long touch.

[0177] According to an embodiment, in the process of determining force touch, the electronic device 101 may learn multiple touch data to adjust the touch sensitivity and adjust the touch area to improve the touch recognition rate. For example, the processor 520 may receive the shape state information and the holding information of the electronic device, and learn the touch recognition area of ​​the user according to the shape state information and the holding information to dynamically adjust the threshold value (or reference touch area) of the force touch (or update the force touch recognition model). The electronic device 101 may learn multiple touch data of the user to adjust the touch recognition area that changes according to the current shape state and the holding state to improve the touch recognition rate.

[0178] At operation 1130 , the processor 520 may display a force touch situation UI based on the recognition of the force touch.

[0179] According to an embodiment, operation 1130 may be omitted.

[0180] At operation 1140 , the processor 520 may determine whether a touch release is generated.

[0181] At operation 1150 , the processor 520 may execute a force touch action function mapped to the force touch based on that the touch release is not generated and the force touch is maintained.

[0182] At operation 1160, the processor 520 may reconfigure the screen layout of the force touch motion function and dynamically change and display it on the display. For example, the processor 520 may determine the screen layout of the force touch motion function based on at least one of the form factor state of the electronic device and / or information about the application being executed.

[0183] At operation 1170, when a touch release is generated in a case where the force touch situation UI is displayed, or when a touch release is generated after the force touch is recognized, the processor 520 may record malfunction situation information of the force touch.

[0184] At operation 1180 , the processor 520 may determine whether an erroneous operation of a force touch occurs repeatedly N times, and if the erroneous operation occurs a number less than N, may return to operation 1110 to determine again whether a force touch is observed.

[0185] At operation 1190, if the erroneous operation of the force touch is repeated N times or more, the processor 520 may perform a force touch learning operation for touch sensitivity. Fig.13 Describes force touch learning operations.

[0186] Fig.12 A touch gesture processing operation according to an embodiment is shown.

[0187] refer to Fig.12 , according to the electronic device of the embodiment (eg, Figure 1A processor (eg, Figure 1 The processor 120 or Figure 5 The processor 520 in the touch sensor may receive touch data (e.g., touch raw data or touch video data) 1210 from the touch sensor. The processor 520 may recognize a force touch gesture by performing a data processing operation 1220 (e.g., converting the touch data to force data), a trigger detection operation 1221 (e.g., determining a force touch calculation), a gesture detection operation 1222 (e.g., determining a force touch), and an interrupt generation operation 1223.

[0188] The processor 120 may process the recognized gesture through the framework layer and the application layer. For example, an interrupt signal for gesture recognition may be sent to the input device manager 1230. For example, if the volume panel control function is mapped to the force touch recognition function, the input device manager 1230 may request the audio service 1231 for volume panel control of the framework layer. The input device manager 1230 may transmit the force touch gesture event to the force touch application 1240 (e.g., Figure 3 The activity manager 1250 may transmit information of the currently executing application to the force touch application 1240.

[0189] The force touch application 1240 may transmit a force touch gesture event (e.g., interrupt) to the action scheduler 1242 based on receiving the event through the force touch event receiver application 1241. The action scheduler 1242 may specify a touch force action function execution routine corresponding to the touch force. The touch action event receiver 1241 may transmit an execution task for the touch force action function execution routine to the activity manager 1250.

[0190] In a state where a force touch gesture is recognized (or an interrupt occurs), when a touch release is generated, the processor 120 may record an erroneous operation situation of the force touch, and when the same situation is repeatedly recorded N times or more, perform a force touch learning process for touch sensitivity.

[0191] Fig.13 A force touch learning method of an electronic device according to an embodiment is shown.

[0192] In the following embodiments, the operations may be performed sequentially, but they do not necessarily need to be performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0193] refer to Fig.13 , according to the electronic device of the embodiment (eg, Figure 1 The electronic device 101 in the embodiment may provide a force touch learning function based on a plurality of touch data. At operation 1310, the processor (eg, Figure 1The processor 120 or Figure 5 The processor 520 in the embodiment may identify whether force touch learning is required. If force touch learning is not required, the processor 520 may terminate Fig.13 process.

[0194] When touch sensitivity correction is required in the force touch-based data processing process, the processor 520 may perform operations 1320 to 1345 to perform a touch sensitivity learning process. Alternatively, when the shape state and / or holding state of the electronic device changes, the processor 120 may perform operations 1350 to 1375 to perform a touch area learning process.

[0195] When explaining the touch sensitivity learning process, at operation 1320, the processor 520 may determine whether an erroneous operation situation caused by touch release (or a situation where a force touch is recognized differently from the user's intention) is repeated N times after force touch recognition. The processor 520 may terminate the touch sensitivity learning process when the erroneous operation situation caused by touch release is not repeated N times after force touch recognition. Fig.13 process.

[0196] At operation 1325, when the erroneous operation situation occurs repeatedly N times, the processor 520 may display a sensitivity correction UI. For example, the sensitivity correction UI may be provided by using a pop-up window, but is not limited thereto.

[0197] At operation 1330 , when the user selects the sensitivity correction UI, the processor 520 may enter a sensitivity correction configuration mode, and at operation 1335 , the processor 520 may determine a touch method based on an input obtained in the sensitivity correction configuration mode.

[0198] At operation 1340, the processor 520 may move the touch controller included in the sensitivity adjustment bar to the user's touch position. At operation 1340, the processor 520 may change the configuration of the touch sensitivity according to the position of the touch controller. The changed configuration of the touch sensitivity may be applied to touch gesture learning of the electronic device.

[0199] When explaining the touch area learning process, at operation 1350 , the processor 520 may identify a plurality of touch data stored in a database (DB) while the electronic device is being used.

[0200] At operation 1355, the processor 520 may identify the appearance state and the holding state of the electronic device. For example, the processor 520 may identify whether the electronic device is operating. The processor 520 may identify the appearance state (e.g., folded / unfolded / intermediate state). The processor 520 may determine whether the electronic device is held by one hand or by two hands by using a holding sensor. The processor 520 may determine the current tilt state of the electronic device by using an acceleration sensor and a gyroscope sensor.

[0201] At operation 1360, the processor 520 may determine whether the touch recognition area of ​​the touch data currently being processed is included in the range from the minimum value to the maximum value of the plurality of touch data stored in the database. The threshold of the minimum value or the maximum value may be dynamically adjusted through the learning model. For example, since there are various situations such as children, adults, men, women, people with large hand areas, and people with small hand areas, the electronic device 101 may support a function of adjusting the threshold of each user through touch area learning.

[0202] In operation 1365 , the processor 520 may perform screening on a region corresponding to a designated value in each frame.

[0203] At operation 1370 , the processor 520 may determine a recognition model based on the dynamic area and touch data stored in a database (DB), and at operation 1375 , the processor 520 may update the recognition model based on the dynamic area.

[0204] Fig.14 A screen showing touch sensitivity adjustment caused by force touch learning in force touch management by an electronic device according to an embodiment is shown.

[0205] refer to Fig.14 , according to the electronic device of the embodiment (eg, Figure 1 The electronic device 101 in the embodiment may be in a state where a pop-up window execution function is mapped (or configured) as a force touch action.

[0206] Screen 1401 may be an example of a main screen 1410 showing the electronic device 101. As shown in screen 1401, the user may perform a long touch input 1420. The electronic device 101 may determine whether a force touch is recognized before long touch detection. For example, even if the user desires to perform a long touch input, the electronic device 101 may recognize the input as a force touch by determining that the skin area gradually increases after a predetermined time based on the touch area.

[0207] As shown in screen 1402 , the electronic device 101 may display a force touch situation UI 1430 on the display based on the recognition of the force touch.

[0208] As shown in screen 1403, the user can recognize that the force touch that the user does not expect is recognized through the force touch situation UI and perform a touch release operation. When the touch release is generated while the force touch situation UI is displayed, the electronic device 101 can recognize the force touch misoperation situation that the user does not expect and record the misoperation situation.

[0209] Screen 1404 may illustrate an example in which a user experience of a force touch malfunction (such as occurrence of a touch release) has repeatedly occurred N times after the electronic device 101 recognizes the force touch 1423 .

[0210] As shown in screen 1405, after force touch misoperation occurs N times, the electronic device 101 may display a sensitivity correction UI 1440 on the display that recommends force touch sensitivity adjustment to the user. The sensitivity correction UI 1440 may be implemented as a pop-up window including a touch sensitivity adjustment bar, but is not limited thereto.

[0211] As shown in screen 1406 , the user may enter a sensitivity correction configuration mode through a sensitivity correction UI 1440 , and change the touch sensitivity configuration by adjusting an adjustment bar 1450 .

[0212] Although referenced Fig.14 The example screen of describes an example of a user adjusting the touch sensitivity configuration through the sensitivity correction UI, but according to an embodiment, the electronic device 101 may support the following functions: analyzing the user's force touch pattern based on an artificial intelligence network, and automatically changing the touch sensitivity configuration based on determining that the force touch sensitivity needs to be adjusted based on the analysis results. For example, the electronic device 101 supports an automatic touch sensitivity configuration option, and when the user configures the automatic touch sensitivity configuration option to be turned on, the electronic device may automatically change the touch sensitivity configuration according to the user's force touch pattern, and display a notification message such as "The force touch sensitivity configuration has changed. To recognize a force touch, press slightly heavier / lighter than you currently do" on the display.

[0213] Fig.15 A manual learning configuration screen for touch area learning at the time of force touch management by an electronic device according to an embodiment is shown.

[0214] refer to Fig.15 , according to the electronic device of the embodiment (eg, Figure 1 The electronic device 101 in the figure may provide a function of manually learning a force touch area according to an appearance state and a holding state of the electronic device 101.

[0215] Screen 1501 shows an example of a touch area learning UI 1510 according to the grip of two hands of a user in a first state of the foldable electronic device. The user can input force touch 1520 at various locations while holding the electronic device with two hands.

[0216] Screen 1502 shows an example of a touch area learning UI 1515 according to the grip of a single hand of a user in the second state of the foldable electronic device. The user can input force touch 1525 at various locations while holding the electronic device with one hand.

[0217] The electronic device 101 may provide the screen 1501 or the screen 1502 according to the user's request to configure the force touch area learning, so as to learn the user's force touch area and adjust the threshold of the touch area.

[0218] exist Fig.15 In the example of , a screen for manually learning a force touch area is shown. However, the electronic device 101 may support a function of automatically learning a touch area based on an artificial intelligence network (e.g., machine learning or deep learning) and based on a plurality of touch data of a user according to an outer shape state and a holding state of the electronic device. For example, when the outer shape state or holding state of the electronic device is changed according to touch area learning, the electronic device 101 may dynamically change a threshold value (e.g., a reference touch area) for force touch recognition.

[0219] In the following, Figures 16 to 20 An example of force touch action execution for each force touch action configuration is shown in FIG.

[0220] Figures 16 to 20 A force touch management invention configured for each force touch action is shown according to various embodiments. Figures 16 to 20 The example screens of omits a process of displaying a force touch situation UI based on force touch recognition, but each example screen may provide a force touch situation UI before executing a force touch action function.

[0221] Fig.16 The screen of the embodiment may show an example of a window screen fixing function for a split window (eg, a 3-split window) on a display. Figure 1 The electronic device 101 in FIG. 1601 may be in a state where a multi-window screen fixing function is mapped (or configured) as a force touch action. For example, as shown in screen 1601, the electronic device 101 may display a split window screen 1610 including an application A screen 1620, an application B screen 1621, and an application C screen 1622 on the display. The electronic device 101 may, based on recognizing a force touch 1630 in a state where the split window screen 1610 is displayed, perform a configured force touch action function, for example, fix the application A screen 1620 including the position where the force touch is recognized, as shown in screen 1602. As a result of performing the force touch action function, the electronic device 101 may display an indicator 1640 that the application A screen fixing is completed. In screen 1602, the application B screen 1621 and the application C screen 1622 may be subject to window position changes and screen changes to another application.

[0222] Fig.17The screen of may show an example of a real-time subtitle function of an audio application (e.g., ongoing radio playback or media reproduction). The electronic device 101 according to an embodiment may be in a state where the real-time subtitle function is mapped (or configured) as a force touch action. The real-time subtitle function may be an action function that converts the audio reproduced in real time into text and displays the text on the screen. The electronic device 101 may perform an audio function (e.g., radio playback or music playback) in the background as shown in screen 1701, and display the main screen 1710 on the display.

[0223] As shown in screen 1702, the electronic device 101 may execute a configured force touch motion function, such as a real-time subtitle function, based on recognition of the force touch 1720, and reconfigure and display a layout screen 1730 on the display based on the real-time subtitle function. For example, the electronic device 101 may convert audio being executed on the background into text, and convert the screen into a layout screen 1730 for displaying the converted text 1735 to display the layout screen. The text 1735 displayed on the screen 1702 may support a copy / paste function.

[0224] Fig.18 The screen may show an example of a quick memo function of a message application. The electronic device 101 according to an embodiment may be in a state where the quick memo function is mapped (or configured) as a force touch action. The quick memo function may be a function of determining a specific object (or object area) for a specific object (e.g., a message item) displayed on the screen and displaying the specific object by using a pop-up layout. The electronic device 101 may display a message application execution screen 1810 including a message list on the display as shown in screen 1801. When the user performs an input of a force touch 1820 to select a specific message, the electronic device 101 may, as shown in screen 1802, perform a configured force touch action function, such as a quick memo function, based on the recognition of the force touch to display the selected message by using a pop-up window (or quick memo window) 1830. The pop-up window (i.e., the quick memo window) 1830 may disappear from the display after N seconds of being displayed and configured.

[0225] Fig.19The screen may show an example of a clipboard input function related to a text input field property. The electronic device 101 according to an embodiment may be in a state where a clipboard input function is mapped (or configured) as a force touch action. The clipboard subtitle function may be a function that automatically inserts the content last stored in the clipboard into the text input field when a force touch input is received on the text input field. The electronic device 101 may display a message conversation screen 1910 including a text input field on a display as shown in screen 1901. When a user performs an input of a force touch 1920 to select a text input field, the electronic device 101 may, as shown in screen 1902, based on recognition of the force touch, perform a configured force touch action function, such as a clipboard input function, to insert the content (e.g., text or image) 1930 last stored in the clipboard into the input field and send the content.

[0226] Fig. 20 The screen of FIG. 100 may show an example of a function of controlling a flashlight (or a torch) 2030. The electronic device 101 according to an embodiment may be in a state where a flashlight control function is mapped (or configured) as a force touch action. Fig. 20 As shown in , the electronic device 101 may display a home screen 2010 on the display. When the user performs a force touch 2020 input on the home screen, the electronic device 101 may turn on a configured force touch action function, such as a flashlight function, based on recognition of the force touch. When a force touch input is detected again when the flashlight function is turned on, the electronic device may turn off the flashlight function.

[0227] Although not shown in the drawings, the force touch function may be configured in various ways. For example, the electronic device may provide a function of generating a screenshot and then sharing the screenshot with another electronic device, a function of searching for a specific word in a text screen by using a dictionary, a function of selecting a file and then previewing the file, a function of selecting a schedule and then creating a new calendar, a function of changing the fast forward or rewind speed during image reproduction, or a function of editing a file name as a force touch action function, but the present disclosure is not limited thereto.

[0228] Fig.21 A screen showing an on / off configuration of supporting a force touch motion function in an electronic device according to an embodiment.

[0229] refer to Fig.21 , according to the electronic device of the embodiment (eg, Figure 1 The electronic device 101 in the embodiment may also support a function of executing an application through a force touch input or a function of adjusting a volume through a force touch input. Fig.21The screen 2130 of the embodiment may indicate a screen for configuring a quick execution function triggered by a specific gesture. For example, when the user selects the item 2140 for executing an application by using a force touch, the electronic device 101 may provide a configuration mode for an application executed in response to the force touch, and configure the application specified by the user's selection to be executed by the force touch. As another example, when the user selects the item 2150 for adjusting the volume by using a force touch, the electronic device 101 may provide a configuration mode for an application whose volume is adjusted in response to the force touch.

[0230] A method for improving a force touch operation according to an example embodiment includes receiving touch data received from a touch sensor. The method according to an example embodiment includes: based on the touch data, determining whether a force touch with a gradually increasing change in skin pressure according to a touch area is recognized within a long touch detection time. The method according to an example embodiment includes: based on recognizing the force touch, executing a force touch action function mapped to the force touch. The method according to an example embodiment includes: when recognizing the force touch, learning a misoperation situation of the force touch, and displaying a sensitivity correction UI on a display if sensitivity adjustment of the force touch is required.

[0231] The display according to example embodiments includes a flexible display having a variable display area, wherein visual information is displayed in the variable display area.

[0232] The method according to example embodiments includes determining whether the force touch is recognized.

[0233] The method according to example embodiments includes receiving an electronic device form factor status and / or a grip status from a sensor module.

[0234] The method according to an example embodiment includes learning a touch recognition area according to an appearance state and / or a holding state of the electronic device to dynamically adjust a threshold of a force touch recognition model, wherein the force touch recognition model is configured to determine skin pressure to determine whether to recognize the force touch.

[0235] The force touch recognition model according to example embodiments is configured to be based on an artificial intelligence network, receive touch data as input data, and output force touch data.

[0236] Determining whether the force touch is recognized according to an example embodiment includes displaying a force touch situation UI on the display based on the recognition of the force touch.

[0237] Displaying the sensitivity correction UI on the display according to an example embodiment includes: when a user's touch release occurs in a state where a force touch situation UI is displayed or after the force touch is recognized, recording the erroneous operation situation, and when the erroneous operation situation is repeatedly configured N times or more, identifying a state requiring sensitivity adjustment of the force touch, and displaying the sensitivity correction UI.

[0238] Determining whether the force touch is recognized according to an example embodiment includes observing whether the force touch occurs in a force touch observation interval before a configured long touch detection period, and performing pressure calculation in a force touch determination interval to determine whether the force touch is recognized.

[0239] Observing whether the force touch occurs in the force touch observation interval according to an example embodiment includes: observing the corresponding touch as a force touch if a change in skin pressure based on the touch area gradually increases over time, and observing the corresponding touch as a long touch if the change in skin pressure based on the touch area remains at a constant size over time.

[0240] The force touch motion function according to example embodiments includes a motion function that can be specified for each application and / or a global motion function applicable to an electronic device system.

[0241] The method according to an example embodiment also includes an operation of receiving information about an application being executed. The operation of receiving information about an application being executed includes: in a state where the force touch is recognized while the application is being executed, reconfiguring a screen layout of the force touch action function based on the information about the application to dynamically change and display a screen.

[0242] The method according to an example embodiment further includes an operation of receiving a user input for adjusting a touch controller displayed on the sensitivity correction UI and an operation of changing a touch sensitivity setting according to a position of the touch controller.

Claims

1. An electronic device, comprising: monitor; Touch sensor; Memory; as well as processor, The memory includes instructions configured to cause the processor to perform the following operations: determining, based on touch data received from the touch sensor, whether a force touch having a gradually increasing change in skin pressure according to a touch area is recognized within a long touch detection time; executing a force touch action function mapped to the force touch based on recognition of the force touch being maintained for a predetermined time; and An erroneous operation situation of the force touch is learned, and if sensitivity adjustment of the force touch is required, a sensitivity correction UI is displayed on the display.

2. The electronic device according to claim 1, wherein: The display comprises a flexible display having a variable display area, wherein visual information is displayed in the variable display area, The processor is further configured to: receive an electronic device appearance state and / or a holding state from the sensor module, and learn a touch recognition area according to the electronic device appearance state and / or the holding state to dynamically adjust a threshold of a force touch recognition model configured to determine skin pressure, and The force touch recognition model is configured to be based on an artificial intelligence network, receive touch data as input data and output force touch data.

3. The electronic device according to claim 1 or 2, wherein: The processor is configured to display a force touch condition UI on the display based on the identification of the force touch.

4. The electronic device according to claim 3, wherein: The processor is configured to: record the erroneous operation situation when the user's touch release occurs while the force touch situation UI is displayed or after the force touch is recognized, and recognize that sensitivity adjustment of the force touch is required when the erroneous operation situation is repeatedly configured N times or more.

5. The electronic device according to claim 1, wherein: The processor is configured to observe whether a force touch occurs in a force touch observation interval before the long touch detection period, and perform pressure calculation in a force touch determination interval to determine whether the force touch is recognized.

6. The electronic device according to claim 5, wherein: The processor is configured to observe the corresponding touch as a force touch when the change in skin pressure based on the touch area gradually increases over time, and observe the corresponding touch as a long touch when the change in skin pressure based on the touch area remains at a constant size over time.

7. The electronic device according to claim 1, wherein: The force touch motion function includes a motion function that can be specified for each application and / or a global motion function applicable to the electronic device system.

8. The electronic device according to claim 1, wherein: The processor is configured to receive information about an application being executed, and if the force touch is recognized in a state where the application is executed, reconfigure a screen layout of the force touch motion function based on the information about the application to dynamically change and display a screen.

9. The electronic device according to claim 1, wherein: The sensitivity correction UI includes a touch controller, and is configured to enable touch sensitivity adjustment according to a position of the touch controller depending on a user input.

10. A method for improving force touch operation of an electronic device, the method comprising: receiving touch data received from a touch sensor; determining, based on the touch data, whether a force touch having a gradually increasing change in skin pressure according to a touch area is recognized within a long touch detection time; Based on identifying the force touch, executing a force touch action function mapped to the force touch; as well as When the force touch is recognized, an erroneous operation situation of the force touch is learned, and if sensitivity adjustment of the force touch is required, a sensitivity correction UI is displayed on a display.

11. The method according to claim 10, wherein: The display comprises a flexible display having a variable display area, wherein visual information is displayed in the variable display area, Wherein, the method further comprises: determining whether the force touch is recognized; Receiving an appearance state and / or a holding state of the electronic device from a sensor module; and learning a touch recognition area according to the shape state and / or the holding state of the electronic device to dynamically adjust a threshold of a force touch recognition model, wherein the force touch recognition model is configured to determine skin pressure to determine whether to recognize the force touch, and The force touch recognition model is configured to be based on an artificial intelligence network, receive touch data as input data and output force touch data.

12. The method according to claim 10 or 11, wherein: Determining whether the force touch is recognized further includes displaying a force touch condition UI on the display based on the recognition of the force touch.

13. The method according to claim 12, wherein: Displaying the sensitivity correction UI on the display further includes: recording the erroneous operation situation when a touch release of a user occurs in a state where the force touch situation UI is displayed or after the force touch is recognized, and Wherein, in a case where the erroneous operation situation is repeatedly configured N times or more, a state requiring sensitivity adjustment of the force touch is recognized, and the sensitivity correction UI is displayed.

14. The method according to claim 10, wherein: Determining whether the force touch is recognized includes observing whether the force touch occurs in a force touch observation interval before a configured long touch detection period, and performing pressure calculation in a force touch determination interval to determine whether the force touch is recognized.

15. The method according to claim 14, wherein: Observing whether a force touch occurs in the force touch observation interval includes: observing the corresponding touch as a force touch when a change in skin pressure based on the touch area gradually increases over time, and observing the corresponding touch as a long touch when the change in skin pressure based on the touch area remains at a constant size over time.

Images