Electronic devices and methods for operating electronic devices
By integrating a communication module into the electronic device to establish a connection with the stylus, errors in gesture input can be identified and corrected, thus solving the problem of error detection in stylus gesture input and improving the user experience.
Patent Information
- Application Number
- CN202280007843.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-02-24
- Filing Date
- 2022-02-14
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-02-14
AI Technical Summary
Existing electronic devices struggle to effectively detect and correct gesture recognition errors when using styluses for gesture input, leading to a decline in user experience.
By integrating a communication module into the electronic device to establish a connection with the stylus, it can recognize hand gestures when the stylus is separated from the display, and correct gesture recognition errors by displaying visual guidance on the display.
It enables error detection and correction for handwriting gesture input, improving the user experience and helping users identify and correct input errors through visual guidance.
Smart Images

Figure CN116547638B_ABST
Abstract
Description
Technical Field
[0001] Specific embodiments of this disclosure relate to an electronic device used in conjunction with a stylus, and the interoperability between the two devices. Background Technology
[0002] Portable electronic devices (such as smartphones) can perform a variety of functions. These devices provide a user interface through which users can easily access and control these functions. The user interface can be displayed on a monitor.
[0003] Recent electronic devices allow for more precise user input using a stylus. Users can operate the device by touching the stylus to the display, inputting commands while the stylus is a short distance away from the display (i.e., in a so-called "hovering" state), or inputting commands via the stylus while remaining at a greater distance from the device. Furthermore, the device can recognize gesture input performed via the stylus and execute functions corresponding to the recognized gestures. Summary of the Invention
[0004] Technical issues
[0005] Specific embodiments of this disclosure provide an electronic device and method thereof capable of detecting errors related to gesture input generated via a stylus and providing prompts and guidance for correcting the errors.
[0006] Technical solution
[0007] According to a specific embodiment of this disclosure, an electronic device includes: a display; a communication module configured to perform a wireless connection with a stylus pen; a processor operatively coupled to the display and the communication module; and a memory storing programming instructions, wherein the programming instructions are executable by the processor to cause the electronic device to perform the following operations: receiving gesture input from the stylus pen via the communication module when the stylus pen is spaced apart from the display; determining that a gesture recognition error has occurred based on the received gesture input; and controlling the display to display visual guidance related to the gesture recognition error.
[0008] According to an embodiment, a method of operating an electronic device includes: establishing a communication connection with a stylus via a communication module, identifying via at least one processor that the stylus is spaced apart from a display, receiving gesture input from the stylus, determining that a gesture recognition error has occurred based on the received gesture input, and displaying visual guidance related to the gesture recognition error via the display.
[0009] Beneficial effects
[0010] The disclosed electronic device and method of operating the electronic device can detect errors related to hand gesture input with a stylus and provide visual guidance for correcting these errors. Users can identify errors in gesture input through the visual guidance and be prompted to take remedial actions to resolve the errors. Attached Figure Description
[0011] Figure 1 This is a block diagram of an electronic device in a network environment according to a specific embodiment of the present disclosure.
[0012] Figure 2 This is a block diagram illustrating a procedure according to a particular embodiment.
[0013] Figure 3 An electronic device and a stylus are shown according to an embodiment.
[0014] Figure 4 The detailed structure of the stylus according to an embodiment is shown.
[0015] Figure 5 This is a block diagram of a stylus according to an embodiment.
[0016] Figure 6 This is a flowchart describing a method of operating an electronic device according to an embodiment.
[0017] Figure 7 It is used for a more detailed description Figure 6 The flowchart of the method shown is shown.
[0018] Figure 8 The hovering state of the electronic device and stylus according to an embodiment is shown.
[0019] Figure 9 The operation of a stylus, processed to be effective in a hover state according to an embodiment, is shown.
[0020] Figure 10 An example of the first guiding content displayed in response to a first gesture recognition error in a hover state is shown.
[0021] Figure 11 This demonstrates the occurrence of errors in recognizing a second gesture based on the tilt of the stylus while it is in the air.
[0022] Figure 12 An example of a second guide content displayed in response to a second gesture recognition error while in mid-air is shown.
[0023] Figure 13 An example of a reference gesture that can be recognized in mid-air is shown.
[0024] Figure 14 An example of a method for recognizing hand gesture input from a stylus is shown.
[0025] Figure 15 This illustrates the occurrence of a third gesture recognition error based on the identified abnormal gesture.
[0026] Figure 16 An example of third-party guidance content displayed in response to a third gesture recognition error is shown. Detailed Implementation
[0027] Figure 1 This is a block diagram of an electronic device in a network environment according to a specific embodiment of the present disclosure.
[0028] Reference Figure 1 In network environment 100, electronic device 101 can communicate with electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or with at least one of electronic device 104 or server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, electronic device 101 can communicate with electronic device 104 via server 108. According to an embodiment, electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module 196, or an antenna module 197. In some embodiments, at least one component (e.g., connection terminal 178) may be omitted from electronic device 101, or one or more other components may be added to electronic device 101. In some embodiments, some of the components (e.g., sensor module 176, camera module 180, or antenna module 197) may be integrated into a single component (e.g., display module 160).
[0029] Processor 120 can execute software (e.g., program 140) to control at least one other component (e.g., hardware or software component) of electronic device 101 connected to processor 120 and perform various data processing or operations. According to embodiments, as at least part of the data processing or operation, processor 120 can store commands or data received from other components (e.g., sensor module 176 or communication module 190) in volatile memory 132, process the commands or data stored in volatile memory 132, and store the resulting data in non-volatile memory 134. According to embodiments, processor 120 may include a main processor 121 (e.g., a central processing unit or application processor) or an auxiliary processor 123 (e.g., a graphics processing unit, neural network processing unit (NPU), image signal processor, sensor hub processor, or communication processor) that can operate independently or in conjunction with the main processor 121. For example, when electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or may be configured to be dedicated to a specific function. The auxiliary processor 123 can be implemented separately from the main processor 121 or as part of the main processor 121.
[0030] When the main processor 121 is inactive (e.g., in sleep mode), the auxiliary processor 123 may, in place of the main processor 121, control at least some of the functions or states associated with at least one component of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190). Alternatively, when the main processor 121 is active (e.g., application execution), the auxiliary processor 123 may, together with the main processor 121, control at least some of the functions or states associated with at least one component of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190). According to embodiments, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another functionally associated component (e.g., camera module 180 or communication module 190). According to embodiments, the auxiliary processor 123 (e.g., an NPU) may include hardware architecture dedicated to processing artificial intelligence models. Artificial intelligence models can be generated through machine learning. This learning can be performed by the electronic device 101 that executes the artificial intelligence model itself, or it can be performed by a separate server (e.g., server 108). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but this disclosure is not limited to the examples described above. The artificial intelligence model can include multiple layers of artificial neural networks. The artificial neural network can be one of the following: deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted Boltzmann machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware architecture, the artificial intelligence model may additionally or optionally include a software architecture.
[0031] Memory 130 may store various types of data used by at least one component of electronic device 101 (e.g., processor 120 or sensor module 176). The data may include, for example, software (e.g., program 140) and input or output data regarding commands associated with it. Memory 130 may include volatile memory 132 or non-volatile memory 134.
[0032] The program 140 may be stored as software in the memory 130, and the program 140 may include, for example, an operating system 142, middleware 144, or application 146.
[0033] Input module 150 can receive commands or data from outside the electronic device 101 (e.g., a user) to be used by components of the electronic device 101 (e.g., processor 120). Input module 150 may include, for example, a microphone, mouse, keyboard, keys (e.g., buttons), or digital pen (e.g., stylus).
[0034] The audio output module 155 can output audio signals to the outside of the electronic device 101. The audio output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to embodiments, the receiver can be implemented separately from the speaker or as part of the speaker.
[0035] Display module 160 can visually provide information to the outside of electronic device 101 (e.g., to a user). Display module 160 may include, for example, a display, a holographic device, or a projector, and control circuitry for controlling the corresponding device. According to an embodiment, display module 160 may include a touch sensor configured to sense touch or a pressure sensor configured to measure the intensity of the force generated by touch.
[0036] The audio module 170 can convert sound into electrical signals, or conversely, convert electrical signals into sound. According to an embodiment, the audio module 170 can acquire sound through the input module 150, or output sound through the sound output module 155 or through an external electronic device (e.g., electronic device 102, such as a speaker or headphones) directly or wirelessly connected to the electronic device 101.
[0037] Sensor module 176 can detect the operating state of electronic device 101 (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to embodiments, sensor module 176 may include, for example, a gesture sensor, gyroscope sensor, barometric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biometric sensor, temperature sensor, humidity sensor, or illuminance sensor.
[0038] Interface 177 may support one or more specified protocols that can be used by electronic device 101 to connect directly or wirelessly to external electronic devices (e.g., electronic device 102). According to embodiments, interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.
[0039] Connection terminal 178 may include a connector through which electronic device 101 can be physically connected to an external electronic device (e.g., electronic device 102). According to embodiments, connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).
[0040] The tactile module 179 can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that can be perceived by a user through touch or kinesthesia. According to embodiments, the tactile module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.
[0041] Camera module 180 can capture still images and moving images. According to an embodiment, camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.
[0042] The power management module 188 manages the power supplied to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).
[0043] Battery 189 can supply power to at least one component of electronic device 101. According to an embodiment, battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.
[0044] Communication module 190 can support the establishment of a direct communication channel (e.g., a wired communication channel) or a wireless communication channel between electronic device 101 and external electronic devices (e.g., electronic device 102, electronic device 104, or server 108), and supports communication performance through the established communication channel. Communication module 190 may include one or more communication processors that operate independently of processor 120 (e.g., application processor), and supports direct (e.g., wired) or wireless communication. According to embodiments, communication module 190 may include wireless communication module 192 (e.g., cellular communication module, short-range communication module, or Global Navigation Satellite System (GNSS) communication module) or wired communication module 194 (e.g., local area network (LAN) communication module or power line communication module). The corresponding communication module in the communication module can communicate with the external electronic device 104 via a first network 198 (e.g., a short-range communication network such as Bluetooth™, WiFi Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, 5G network, next-generation communication network, Internet, or computer network (e.g., LAN or WAN)). Such various types of communication modules can be integrated into a single component (e.g., a single chip), or they can be implemented as multiple separate components (e.g., multiple chips). The wireless communication module 192 can use subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 to identify or authenticate the electronic device 101 within the communication network (such as the first network 198 or the second network 199).
[0045] Wireless communication module 192 can support next-generation communication technologies beyond 5G and 4G networks, such as supporting New Radio Access (NR). NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband, eMBB), reduced terminal power usage and access to multiple terminals (massive machine-type communications, mMTC), or high reliability and low latency (ultra-reliable and low-latency communications, URLLC). Wireless communication module 192 can support high-frequency bands (e.g., millimeter-wave bands) to achieve high data rates. Wireless communication module 192 can support various technologies used to ensure performance in high-frequency bands, such as beamforming, massive MIMO, full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. Wireless communication module 192 can support various requirements specified in electronic device 101, external electronic devices (e.g., electronic device 104), or network systems (e.g., second network 199). According to an embodiment, the wireless communication module 192 may support peak data rates (e.g., 20 Gbps or higher) for implementing eMBB, lost coverage (e.g., 164 dB or lower) for implementing mMTC, or U-plane latency (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less round trip) for implementing URLLC.
[0046] Antenna module 197 can transmit or receive signals or power to or from an external source (e.g., an external electronic device). According to embodiments, antenna module 197 may include an antenna comprising a radiator formed of a conductor or conductive pattern formed on a substrate (e.g., a printed circuit board PCB). According to embodiments, antenna module 197 may include multiple antennas (e.g., an array antenna). In this case, for example, communication module 190 may select at least one antenna from the multiple antennas that is suitable for a communication method used in a communication network (such as a first network 198 or a second network 199). Signals or power can be transmitted or received between communication module 190 and an external electronic device via the selected at least one antenna. According to some embodiments, other components besides the radiator (e.g., a radio frequency integrated circuit RFIC) may be additionally incorporated into antenna module 197.
[0047] According to a specific embodiment, antenna module 197 may form a millimeter-wave antenna module. According to the embodiment, the millimeter-wave antenna module may include a PCB, a radio frequency integrated circuit (RFIC), and multiple antennas (e.g., an array antenna), wherein the RFIC is disposed on or adjacent to a first surface (e.g., a lower surface) of the PCB and is capable of supporting a specified high-frequency band (e.g., a millimeter-wave band), and the multiple antennas (e.g., the array antenna) are disposed on or adjacent to a second surface (e.g., an upper surface or a side surface) of the PCB and are capable of transmitting or receiving signals of the specified high-frequency band.
[0048] At least some of the components can be connected to each other and exchange signals (e.g., commands or data) with each other through peripheral communication methods (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industrial processor interface (MIPI)).
[0049] According to an embodiment, commands or data can be sent or received between electronic device 101 and external electronic device 104 via server 108 connected to the second network 199. Each of the external electronic devices 102 and 104 may be provided in the same or a different type as electronic device 101. According to an embodiment, all or part of the operations performed in electronic device 101 may be performed by one or more of the external electronic devices 102, 104, and 108. For example, when electronic device 101 needs to automatically perform a function or service or to perform a function or service in response to a request from a user or another device, electronic device 101 may request the one or more external electronic devices to perform at least a portion of the function or service instead of performing the function or service itself, or electronic device 101 may request the one or more external electronic devices to perform at least a portion of the function or service in addition to performing the function or service itself. Upon receiving the request, the one or more external electronic devices may perform at least a portion of the requested function or service, or perform additional functions or services related to the request, and send the result of the execution to electronic device 101. Electronic device 101 can process the result as is or additionally, and provide the processing result as at least part of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technologies can be used, for example. Electronic device 101 can use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and / or neural networks. According to embodiments, external electronic device 104 or server 108 may be included in a second network 199. Electronic device 101 can be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technologies.
[0050] The electronic device according to the specific embodiments disclosed in this specification can be of various types. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The electronic device according to the embodiments of this document is not limited to the devices described above.
[0051] The specific embodiments of this disclosure and the terminology used herein are not intended to limit the technical features of this disclosure to the specific embodiments, but should be understood to cover all modifications, equivalents, and substitutions falling within the scope of this disclosure. Throughout the description of the drawings, the same reference numerals refer to the same elements. Unless the context clearly indicates otherwise, the singular forms preceding items with "a," "an," and "the" are also intended to include the plural forms. In this disclosure, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first," "second," etc., are used to distinguish one element from another and do not modify the element in other respects (e.g., importance or order). When one (e.g., the first) element is referred to as being "coupled" or "connected" to another (e.g., the second) element, with or without the terms "functionally" or "communically," it means that the first element is connected to the second element directly (e.g., wired), wirelessly, or via a third element.
[0052] In specific embodiments of this specification, the term "module" may include units implemented in hardware, software, or firmware, and is used interchangeably with terms such as, for example, logic, logic block, component, or circuit. A module may be a component formed integrally, or the smallest unit or part of a component that performs one or more functions. For example, according to an embodiment, a module may be implemented as an application-specific integrated circuit (ASIC).
[0053] Specific embodiments of this disclosure may be implemented by software comprising one or more instructions readable by a machine (e.g., electronic device 101) stored in a machine-readable storage medium (e.g., internal memory 1367 or external memory 138). For example, a machine may invoke and execute at least one instruction from the instructions stored in the storage medium, which in turn operates the machine to perform at least one function according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, when the storage medium is referred to as “non-transitory,” it is understood that the storage medium is tangible and does not include signals; rather, data is stored semi-permanently or temporarily in the storage medium.
[0054] According to embodiments, methods according to specific embodiments disclosed herein may be provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)) or through an app store (e.g., the Play Store). TM Online distribution of computer program products, or distribution of computer program products directly or online (e.g., downloading or uploading) between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be stored at least semi-permanently in a storage medium, or may be temporarily generated in a storage medium, such as the memory of a manufacturer's server, an app store's server, or a forwarding server.
[0055] According to a specific embodiment, each of the above-described elements (e.g., modules or programs) may include one or more entities, or some sub-elements may be omitted, or other sub-elements may be added. Optionally or additionally, multiple elements (e.g., modules or programs) may be integrated into one element. In this case, the integrated element may perform one or more functions of each of the multiple elements in the same or similar manner as they were performed by corresponding elements of the multiple components prior to integration. According to a specific embodiment, the operations performed by modules, programs, or other elements may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more operations may be performed in a different order or omitted, or one or more other operations may be added.
[0056] Figure 2 This is a block diagram illustrating a procedure according to a particular embodiment.
[0057] According to an embodiment, program 140 may include an operating system 142, middleware 144, or an application 146 executable in the operating system 142 for controlling one or more resources of the electronic device 101. Operating system 142 may include, for example, Android. TM iOS TM Windows TM Symbian TM Tizen TM Or Bada TM At least some of the programs 140 may be preloaded into electronic device 101 at the time of manufacture, or may be downloaded from external electronic device (e.g., electronic device 102 or 104) or server 108 or updated when used by a user.
[0058] Operating system 142 can control the management (e.g., allocation or retrieval) of one or more system resources (e.g., processes, memory, or power supply) of electronic device 101. Operating system 142 may additionally or optionally include one or more drivers for driving other hardware devices of electronic device 101 (e.g., input module 150, sound output module 155, display module 160, audio module 170, sensor module 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, user identification module 196, or antenna module 197).
[0059] Middleware 144 may provide various functions to application 146, making functions or information available from one or more resources of electronic device 101 usable by application 146. Middleware 144 may include, for example, application manager 201, window manager 203, multimedia manager 205, resource manager 207, power manager 209, database manager 211, package manager 213, connection manager 215, notification manager 217, location manager 219, graphics manager 221, security manager 223, telephone manager 225, or voice recognition manager 227.
[0060] Application Manager 201 manages the lifecycle of application 146. Window Manager 203 manages one or more GUI resources used on the screen. Multimedia Manager 205 identifies one or more formats for playing back media files and encodes or decodes the corresponding media files using a codec suitable for the selected format. Resource Manager 207 manages the source code of application 146 or the memory space of memory 130. Power Manager 209 manages, for example, the capacity, temperature, or power of battery 189 and uses the corresponding information to determine or provide relevant information for the operation of electronic device 101. According to an embodiment, Power Manager 209 may communicate with the Basic Input / Output System (BIOS) (not shown) of electronic device 101.
[0061] For example, database manager 211 can create, retrieve, or modify databases to be used by application 146. Package manager 213 can manage, for example, the installation or updates of applications distributed as package files. Connection manager 215 can manage, for example, wireless or direct connections between electronic device 101 and external electronic devices. Notification manager 217 can provide, for example, functionality for notifying the user of the occurrence of specified events (e.g., incoming calls, messages, or alarms). Location manager 219 can manage location information of electronic device 101. Graphics manager 221 can manage, for example, one or more graphical effects to be provided to the user or related user interface.
[0062] Security manager 223 can provide, for example, system security or user authentication. Telephone manager 225 can manage, for example, voice or video calling functions provided by electronic device 101. Voice recognition manager 227 can send user voice data to server 108 and receive commands from server 108 corresponding to functions performed in electronic device 101 based at least in part on voice data or text data converted from voice data. According to an embodiment, middleware 244 can dynamically remove some existing components or add new components. According to an embodiment, at least a portion of middleware 144 may be included as part of operating system 142 or implemented as software separate from operating system 142.
[0063] Application 146 may include a homepage application 251, a dialer application 253, an SMS / MMS application 255, an instant messaging (IM) application 257, a browser application 259, a camera application 261, an alarm clock application 263, a contacts application 265, a voice recognition application 267, an email application 269, a calendar application 271, a media player application 273, a photo album application 275, a watch application 277, a health application 279 (such as measuring biometric information, such as exercise or blood sugar) or an environmental information application 281 (such as measuring atmospheric pressure, humidity, or temperature information). According to an embodiment, application 146 may also include an information exchange application (not shown) capable of supporting information exchange between electronic device 101 and external electronic devices. The information exchange application may include, for example, a notification forwarding application configured to send specified information (e.g., a call, message, or alarm) to an external electronic device, or a device management application configured to manage external electronic devices. The notification forwarding application can send notification information corresponding to a specified event (e.g., email reception) generated in another application (e.g., email application 269) of electronic device 101 to an external electronic device. Alternatively or additionally, the notification forwarding application can receive notification information from the external electronic device and provide the received notification information to the user of electronic device 101.
[0064] The device management application can control the power (e.g., turning on or off) or functions (e.g., brightness, resolution, or focus) of an external electronic device that communicates with electronic device 101 or a component of electronic device 101 (e.g., a display module or camera module of the external electronic device). The device management application may additionally or optionally support the installation, removal, or updating of applications operating in the external electronic device.
[0065] Figure 3 An electronic device and a stylus are shown according to an embodiment.
[0066] Reference Figure 3 The electronic device 101 according to the embodiment can be implemented as a mobile device (such as a smartphone) and may include Figure 1 The configuration is shown. Additionally, the electronic device 101 can be connected to the stylus 300 and can perform various functions by interacting with the stylus 300. For example, the electronic device 101 can receive gesture input from the stylus 300 and can perform functions corresponding to the gesture input. The electronic device 101 is not limited to a smartphone and can be implemented as various types of devices that can interact with the stylus 300. For example, the electronic device 101 can be a device such as a foldable smartphone, a rollable smartphone, a tablet PC, a portable multimedia device, a portable medical device, or a camera.
[0067] Electronic device 101 may include a structure into which a stylus 300 can be inserted. Electronic device 101 may include a housing 210 and a hole 214 disposed in a portion of a side surface 210a of the housing 210. Electronic device 101 may include a receiving space 212 connected to the hole 214, and the stylus 300 can be inserted into the receiving space 212. Optionally, electronic device 101 may include a structure to which the stylus 300 can be attached. A magnet may be disposed on the side or rear surface of the housing 210, and the stylus 300 may be magnetically attached to the housing 210. In addition to the above, various structures for receiving the stylus 300 may be applied to electronic device 101.
[0068] Users can use the stylus 300 to input various commands to the electronic device 101 and perform various functions of the electronic device 101. For example, touch input when the stylus 300 is in contact with the display 160 of the electronic device 101, hover input when the stylus 300 is spaced apart from the display 160 and hovering at a predetermined distance from the display 160, or gesture input when the stylus 300 is in the air away from the display 160 can be performed. Gesture input in the air can be referred to as air motion input. User input using the stylus 300 is for illustrative purposes only and is not limited thereto, and may include various types of input.
[0069] As described above, electronic device 101 may include a display (or display module) 160, a communication module 190, a processor 120, and / or a memory 130. Processor 120 may be operatively and / or electrically connected to display 160, communication module 190, or memory 130. Memory 130 may store various instructions executable by processor 120. Instructions stored in memory 130 may include arithmetic operations, logical operations, data movement, or control instructions, such as input / output that can be processed by processor 120.
[0070] The processor 120 of the electronic device 101 can acquire position data of the stylus 300. Based on the position data, the processor 120 can calculate the distance between the stylus 300 and the display 160, and identify the arrangement state of the stylus 300 based on this distance. For example, the processor 120 can determine whether the stylus 300 is in a hovering state, spaced apart from the display 160 at a predetermined distance, or in an air state, spaced apart from the display 160 at a predetermined distance.
[0071] Additionally, when the stylus 300 is in a hovering or airborne state, the processor 120 of the electronic device 101 can receive gesture input from the stylus 300 via the communication module 190. The processor 120 of the electronic device 101 can detect gesture recognition errors in response to gesture input from the stylus 300 and can control the display 160 to display guidance content related to the gesture recognition error. Figure 6 Describe in detail the gesture recognition error and the corresponding guidance content displayed.
[0072] According to an embodiment, the display 160 may include at least one of a liquid crystal display (LCD), a thin-film transistor liquid crystal display (TFT-LCD), an organic light-emitting diode (OLED), a light-emitting diode (LED), an active-matrix organic LED (AMOLED), a micro LED, a mini LED, a flexible display, or a three-dimensional display. Additionally, some of the displays may be configured as transparent or light-transmitting displays, through which the outside can be seen. Such displays may be configured as transparent displays including transparent OLEDs (TOLEDs).
[0073] although Figure 3 The illustrated electronic device 101 has a strip-shaped or plate-shaped appearance, but this is not the limitation of certain embodiments of the present disclosure. For example, the illustrated electronic device 101 may be part of a rollable or foldable electronic device.
[0074] Figure 4 The detailed structure of the stylus according to an embodiment is shown.
[0075] Reference Figure 4 According to an embodiment, the stylus 300 may include a pen housing 303 forming the appearance of the stylus 300 and internal components inside the pen housing 303. Figure 4 The internal components may include all the various components installed inside the pen and can be inserted into the pen housing 303 in a single assembly operation.
[0076] The pen housing 303 may have an elongated shape between a first end 300a and a second end 300b, and may include a receiving space 301 within the pen housing 303. The pen housing 303 may have an elliptical cross-section including a major axis and a minor axis, and may be integrally formed as an elliptical cylinder. The receiving space 212 of the electronic device 101 may also have an elliptical cross-section to correspond to the shape of the pen housing 303. The pen housing 303 may comprise a synthetic resin (e.g., plastic) and / or a metallic material (e.g., aluminum). According to an embodiment, the second end 300b of the pen housing 303 may be formed of a synthetic resin material.
[0077] The internal components may have an elongated shape to correspond to the shape of the pen housing 303. The internal components may be generally divided into three parts along the longitudinal direction. For example, the internal components may include a pop-out member 310 disposed at a position corresponding to a first end 300a of the pen housing 303, a coil member 320 disposed at a position corresponding to a second end 300b of the pen housing 303, and a circuit board member 330 disposed at a position corresponding to the body of the housing.
[0078] The pop-out member 310 may include components for withdrawing the stylus 300 from the receiving space 212 of the electronic device 101. According to an embodiment, the pop-out member 310 may include a shaft 311, a pop-out body 312 disposed around the shaft 311 and forming the overall appearance of the pop-out member 310, and a button component 313. When the internal components are fully inserted into the pen housing 303, the portion including the shaft 311 and the pop-out body 312 may be surrounded by a first end 300a of the pen housing 303, and the button component 313 may be exposed outside the first end 300a. Multiple components (e.g., cam components or resilient components, not shown) may be disposed within the pop-out body 312 to form a push-pull structure. In one embodiment, the button component 313 may be substantially coupled to the shaft 311 for linear reciprocating motion relative to the pop-out body 312. According to a particular embodiment, the button component 313 may include a button with a locking structure, allowing a user to remove the stylus 300 using a fingernail. According to an embodiment, the stylus 300 may provide an alternative input method by including a sensor for detecting the linear reciprocating motion of the shaft 311.
[0079] The coil component 320 may include a pen tip 321 exposed to the outside of the second end 300b when the internal components are fully inserted into the pen housing 303, a sealing ring 322, a coil 323 wound multiple times, and / or a pen pressure sensing component 324 for obtaining pressure changes according to the pressure applied to the pen tip 321. The sealing ring 322 may include epoxy resin, rubber, polyurethane, or silicone resin. The sealing ring 322 provides water and dust protection for the user and protects the coil component 320 and the circuit board component 330 from water immersion or dust. According to an embodiment, the coil 323 may form a resonant frequency in a set frequency band (e.g., 500 kHz) and may be combined with at least one element (e.g., a capacitor element) to adjust the resonant frequency formed by the coil 323 within a certain range.
[0080] The circuit board component 330 may include a printed circuit board 332, a base 331 surrounding at least one surface of the printed circuit board 332, and an antenna. According to an embodiment, the base 331 may include a substrate mounting member 333 formed on its upper surface, the printed circuit board 332 is disposed on the substrate mounting member 333, and the printed circuit board 332 disposed on the substrate mounting member 333 may be fixed. The printed circuit board 332 may include an upper surface and a lower surface, and a variable capacitance capacitor or switch 334 connected to the coil 323 may be disposed on the upper surface, while a charging circuit, battery, or communication circuit may be disposed on the lower surface. The battery may include an electric double-layer capacitor (EDLC). The charging circuit may be located between the coil 323 and the battery, and may include a voltage detector circuit and a rectifier.
[0081] Antennas may include, for example Figure 4 The antenna structure 339 shown is an antenna embedded in a printed circuit board 332. According to a particular embodiment, a switch 334 may be disposed on the printed circuit board 332. A side button 337 disposed on the stylus 300 can be used to press the switch 334 and is exposed to the outside through a side opening 302 of the pen housing 303. The side button 337 is supported by a support member 338, and unless an external force is applied to the side button 337, the side button 337 can be restored or held in a predetermined position by the elastic restoring force provided by the support member 338.
[0082] The circuit board component 330 may include other sealing rings, such as O-rings. For example, O-rings formed of an elastic material may be disposed at both ends of the base 331 to form a sealing structure between the base 331 and the pen housing 303. In some embodiments, the support member 338 may partially contact the inner wall of the pen housing 303 around the side opening 302 to form a sealing structure. For example, the circuit board component 330 may also have a waterproof and dustproof structure similar to the sealing ring 322 of the coil component 320.
[0083] The stylus 300 may include a battery mounting member 335 formed on the upper surface of the base 331 to mount a battery 336 on the battery mounting member 335. The battery 336 mounted on the battery mounting member 335 may include, for example, a cylindrical battery.
[0084] The stylus 300 may include a microphone (not shown). The microphone may be directly connected to the printed circuit board 332, or connected to a separate flexible printed circuit board (FPCB) (not shown) connected to the printed circuit board 332. According to a particular embodiment, the microphone may be positioned parallel to the side button 337 in the longitudinal direction of the stylus 300.
[0085] Figure 5 This is a block diagram of a stylus according to an embodiment.
[0086] Reference Figure 5 The stylus 300 includes a processor 351, a memory 353, a resonant circuit 355, a charging circuit 357, a battery 359, a communication circuit 361, an antenna 363, a trigger circuit 365, and / or a sensor 367. The stylus 300 can implement certain embodiments of this disclosure even when at least some of the illustrated components are omitted or substituted. At least a portion of the processor 351, the resonant circuit 355, and / or the communication circuit 361 may be configured on a printed circuit board or in the form of a chip. The processor 351, the resonant circuit 355, and / or the communication circuit 361 may be electrically connected to the memory 353, the charging circuit 357, the battery 359, the antenna 363, or the trigger circuit 365. The stylus 300 according to an embodiment may include the resonant circuit 355 and a button.
[0087] The processor 351 can control each component of the stylus 300. For example, the processor 351 can control the communication circuit 361 so that user input using a button and / or gesture input from the stylus 300 detected by the sensor 367 is sent to the communication module 190 of the electronic device 101.
[0088] Processor 351 may include a general-purpose processor configured to execute custom hardware or software modules (e.g., applications). Processor 351 may include hardware components (functions) or software components (programs), including at least one of various sensors, data measurement modules, input / output interfaces, modules for managing the state or environment of stylus 300, or communication modules disposed within stylus 300. Processor 351 may include one or a combination of two or more of hardware, software, or firmware. According to an embodiment, processor 351 may receive a proximity signal corresponding to an electromagnetic field signal generated from a digital converter of electronic device 101 via resonant circuit 355. When a proximity signal is detected, processor 351 may control resonant circuit 355 to send an electromagnetic resonance (EMR) input signal to electronic device 101.
[0089] The memory 353 may store information related to the operation of the stylus 300. For example, the memory 353 may store information for communicating with the electronic device 101 and frequency information related to the input operation of the stylus 300.
[0090] The resonant circuit 355 may include at least one of a coil, an inductor, or a capacitor. The resonant circuit 355 may be used by the stylus 300 to generate a signal including a resonant frequency. For example, the stylus 300 may use at least one of electromagnetic resonance (EMR), active electrostatic discharge (AES), or electrocoupled resonance (ECR) methods to generate the signal. When the stylus 300 transmits a signal via the EMR method, the stylus 300 may generate a signal including the resonant frequency based on an electromagnetic field generated from the sensing panel of the electronic device 101. When the stylus 300 transmits a signal via the AES method, the stylus 300 may generate a signal using capacitive coupling with the electronic device 101. When the stylus 300 transmits a signal via the ECR method, the stylus 300 may generate a signal including the resonant frequency based on an electric field generated from a capacitive device of the electronic device 101. According to an embodiment, the resonant circuit 355 may be used to change the strength or frequency of the electromagnetic field according to the user's operating state. For example, the resonant circuit 355 can provide a frequency for recognizing hover inputs, drawing inputs, button inputs, gesture inputs, or erase inputs.
[0091] When the switching circuit is connected to the resonant circuit 355, the charging circuit 350 can rectify the resonant signal generated in the resonant circuit 355 into a DC signal and provide the rectified resonant signal to the battery 359. According to an embodiment, the stylus 300 can use the voltage level of the DC signal detected by the charging circuit 357 to identify whether the stylus 300 is inserted into the electronic device 101.
[0092] Battery 359 may be configured to store power for operation of stylus 300. Battery 359 may include, for example, a lithium-ion battery or a capacitor, and may be rechargeable or replaceable. According to an embodiment, battery 359 may be charged using power supplied from charging circuit 357 (e.g., a DC signal (direct current power)).
[0093] The communication circuit 361 may be a component for communicating with the communication module 190 of the electronic device 101, and may support short-range wireless communication methods (e.g., Bluetooth). TM Bluetooth TM Low power (BLE), wireless LAN). Communication circuit 361 can use buttons and / or sensors 367 to send wireless signals corresponding to user input to electronic device 101.
[0094] The communication circuit 361 can use a short-range communication method to send the status information and input information of the stylus 300 to the electronic device 101. For example, the communication circuit 361 can send the orientation information of the stylus 300 (e.g., motion sensor data) acquired by the trigger circuit 365, the voice information input through the microphone, and / or the remaining battery level information of the battery 359 to the electronic device 101.
[0095] Antenna 363 can be used to transmit signals or power to or from an external source (e.g., electronic device 101). According to an embodiment, stylus 300 may include a plurality of antennas 363, and at least one antenna 363 suitable for a communication method may be selected from the plurality of antennas 363. Communication circuitry 361 may exchange signals or power with external electronic device 101 through the selected at least one antenna 363.
[0096] Trigger circuit 365 can send a trigger signal to electronic device 101 based on the input signal from the button or the detection signal from sensor 367. For example, processor 351 can check the input method (e.g., touch or press) or button type (e.g., EMR button or BLE button) on the button of stylus 300 and send a trigger signal to electronic device 101 according to the button input.
[0097] The stylus 300 may include at least one sensor 367. Sensor 367 may include a gyroscope sensor and an accelerometer sensor, and may detect movement and motion of the stylus 300. Additionally, sensor 367 may include at least one of a battery sensor, pressure sensor, optical sensor, temperature sensor, geomagnetic sensor, or biosensor. Sensor 367 may generate electrical signals or data corresponding to the operating state of the stylus 300 or the external environmental state. Furthermore, the stylus 300 may include a microphone and a camera.
[0098] Figure 6 This is a flowchart describing a method of operating an electronic device according to an embodiment. Figure 6 The operation shown can be performed via... Figure 1 The electronic device 101 is implemented by the processor 120 executing instructions (commands).
[0099] Reference Figure 6 The electronic device 101 can wirelessly connect to the stylus 300 (operation 601). That is, the processor 120 of the electronic device 101 can control the communication module 190 to establish and maintain a wireless connection with the stylus 300. For example, the electronic device 101 can use Bluetooth. TM Connect to stylus 300.
[0100] The processor 120 of the electronic device 101 can determine a separated state in which the stylus 300 is separated from the display 160. The "separated state" can include a hovering state where the stylus 300 is spaced apart from the display 160 and located within a predetermined distance from the display 160, or an "air state" where the stylus 300 is spaced apart from the display 160 and also located beyond the predetermined distance from the display 160. The processor 120 of the electronic device 101 can acquire position data of the stylus 300 and determine whether the stylus 300 is in a hovering state or an air state based on the position of the stylus 300 (operation 602).
[0101] When the stylus 300 is in a hover or air state, the processor 120 of the electronic device 101 can receive gesture input from the stylus 300 (operation 603). Gesture input can be generated when a user holds the stylus 300 in their hand and moves the stylus to form a gesture while pressing the side button 337. Sensors 367 (such as gyroscope sensors and accelerometers) disposed in the stylus 300 can detect gesture inputs through movement of the stylus 300, and the stylus 300 can transmit the detected gesture input to the electronic device 101 using the communication circuit 361.
[0102] The processor 120 of electronic device 101 can detect gesture recognition errors (operation 604) regarding gesture input received from stylus 300. To perform the functions of electronic device 101 via gesture input from stylus 300, gesture input is typically first recognized. However, gesture recognition errors may occur when the recognition of gesture input encounters some errors or otherwise fails. For example, electronic device 101 may store multiple predetermined reference gestures corresponding to each function of electronic device 101, but when the gesture input from stylus 300 does not match any of the stored predetermined reference gestures, electronic device 101 can determine that a gesture recognition error has occurred.
[0103] The processor 120 of electronic device 101 can control display 160 to display guidance content related to gesture recognition errors (operation 605). The processor 120 of electronic device 101 can store multiple potential guidance contents (e.g., visual guidance prompts, illustrations, and other instructions prompting the user to perform a certain action) and map the stored guidance contents to corresponding types of gesture recognition errors. Guidance content may include floating icons and / or pop-up messages, but can be provided in various forms and is not limited thereto. Guidance content may include text, static images, and / or animated images. Guidance content including text may be displayed for a first predetermined time (e.g., three seconds). Guidance content including images may be displayed for a second predetermined time (e.g., five seconds). Additionally, when interaction with the stylus 300 occurs while the guidance content is being displayed, the processor 120 of electronic device 101 can immediately terminate the display of the guidance content. By viewing the guidance content, the user can identify that a gesture input error has occurred and can receive instructions on how to take remedial actions to resolve the error.
[0104] The processor 120 of the electronic device 101 can count the number of gesture recognition errors. When a first predetermined number of consecutive gesture recognition errors is detected (e.g., two), the processor 120 can control the display 160 to display guidance content. Additionally, when a predetermined second number of cumulative gesture recognition errors is detected (e.g., five), the processor 120 can control the display 160 to display guidance content. The processor 120 can initialize (e.g., reset to zero) the count of gesture recognition errors after outputting the guidance content on the display 160.
[0105] Figure 7 It is used for a more detailed description Figure 6 The flowchart of the method shown is shown.
[0106] Reference Figure 7 The processor 120 of electronic device 101 can receive gesture input from stylus 300 (operation 701). Furthermore, the processor 120 of electronic device 101 can detect gesture recognition errors in the received gesture input. For example, the processor 120 of electronic device 101 can detect a first gesture recognition error caused by gesture input in a hovering state, a second gesture recognition error caused by tilting of the stylus 300 in the air, or a third gesture recognition error caused by recognizing an abnormal gesture in the air. The processor 120 of electronic device 101 can control the display 160 to display guidance content corresponding to each of the first, second, or third gesture recognition errors, depending on which error actually occurred.
[0107] The processor 120 of electronic device 101 can detect errors in gesture input received in the hover state, indicating that a first gesture recognition error has occurred (operation 702). Electronic device 101 can execute functions when it detects button input generated by using the side button 337 of the stylus 300 when the stylus 300 is set to the hover state. However, in the hover state, there may be no executable function associated with the gesture input. Therefore, the processor 120 of electronic device 101 can determine that the gesture input received in the hover state is unprocessable. Therefore, gesture input generated in the hover state can be immediately detected as indicating a first gesture recognition error.
[0108] The processor 120 of the electronic device 101 can control the display 160 to display guidance content prompting the user to adjust the distance between the stylus 300 and the display 160 based on the detection of a first gesture recognition error in the hover state (operation 703). In other words, the processor 120 can control the display 160 to display first guidance content instructing the user to move the stylus 300 to a position beyond a predetermined distance from the display 160.
[0109] The processor 120 of the electronic device 101 can detect errors in a second gesture recognition error caused by the tilt of the stylus 300 when it is in an air-positioned state (operation 704). When a gesture input is received in the air-positioned state, the processor 120 can acquire tilt information of the stylus 300 relative to the ground (e.g., the Earth's plane). The tilt information can be sent from a sensor 367 disposed in the stylus 300. The processor 120 can detect a second gesture recognition error based on detecting a tilt value greater than or equal to a first angle (e.g., 60 degrees) and less than or equal to a second angle (e.g., 90 degrees). The second gesture recognition error can indicate a situation where the stylus 300 is substantially perpendicular to the ground. When the stylus 300 is perpendicular to the ground, the error rate relative to the coordinates indicating the position and movement of the stylus 300 may increase, which may lead to errors in gesture recognition operation.
[0110] The processor 120 of the electronic device 101 can control the display 160 to display guidance content prompting the user to adjust the tilt of the stylus 300 based on the detection of a second gesture recognition error (operation 705). In other words, the processor 120 can control the display 160 to display second guidance content instructing the user to adjust the tilt of the stylus 300 to less than a first angle.
[0111] Meanwhile, in some embodiments, the processor 120 of the electronic device 101 may, based on recognizing the gesture input as a predetermined exceptional gesture, omit the display of guiding content even when a tilt greater than or equal to a first angle (e.g., 60 degrees) and less than or equal to a second angle (e.g., 90 degrees) is detected, and instead execute the function corresponding to the exceptional gesture. The exceptional gesture may indicate a gesture in which the stylus 300 is likely to be perpendicular to the display when the gesture is input. For example, the exceptional gesture may be an upward gesture, a downward gesture, or a zigzag gesture. That is, in the gesture recognition processing of the processor 120, the temporary state in which the stylus 300 is perpendicular to the ground can be ignored relative to the exceptional gesture.
[0112] The processor 120 of electronic device 101 can detect a third gesture recognition error arising from detecting an abnormal gesture in the air (operation 706). The processor 120 can determine that a third gesture recognition error has occurred based on the detection that a gesture input received in the air differs from a known predetermined reference gesture. For example, reference gestures may include left, right, up, down, clockwise circular, counterclockwise circular, up-back, down-back, left-back, right-back, and / or zigzag gestures. However, when a gesture input by the user in the air does not correspond to any stored reference gesture, it can be determined that an abnormal gesture has been input. The processor 120 of electronic device 101 can control the display 160 to display third guidance content prompting the user to input a known reference gesture based on the detection of the third gesture recognition error (operation 707).
[0113] When no gesture recognition error is detected, the processor 120 of the electronic device 101 can execute functions corresponding to the gesture input (operation 708). For example, when a camera application is running, switching between the front and rear cameras can be performed in response to an up or down gesture. Switching to the previous mode can be performed in response to a left gesture, and switching to the next mode can be performed in response to a right gesture. Additionally, the camera can be zoomed in (e.g., enlarged) in response to a clockwise circular gesture, and zoomed out (e.g., reduced) in response to a counter-clockwise circular gesture. The functions corresponding to gesture input can be different for each application and can be changed according to user settings.
[0114] The processor 120 of the electronic device 101 can count the number of occurrences of each of the first gesture recognition error, the second gesture recognition error, and / or the third gesture recognition error. The processor 120 can control the display 160 to display first, second, or third guidance content based on the number of occurrences of each gesture recognition error. After the guidance content is displayed on the display 160, the processor 120 can initialize (e.g., reset to zero) the count of the number of gesture recognition errors associated with the displayed guidance content.
[0115] Figure 8 The hovering state of the electronic device and stylus according to an embodiment is shown. Figure 9 The operation of a stylus, processed to be effective in a hover state according to an embodiment, is shown. Figure 10 An example of the first guiding content displayed in response to a first gesture recognition error in a hover state is shown.
[0116] Reference Figure 8 and Figure 9 The hover state 920 refers to a state where the stylus 300 is spaced apart from the display 160 and within a predetermined distance "d" from the display 160. When the stylus 300 enters the hover area 901, a mark 801 indicating the position of the pen tip 321 can be displayed on the display 160. Figure 9 As shown, marker 801 can move in conjunction with the movement of stylus 300 within hover area 901, where hover area 901 is shown as an imaginary area above the display surface.
[0117] The air state 930 can refer to a state where the stylus 300 is spaced apart from the display 160 and located beyond a predetermined distance "d" from the display 160. In other words, the air state 930 can refer to a state where the stylus 300 is spaced apart from the display 160 and located outside the hovering area 901. The contact state 910 can refer to a state where the tip 321 of the stylus 300 is in direct contact with the display 160. The processor 120 of the electronic device 101 can acquire the position data of the stylus 300 and calculate the distance between the stylus 300 and the display 160 based on the position of the stylus 300.
[0118] When the stylus 300 receives button input via the side button 337 of the stylus 300 while the stylus 300 is spaced apart from the display 160 and within the hovering area 901 (in hovering state 920), the processor 120 of the electronic device 101 can execute the function corresponding to the button input. For example, when the user clicks the side button 337 of the stylus 300 once, a menu including selectable objects (e.g., icons selectable to execute the corresponding application) can be displayed on the display 160.
[0119] In some cases, the stylus 300 can enter a contact state 910 by contacting the display 160 with its tip 321, or a hover state 901 by moving the stylus 300 into the hover area 901. However, the transition to the contact state 910 or the hover state 901 can occur between pressing and releasing the side button 337 of the stylus 300. Therefore, due to the transitions between the states of the stylus 300 before the side button 337 is fully activated, the state in which the side button 337 is pressed may be uncertain, and the resulting input may be processed as invalid.
[0120] Reference Figure 10 The processor 120 of the electronic device 101 can determine that a gesture input from the stylus 300 (e.g., detected in hover state 920) has encountered a first gesture recognition error. When the first gesture recognition error is detected in hover state 920, first guidance content 1000 prompting the user to adjust the distance between the stylus 300 and the display 160 can be displayed on the display 160. The first guidance content 1000 may instruct the user to move the stylus 300 to a position beyond a predetermined distance "d" from the display 160. For example, the first guidance content 1000 may be provided as a floating icon and / or a pop-up message, and may include at least one of text, a still image, or a moving image.
[0121] Figure 11 This demonstrates a second gesture recognition error caused by inappropriate tilting of the stylus while it is in the air. Figure 12 An example of a second guide content displayed in response to a second gesture recognition error while in mid-air is shown.
[0122] Reference Figure 11 When a gesture input is received in mid-air, the processor 120 of the electronic device 101 can acquire tilt information of the stylus 300 relative to the ground (e.g., the Earth's plane). A second gesture recognition error can be detected when the tilt θ of the stylus 300 relative to the ground (e.g., the XY plane) is greater than or equal to a first angle (e.g., 60 degrees) and less than or equal to a second angle (e.g., 90 degrees). The second gesture recognition error indicates that the stylus 300 is substantially perpendicular to the ground.
[0123] The processor 120 of the electronic device 101 can acquire the position and motion detected by the sensor 367 of the stylus 300 as coordinate data, and the error rate of the coordinate data may increase when the tilt angle θ of the stylus 300 is close to vertical. This phenomenon (i.e., gimbal lock-up) can be caused by the operation of the gyroscope sensor included in the sensor 367. The gyroscope sensor may include a component called a gimbal, which can represent three degrees of freedom of rotation, such as roll, pitch, and yaw. In the case of gimbal lock-up, the two axes become parallel, so the rotation system is limited to two dimensions. When the stylus 300 is perpendicular to the ground, the Z-axis degree of freedom of the gyroscope sensor may be lost, and the processor 120 of the electronic device 101 may lose the coordinate data of the stylus 300. Therefore, gesture input when the stylus 300 is perpendicular to the ground may cause the processor 120 to fail to recognize the gesture.
[0124] However, the temporary verticality of the stylus 300 relative to the ground may not correspond to a second gesture recognition error. In other words, when a portion of the gesture path (e.g., 20% or less of the gesture path) is drawn vertically with the stylus 300 in an upright position, the processor 120 of the electronic device 101 is still able to recognize the input gesture without error.
[0125] Furthermore, even when the stylus 300 is perpendicular to the ground when a gesture input is received in mid-air, the processor 120 of the electronic device 101 can successfully process the gesture recognition based on identifying the gesture input as an exceptional gesture. In other words, even when the stylus 300 is detected to be tilted at a angle greater than or equal to a first angle and less than or equal to a second angle, the processor 120 can successfully recognize the exceptional gesture. However, when a significant portion (e.g., 70% or more of the gesture input) is performed while the stylus 300 is perpendicular to the ground, the processor 120 can determine that the exceptional gesture is unrecognizable and that a second gesture recognition error has occurred.
[0126] Reference Figure 12 The processor 120 of the electronic device 101 can, based on the detection of a second gesture recognition error while in the air, control the display 160 to display second guidance content 1200 prompting the user to adjust the tilt of the stylus 300. The processor 120 can control the display 160 to display the second guidance content 1200 instructing the user to adjust the tilt of the stylus 300 to less than a first angle. Figure 12 The second guidance content 1200 shown can prompt input gestures while keeping the stylus 300 horizontal relative to the ground. The second guidance content 1200 can be provided as a floating icon and / or a pop-up message, and may include at least one of text, a still image, or a moving image.
[0127] Figure 13 An example of a reference gesture that can be recognized in mid-air is shown.
[0128] Reference Figure 13 The electronic device 101 can provide reference gestures that can be recognized by the processor 120. For example, reference gestures may include a left gesture 1301, a right gesture 1302, an up gesture 1303, a down gesture 1304, a clockwise circular gesture 1305, a counter-clockwise circular gesture 1306, an up-back gesture 1307, a down-back gesture 1308, a left-back gesture 1309, a right-back gesture 1310, and / or a zigzag gesture 1311. The function performed by the electronic device 101 in response to each gesture can be preset or can be changed according to user settings. Furthermore, the function performed in response to each gesture can be different for each application.
[0129] For example, while the camera app is running, a left gesture 1301 can switch to the previous mode, and a right gesture 1302 can switch to the next mode. Additionally, an up gesture 1303 or a down gesture 1304 can switch between the front and rear cameras. Furthermore, a clockwise circular gesture 1305 can zoom in (enlarge), and a counter-clockwise circular gesture 1306 can zoom out (shrink).
[0130] The upward return gesture 1307 can be a gesture that returns after moving the stylus 300 upward, the downward return gesture 1308 can be a gesture that returns after moving the stylus 300 downward, the left return gesture 1309 can be a gesture that returns after moving the stylus 300 to the left, and the right return gesture 1310 can be a gesture that returns after moving the stylus 300 to the right. Additionally, the zigzag gesture 1311 can be a gesture that repeatedly moves the stylus 300 to the left and right. The electronic device 101 can perform the functions corresponding to each of the upward return gesture 1307, downward return gesture 1308, left return gesture 1309, right return gesture 1310, and zigzag gesture 1311.
[0131] Figure 14 An example of a method for recognizing hand gesture input from a stylus is shown.
[0132] Reference Figure 14The processor 120 of the electronic device 101 can acquire gesture attributes from the gesture input of the stylus 300, including a start point 1401, an end point 1402, a gesture path 1403, the width w of the gesture, the height h of the gesture, the direction of movement, and / or the shape. Additionally, the gesture attributes may include an angle “a” formed by the center point of the start point 1401, the end point 1402, and the gesture path 1403. The processor 120 can determine a reference gesture that matches the acquired gesture attributes. For example, when the gesture path rotates to the left, when angle A is an acute angle, and when the ratio of width w to height h meets a predetermined reference value, the processor 120 can recognize the input gesture as a counter-clockwise circular gesture.
[0133] Additionally, the processor 120 of the electronic device 101 can calculate the gesture size and / or gesture input speed by analyzing gesture attributes. The processor 120 can calculate the gesture size based on the gesture's width w and height h, and can calculate the gesture input speed based on the distance between points in a coordinate system. When the distance between points is sufficiently short (e.g., based on a distance threshold), the gesture input speed can be determined to be low.
[0134] Figure 15 This illustrates the occurrence of a third gesture recognition error based on the identified abnormal gesture. Figure 16 An example of third-party guidance content displayed in response to a third gesture recognition error is shown.
[0135] Reference Figure 15 The processor 120 of the electronic device 101 can acquire the gesture attributes of a gesture input received in the air and analyze the gesture attributes to determine whether an abnormal gesture has been detected. When a gesture received in the air does not correspond to any known reference gesture, the processor 120 can determine that an abnormal gesture has been input. The processor 120 can determine a third gesture recognition error based on the identified abnormal gesture. For example, such as... Figure 15 As shown, when a gesture path 1500, which extends from the starting point 1501 to the ending point 1502 of the gesture input, bends to the right to reverse the direction of movement of the gesture path 1500, the processor 120 of the electronic device 101 can recognize the gesture input from the stylus 300 as an abnormal gesture. Even when the user intends to draw a counterclockwise circular gesture in the reference gesture, it can be regarded as a third gesture recognition error because the input gesture does not correspond to the reference gesture in practice.
[0136] Additionally, when the user's gesture input is too large, the gesture path can be cut off at a position outside the gesture recognition range, and when the user's gesture input is too small, the processor 120 may not be able to obtain the gesture attributes. That is to say, even when the size of the input gesture is larger than a predetermined reference size threshold or smaller than a predetermined reference size threshold, the processor 120 of the electronic device 101 can still identify that a third gesture recognition error has occurred.
[0137] Reference Figure 16 The processor 120 of the electronic device 101 can, based on the detection of a third gesture recognition error in the air, control the display 160 to display third guidance content 1600 prompting for input of a reference gesture. For example... Figure 16 As shown, the processor 120 can control the display 160 to display third guidance content 1600 indicating that the user should input a gesture smaller than the previously input gesture, based on the size of the input gesture. Additionally, the processor 120 can control the display 160 to display third guidance content 1600 for guiding the user to input a gesture larger than the previously input gesture, based on the size of the input gesture.
[0138] As described above, the disclosed electronic device and method of operating the electronic device can detect errors related to hand gesture input with a stylus and provide guidance on the errors. Users can recognize the errors in the gesture input through the guidance and be guided to take actions to resolve them.
Claims
1. An electronic device comprising: monitor; The communication module is configured to establish a wireless connection with the stylus; The processor is operatively coupled to the display and communication module; as well as Memory, which stores programming instructions. The programming instructions can be executed by a processor to cause the electronic device to perform the following operations: When the stylus is separated from the display, it is identified whether the stylus is in a hover state or in an air state. In the hover state, the stylus is separated from the display and is within a predetermined distance from the display. In the air state, the stylus is separated from the display and is positioned outside the predetermined distance from the display. Receives gesture input from the stylus via the communication module; Based on the received gesture input, a second gesture recognition error was determined to have occurred due to the tilt of the stylus while the hand was in the air; and The control display shows visual guidance corresponding to the second gesture recognition error.
2. The electronic device according to claim 1, wherein, The processor is configured as follows: Based on the received gesture input, it is determined that a first gesture recognition error has occurred, either due to gesture input during hovering or a third gesture recognition error, including abnormal gestures during airborne operations. The control display shows visual guidance corresponding to the first gesture recognition error or the third gesture recognition error.
3. The electronic device according to claim 2, in, Visual guidance corresponding to the first gesture recognition error is configured to prompt the user to move the stylus to a position beyond the predetermined distance.
4. The electronic device according to claim 2, wherein, The processor is configured as follows: When receiving gesture input while in the air, simultaneously obtain the tilt value of the stylus relative to the ground. Specifically, a second gesture recognition error is determined to have occurred based on the detection that the tilt value is greater than or equal to the first angle and less than or equal to the second angle.
5. The electronic device according to claim 4, wherein, The visual guidance corresponding to the second gesture recognition error is configured to prompt the user to adjust the tilt of the stylus to less than the first angle.
6. The electronic device according to claim 4, wherein, The processor is configured as follows: Based on the detection that the gesture input includes a predetermined exception gesture, the display of the visual guidance is omitted, and the function corresponding to the exception gesture is executed.
7. The electronic device according to claim 2, wherein, Abnormal gestures in mid-air are identified by their differences from predetermined reference gestures in mid-air.
8. The electronic device according to claim 7, wherein, Visual guidance corresponding to a third gesture recognition error is configured to prompt the user to input the predetermined reference gesture.
9. The electronic device according to claim 1, wherein, The processor is configured to: The number of times the second gesture recognition error occurs is counted. Specifically, when the number of occurrences of the second gesture recognition error is continuously counted and reaches a predetermined first number, or when the number of occurrences of the second gesture recognition error is not continuously counted and accumulates to reach a predetermined second number, visual guidance corresponding to the second gesture recognition error is displayed.
10. The electronic device according to claim 9, wherein, The processor is configured as follows: The count of occurrences is reset after the visual guidance is displayed.
11. A method of operating an electronic device, the method comprising: A communication connection is established with the stylus via the communication module; The processor identifies that the stylus is separated from the display. The system identifies whether the stylus is in a hover state or in an air state. In the hover state, the stylus is spaced apart from the display and located within a predetermined distance from the display. In the air state, the stylus is spaced apart from the display and located outside the predetermined distance from the display. Receive gesture input from the stylus; Based on the received gesture input, a second gesture recognition error was determined to have occurred due to the tilt of the stylus while the hand was in the air; and Visual guidance corresponding to the second gesture recognition error is displayed on the monitor.
12. The method of claim 11, further comprising: Based on the received gesture input, it is determined that a first gesture recognition error has occurred, either due to gesture input during hovering or a third gesture recognition error, including abnormal gestures during airborne operations. Among them, visual guidance corresponding to the first gesture recognition error or the third gesture recognition error is displayed on the screen.
13. The method according to claim 12, wherein, Visual guidance corresponding to the first gesture recognition error is configured to prompt the user to move the stylus to a position beyond the predetermined distance.
14. The method of claim 12, further comprising: When receiving gesture input while in the air, simultaneously obtain the tilt value of the stylus relative to the ground. Specifically, a second gesture recognition error is determined to have occurred based on the detection that the tilt value is greater than or equal to the first angle and less than or equal to the second angle.
15. The method according to claim 14, wherein, The visual guidance corresponding to the second gesture recognition error is configured to prompt the user to adjust the tilt of the stylus to less than the first angle.
Citation Information
Patent Citations
Teaching gestures with offset contact silhouettes
US20110117535A1
Portable device and guide information provision method thereof
US20140059428A1
Electronic device for identifying gesture performed by stylus pen and method for operating the same
US20210034188A1
