Electronic device including rollable display and the method for handwriting recognition

Abstract

An electronic device according to various embodiments comprises a rollable display, a memory, and a processor operatively connected to the rollable display and the memory, wherein the processor displays content consisting of at least one stroke on the rollable display, expands the rollable display in response to a user's request to expand the display, and while performing the expansion of the rollable display, analyzes at least one stroke included in a first area of ​​the content at a first time point and displays a first text corresponding to the analyzed stroke on one side of the content, and at a second time point analyzes at least one stroke included in a second area including the first area and displays a second text corresponding to the analyzed stroke on one side of the content. Various other embodiments are possible.

Description

Technology Field

[0001] This document relates to an electronic device, and is an invention relating to a method for recognizing handwriting in an electronic device including, for example, a rollable display. Background Technology

[0003] With the development of mobile communication technology and hardware / software technology, portable electronic devices (hereinafter referred to as "electronic devices") have become capable of implementing various functions beyond conventional calling functions. The electronic device may incorporate a touch sensor and can detect user touch input or gestures on a display using the touch sensor. The electronic device can detect user input to the device and display a stroke on the display.

[0004] Additionally, the electronic device may be a rollable device capable of expanding the display by sliding horizontally (or vertically). The electronic device can expand the rollable display in response to user input or button input, and may provide various functions that can be used in conjunction with this. For example, the electronic device may display additional characters or graphic objects in the expanded area of ​​the display that occurs as the rollable display expands. The electronic device may recognize handwriting entered by the user and convert it into text (handwriting recognition, HWR).

[0005] According to various embodiments, handwriting recognition technology that can be provided by an electronic device may include offline handwriting recognition, which converts characters contained in an image or picture file into text, and online handwriting recognition, which detects tip movement of a pointing device such as a PDA or digitizer and converts it into text. The problem to be solved

[0007] In conventional rollable devices, when recognizing strokes, text was analyzed and displayed after the rollable display had fully expanded. Using word recognition technology, sentences were analyzed during the screen expansion process and then displayed all at once after the expansion was complete. This resulted in a delay issue, as text generated during the expansion process could not be displayed.

[0008] In addition, since all strokes displayed on the screen are analyzed at once, there were problems such as slow result output or buffering when hardware performance was poor or when the amount of strokes on the screen was large.

[0009] The various embodiments of this document aim to provide a method for performing rapid result output by efficiently utilizing processor resources when performing handwriting recognition with the expansion of a display on a rollable device as described above. means of solving the problem

[0011] An electronic device according to various embodiments comprises a rollable display, a memory, and a processor operatively connected to the rollable display and the memory, wherein the processor displays content consisting of at least one stroke on the rollable display, expands the rollable display in response to a user's request to expand the display, and while performing the expansion of the rollable display, analyzes at least one stroke included in a first area of ​​the content at a first time point and displays a first text corresponding to the analyzed stroke on one side of the content, and at a second time point analyzes at least one stroke included in a second area including the first area and displays a second text corresponding to the analyzed stroke on one side of the content.

[0012] A handwriting recognition method using a rollable display according to various embodiments includes an operation of displaying content consisting of at least one stroke on a rollable display, and an operation of expanding the rollable display in response to a user's request for display expansion. During the expansion of the rollable display, the method may include an operation of analyzing at least one stroke included in a first area of ​​the content at a first time point, an operation of displaying a first text corresponding to the analyzed stroke on one side of the content, an operation of analyzing at least one stroke included in a second area including the first area at a second time point, and an operation of displaying a second text corresponding to the analyzed stroke on one side of the content. Effects of the invention

[0014] According to various embodiments, the electronic device can rapidly output results for a given area by analyzing a given area while the rollable display is expanding and recognizing strokes in word area units for each line to display the execution results. Accordingly, the electronic device can provide a seamless display to the user even when there are many strokes to process, or when the load on the electronic device is heavy or specifications are poor.

[0015] In addition, effects that can be obtained or predicted by various embodiments of the electronic device will be disclosed directly or implicitly in the detailed description of the embodiments of the electronic device. For example, various effects predicted according to various embodiments of the electronic device will be disclosed in the detailed description to be set forth below. Brief explanation of the drawing

[0017] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments. FIG. 2a is a front perspective view of an electronic device in a closed state according to one embodiment. FIG. 2b is a rear perspective view of an electronic device in a closed state according to one embodiment. FIG. 3a is a front perspective view of an electronic device in an open state according to one embodiment. FIG. 3b is a rear perspective view of an electronic device in an open state according to one embodiment. FIG. 4 illustrates the sequential recognition of handwriting as the display of an electronic device expands according to various embodiments. FIG. 5 is a block diagram of an electronic device according to various embodiments. FIGS. 6a and 6b are drawings illustrating an electronic device according to various embodiments analyzing a stroke to generate a line list. FIG. 7 illustrates an analysis area that varies according to the reference time of an electronic device according to various embodiments. FIG. 8 illustrates an electronic device according to various embodiments recognizing word regions of content by reference time. FIG. 9 illustrates an electronic device according to various embodiments recognizing word regions over time using a separate thread for each line. FIG. 10 is a diagram showing that stroke recognition is delayed due to performance delay of an electronic device according to various embodiments. FIGS. 11a and FIGS. 11b illustrate embodiments in which the display of an electronic device according to various embodiments expands vertically. FIG. 12 illustrates an embodiment in which the display of an electronic device according to various embodiments displays text without expanding. FIG. 13 is a flowchart of an electronic device according to various embodiments. Specific details for implementing the invention

[0018] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0019] In describing the embodiments, technical details that are well known in the technical field to which the present invention belongs and are not directly related to the present invention are omitted. Additionally, detailed descriptions of components having substantially the same configuration and function are omitted.

[0020] For the same reason, some components in the attached drawings are exaggerated, omitted, or schematically depicted, and the size of each component does not entirely reflect its actual size. Accordingly, the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0021] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0022] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0023] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0024] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0025] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0026] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0027] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0028] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0029] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0030] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0031] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0032] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0033] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0034] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0035] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0036] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0037] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0038] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0039] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0040] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0041] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0042] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0043] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0044] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0045] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0046] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0047] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0048] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0049] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0051] FIG. 2a is a front perspective view of an electronic device (200) in a closed state according to one embodiment. FIG. 2b is a rear perspective view of an electronic device (200) in a closed state according to one embodiment. FIG. 3a is a front perspective view of an electronic device (200) in an open state according to one embodiment. FIG. 3b is a rear perspective view of an electronic device (200) in an open state according to one embodiment.

[0052] According to various embodiments, the electronic device (200) of FIG. 2a may include the electronic device (101) of FIG. 1.

[0053] Referring to FIGS. 2a, 2b, 3a, and 3b, in one embodiment, the electronic device (200) may be implemented to allow the screen (2301) to be extended in a sliding manner. For example, the screen (2301) may be an area of ​​the flexible display (230) that is visible to the outside. FIGS. 2a and 2b illustrate the electronic device (200) in a state where the screen (2301) is not extended, and FIGS. 3a and 3b illustrate the electronic device (200) in a state where the screen (2301) is extended. The state where the screen (2301) is not extended is a state in which the sliding plate (220) for the sliding motion of the display (230) is not slid out, and may be referred to as the 'closed state' below. The expanded state of the screen (2301) is a maximally expanded state in which the screen (2301) is no longer expanded by the sliding out of the sliding plate (220), and may be referred to as the 'open state' below. For example, the sliding out may be the sliding plate (220) moving at least partially in a first direction (e.g., +x axis direction) when the electronic device (200) is switched from a closed state to an open state. According to various embodiments, the open state may be defined as a state in which the screen (2301) is expanded compared to a closed state, and may provide screens of various sizes depending on the movement position of the sliding plate (220). According to various embodiments, an intermediated state may refer to a state between the closed state of FIG. 2a and the open state of FIG. 3a. The screen (2301) may include an active area of ​​a flexible display (230) that is visually exposed to enable the output of an image, and the electronic device (200) may adjust the active area according to the movement of the sliding plate (220) or the movement of the flexible display (230). In the following description, the open state may refer to a state in which the screen (2301) is fully extended.In some embodiments, a flexible display (230) that is slidably positioned in the electronic device (200) of FIG. 2a to provide a screen (2301) may be referred to as a 'slide-out display' or an 'expandable display'.

[0054] According to one embodiment, the electronic device (200) may include a sliding structure associated with a flexible display (230). For example, when the flexible display (230) is moved by an external force to a set distance, due to the elastic structure included in the sliding structure, it may be switched from a closed state to an open state or from an open state to a closed state without further external force (e.g., semi-automatic slide operation).

[0055] According to some embodiments, when a signal is generated through an input device included in the electronic device (200), the electronic device (200) can be switched from a closed state to an open state, or from an open state to a closed state, due to a driving device such as a motor connected to the flexible display (230). For example, when a signal is generated through a hardware button, or a software button provided through a screen, the electronic device (200) can be switched from a closed state to an open state, or from an open state to a closed state.

[0056] According to various embodiments, when a signal is generated from various sensors, such as a pressure sensor, the electronic device (200) may be switched from a closed state to an open state, or from an open state to a closed state. For example, when the electronic device (200) is carried or held by hand, a squeeze gesture in which a part of the hand (e.g., palm or fingers) presses within a designated section of the electronic device (200) may be detected by the sensor, and in response, the electronic device (200) may be switched from a closed state to an open state, or from an open state to a closed state.

[0057] According to one embodiment, the display (230) may include a second section (②) (see FIG. 3a). The second section (②) may include an extended portion of the screen (2301) when the electronic device (200) is switched from a closed state to an open state. When the electronic device (200) is switched from a closed state to an open state, the second section (②) is slid out from the internal space of the electronic device (200), thereby expanding the screen (2301). When the electronic device (200) is switched from an open state to a closed state, at least a portion of the second section (②) is slid into the internal space of the electronic device (200), thereby shrinking the screen (2301). When the electronic device (200) is switched from an open state to a closed state, at least a portion of the second section (②) may be bent and moved into the internal space of the electronic device (200). For example, the flexible display (230) may include a flexible substrate (e.g., a plastic substrate) formed from a polymer material including polyimide (PI (polyimide)) or polyester (PET (polyester)). The second section (②) is a part of the flexible display (230) that bends when the electronic device (200) switches between an open state and a closed state, and may be referred to, for example, as a bendable section. In the following description, the second section (②) will be referred to as a bendable section.

[0058] According to one embodiment, the electronic device (200) may include a housing (210), a sliding plate (220), or a flexible display (230).

[0059] The housing (or case) (210) may include, for example, a back cover (212), a first side cover (213), or a second side cover (214). The back cover (212), the first side cover (213), or the second side cover (214) may be connected to a support member (not shown) located inside the electronic device (200) and may form at least a part of the exterior of the electronic device (200).

[0060] The back cover (212) may, for example, form at least a portion of the rear surface (200B) of the electronic device (200). In one embodiment, the back cover (212) may be substantially opaque. For example, the back cover (212) may be formed by coated or colored glass, ceramic, polymer, metal (e.g., aluminum, stainless steel (STS), or magnesium), or a combination of at least two of the above materials. According to some embodiments, when the bendable section (②) of the flexible display is retracted into the internal space of the housing (210) (e.g., closed), at least a portion of the bendable section (②) may be positioned so that it is visible from the outside through the back cover (212). In this case, the back cover (212) may be formed of a transparent material and / or a translucent material.

[0061] According to one embodiment, the back cover (212) may include a flat portion (212a) and curved portions (212b, 212c) located opposite each other with the flat portion (212a) in between. The curved portions (212b, 212c) are formed adjacent to each of the relatively long edges (not shown) on both sides of the back cover (212) and may be curved and seamlessly extended toward a screen located opposite the back cover (212). According to some embodiments, the back cover (212) may include one of the curved portions (212b, 212c) or be implemented without the curved portions (212b, 212c).

[0062] According to one embodiment, the first side cover (213) and the second side cover (214) may be positioned opposite each other. For example, the first side cover (213) and the second side cover (214) may be positioned opposite each other with the flexible display (230) in between in a second direction (e.g., y-axis direction) that is orthogonal to the first direction (e.g., +x-axis direction) of the sliding plate (220) sliding out. The first side cover (213) may form at least a portion of the first side (213a) of the electronic device (200), and the second side cover (214) may form at least a portion of the second side (214a) of the electronic device (200) facing in the opposite direction to the first side (213a). The first side cover (213) may include a first rim portion (or, first rim) (213b) extending from the edge of the first side (213a). For example, the first rim portion (213b) may form at least a part of the bezel on one side of the electronic device (200). The second side cover (214) may include a second rim portion (or, second rim) (214b) extending from the edge of the second side (214a). For example, the second rim portion (214b) may form at least a part of the bezel on the other side of the electronic device (200). According to one embodiment, in the closed state of FIG. 2a, the surface of the first edge portion (213b), the surface of the second edge portion (214b), and the surface of the sliding plate (220) are smoothly connected to form a one-sided curved portion (not shown) corresponding to the first curved portion (230b) of the screen (2301). According to various embodiments, the surface of the first edge portion (213b) or the surface of the second edge portion (214b) may include a other-sided curved portion (not shown) corresponding to the second curved portion (230c) of the screen (2301) located opposite to the first curved portion (230b).

[0063] According to one embodiment, the sliding plate (220) may slide on a support member (not shown) located inside the electronic device (200). At least a portion of the flexible display (230) may be placed on the sliding plate (220), and the closed state of FIG. 2a or the open state of FIG. 3a may be formed based on the position of the sliding plate (220) on the support member. According to one embodiment, the flexible display (230) may be attached to the sliding plate (120) through an adhesive member (or adhesive member) (not shown). According to one embodiment, the adhesive member may include a thermal adhesive member, a photoreactive adhesive member, a general adhesive and / or double-sided tape. According to some embodiment, the flexible display (230) may be placed and fixed on the sliding plate (220) by slidingly inserting it into a recess formed in the sliding plate (220). The sliding plate (230) serves to support at least a portion of the flexible display (230), and in some embodiments may be referred to as a display support structure.

[0064] According to one embodiment, the sliding plate (220) may include a third edge portion (220b) that forms the outer surface of the electronic device (200) (e.g., a surface exposed to the outside to form the exterior of the electronic device (200)). For example, the third edge portion (220b) may form a bezel around the screen together with the first edge portion (213b) and the second edge portion (214b) in the closed state of FIG. 2a. The third edge portion (220b) may extend in a second direction (e.g., the y-axis direction) to connect one end of the first side cover (213) and one end of the second side cover (214) in the closed state. For example, in the closed state of FIG. 2a, the surface of the third edge portion (220b) may be smoothly connected to the surface of the first edge portion (213b) and / or the surface of the second edge portion (214b).

[0065] According to one embodiment, due to the sliding out of the sliding plate (220), at least a portion of the bendable section (②) comes out from inside the electronic device (200), and the screen (2301) can be provided in an extended state (e.g., open state) as in FIG. 3a.

[0066] According to one embodiment, in the closed state of FIG. 2a, the screen (2301) may include a flat section (230a) and a first curved section (230b) and / or a second curved section (230c) located opposite each other with the flat section (230a) in between. For example, the first curved section (230b) and the second curved section (230c) may be substantially symmetrical with respect to the flat section (230a). For example, in the closed state of FIG. 2a, the first curved section (230b) and / or the second curved section (230c) may be positioned corresponding to the curved sections (212b, 212c) of the back cover (212), respectively, and may be curved toward the back cover (212). When transitioning from the closed state of FIG. 2a to the open state of FIG. 3a, the flat section (230a) may be extended. For example, a portion of the bendable section (②) forming the second curved section (230c) in the closed state of FIG. 2a may be included in the expanded planar section (230a) when transitioning from the closed state of FIG. 2a to the open state of FIG. 3a, and may be formed as another portion of the bendable section (②).

[0067] According to one embodiment, the electronic device (200) may include an opening (not shown) for inserting or withdrawing a bendable section (②), and / or a pulley (not shown) located at the opening. The pulley may be positioned corresponding to the bendable section (②), and the movement of the bendable section (②) and the direction of movement thereof may be guided through the rotation of the pulley in the transition between the closed state of FIG. 2a and the open state of FIG. 3a. A first curved surface (230b) may be formed corresponding to a curved surface formed on one side of the sliding plate (220). A second curved surface (230c) may be formed by a portion of the bendable section (②) corresponding to the curved surface of the pulley. The first curved surface (230c) may be positioned on the opposite side of the second curved surface (230b) in the closed or open state of the electronic device (200) to improve the aesthetics of the screen (2301). According to some embodiments, the planar portion (230a) may be implemented in an extended form without the first curved portion (230b).

[0068] According to one embodiment, the flexible display (230) may further include a touch sensing circuit (e.g., a touch sensor). According to various embodiments (not shown), the flexible display (230) may be combined with or placed adjacent to a pressure sensor capable of measuring the intensity (pressure) of a touch, and / or a digitizer capable of detecting a magnetic field-based pen input device (e.g., a stylus pen). For example, the digitizer may include a coil member placed on a dielectric substrate to detect an electromagnetic induction-based resonant frequency applied from the pen input device.

[0069] According to one embodiment, the electronic device (200) may include a microphone hole (251) (e.g., input module (150) of FIG. 1), a speaker hole (252) (e.g., sound output module (155) of FIG. 1), a connector hole (253) (e.g., connection terminal (178) of FIG. 1), a camera module (254) (e.g., camera module (180) of FIG. 1), or a flash (255). According to various embodiments, the flash (255) may be implemented by being included in the camera module (254). In some embodiments, the electronic device (200) may omit at least one of the components or additionally include other components.

[0070] A microphone hole (251) may be formed in at least a portion of the second side (214a) corresponding to a microphone (not shown) located inside the electronic device (200), for example. The location of the microphone hole (251) may vary and is not limited to the embodiment of FIG. 2a. According to some embodiments, the electronic device (200) may include a plurality of microphones capable of detecting the direction of sound.

[0071] A speaker hole (252) may be formed in at least a portion of the second side (214a) corresponding to a speaker located inside the electronic device (200), for example. The location of the speaker hole (252) may vary and is not limited to the embodiment of FIG. 2a. According to various embodiments, the electronic device (200) may include a receiver hole for communication. In some embodiments, the microphone hole (251) and the speaker hole (252) may be implemented as a single hole, or the speaker hole (252) may be omitted, such as in a piezo speaker.

[0072] A connector hole (253) may be formed in at least a portion of the second side (214a) corresponding to a connector (e.g., a USB connector) located inside the electronic device (200), for example. The electronic device (200) may transmit and / or receive power and / or data to and from an external electronic device electrically connected to the connector through the connector hole (253). The location of the connector hole (253) may vary and is not limited to the embodiment of FIG. 2a.

[0073] A camera module (254) and a flash (255) may be located, for example, on the rear (200B) of the electronic device (200). The camera module (154) may include one or more lenses, an image sensor, and / or an image signal processor. The flash (255) may include, for example, a light-emitting diode or a xenon lamp. In some embodiments, two or more lenses (infrared camera, wide-angle and telephoto lenses) and image sensors may be located on one side of the electronic device (200). According to various embodiments, the electronic device (200) may include a plurality of camera modules, not limited to the embodiment of FIG. 2b or 3b. The camera module (254) may be one of the plurality of camera modules. For example, the electronic device (200) may include a plurality of camera modules (e.g., dual camera, or triple camera), each having different attributes (e.g., field of view) or functions. For example, a plurality of camera modules (e.g., camera module (254)) including lenses having different angles of view may be configured, and the electronic device (200) may control the change of the angle of view of the camera module performed in the electronic device (200) based on the user's selection. Additionally, the plurality of camera modules may include at least one of a wide-angle camera, a telephoto camera, a color camera, a monochrome camera, or an IR (infrared) camera (e.g., a TOF (time of flight) camera, a structured light camera). According to one embodiment, the IR camera may be operated as at least part of a sensor module (not shown).

[0074] According to various embodiments (not shown), the electronic device (200) may further include a camera module (e.g., front camera) that generates an image signal based on light received through one side (e.g., front (200A)) of the electronic device (200) positioned in the direction facing the screen (2301). For example, the camera module (254) may be located inside the housing (210) aligned with an opening (e.g., through hole, or notch) formed in the flexible display (230), not limited to the embodiment of FIG. 2b or 3b. The camera module (254) may generate an image signal by receiving light through the opening and a portion of a transparent cover that overlaps with the opening. The transparent cover serves to protect the flexible display (230) from the outside and may include a material such as polyimide or ultrathin glass (UTG).

[0075] According to some embodiments, not limited to the embodiments of FIG. 2b or 3b, the camera module (254) may be positioned at the bottom of at least a portion of the screen (2301) of the flexible display (230), and the position of the camera module (254) may not be visually distinguishable (or exposed) and may perform related functions (e.g., image capture). In this case, for example, when viewed from above the screen (2301) (e.g., when viewed in the -z axis direction), the camera module (254) may be positioned to overlap at least a portion of the screen (2301) so as to acquire an image of an external subject without being exposed to the outside.

[0076] According to various embodiments, the electronic device (200) may further include a key input device (e.g., the input module (150) of FIG. 1). The key input device may be located, for example, on a first side (213a) of the electronic device (200) formed by a first side cover (213). In some embodiments, the key input device may include at least one sensor module.

[0077] According to various embodiments, the electronic device (200) may include various sensor modules (e.g., sensor module (176) of FIG. 1). The sensor module may generate electrical signals or data values ​​corresponding to the internal operating state of the electronic device (200) or the external environmental state. For example, the sensor module may include a proximity sensor that generates a signal regarding the proximity of an external object based on light received through the front (200A) of the electronic device (200) positioned in the direction facing the screen (2301). In another example, the sensor module may include various biometric sensors, such as a fingerprint sensor or an HRM sensor, for detecting biometric information based on light received through the front (200A) or rear (200B) of the electronic device (200). The electronic device (200) may include at least one of various other sensor modules, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0078] According to various embodiments, the electronic device (200) may be implemented in a structure in which the screen is extended toward the third edge portion (220b) when the sliding plate (220) slides out, not limited to the embodiments of FIG. 2a, 2b, 3a, and 3c. For example, a portion of the flexible display (230) forming the first curved portion (230b) in the closed state of FIG. 2a may be included in the extended flat portion (230a) when transitioning from the closed state of FIG. 2a to the open state of FIG. 3a, and may be formed as another portion of the flexible display (230).

[0080] FIG. 4 illustrates the sequential recognition of handwriting as the display of an electronic device expands according to various embodiments.

[0081] According to various embodiments, the electronic device (400) can recognize touch input of a handwriting means (e.g., a stylus) on a display. The electronic device (400) can detect a point of contact between the handwriting means and the display through a sensor device such as a pressure sensor or a touch sensor. For example, the electronic device (400) can detect a change in pressure or a flow of current occurring on the display and determine that a touch input by the handwriting means has occurred at that point.

[0082] According to various embodiments, the electronic device (400) can expand one housing of the display in response to user input. According to one embodiment, the electronic device (400) may have a button member including a button projection inserted on one side of the housing to expand the display. The user may operate the button member to instruct the expansion of the display. The electronic device (400) may detect the user's operation of the button and control a drive motor to expand the display.

[0083] According to various embodiments, the electronic device (400) can perform stroke analysis by recognizing content displayed on a display. The electronic device (400) can recognize the shape of text written by a person or printed by a machine using optical character recognition (OCR). According to one embodiment, the electronic device (400) can acquire a sample for analyzing a typeface and perform stroke analysis based thereon. Alternatively, the electronic device (400) can generate stroke analysis rules through learning a large amount of data and perform stroke analysis based thereon.

[0084] For example, the electronic device (400) can recognize handwriting through the processes of preprocessing, classification, and postprocessing. In the preprocessing stage, the electronic device (400) can perform tasks such as increasing the contrast of each pixel of the display, removing stains, and removing lines to distinguish text areas. Subsequently, the electronic device (400) can find the most similar handwriting pattern by comparing it with a pre-built handwriting model and generate result candidates using an algorithm (e.g., neural network, HMM (hidden Markov model), SVM (support vector machine)). In the postprocessing stage, a language model can be constructed by modeling the linguistic information of the target language and correcting errors found in the handwriting recognition results to construct natural words or sentences. The electronic device (400) can produce text by analyzing the hook, size, speed, and skew, slant of the stroke. The electronic device (400) can convert the analyzed content into text and display it on one side of the display.

[0085] According to various embodiments, the electronic device (400) can sequentially expand the analysis area of ​​the content over time. The electronic device (400) can analyze strokes in a wider area of ​​the content over time. For example, the electronic device (400) can analyze a part of the content at a first point in time to generate a first text (412) and display the generated first text (412) on one side of the content. As the display expands, the electronic device (400) can expand the analysis area of ​​the content and display a second text (422) generated at a second point in time on one side of the content. To do this, the electronic device (400) determines a reference time and, whenever the reference time elapses, determines an analysis area and can convert strokes within that area into text. A specific method for analyzing strokes will be described below through FIGS. 5 to 13.

[0087] FIG. 5 is a block diagram of an electronic device according to various embodiments.

[0088] Referring to FIG. 5, the electronic device (500) may include a display (520), a drive motor (530), a processor (510), and a memory (540), and in various embodiments, some of the illustrated components may be omitted or substituted. The electronic device (500) may further include at least some of the components and / or functions of the electronic device (101) of FIG. 1. At least some of each component of the illustrated (or unillustrated) electronic device (500) may be operatively, functionally, and / or electrically connected.

[0089] According to various embodiments, the display (520) can display various images under the control of the processor (510). The display (520) may be implemented as any one of a liquid crystal display (LCD), a light-emitting diode (LED) display, a micro LED display, a quantum dot (QD) display, or an organic light-emitting diode (OLED) display, but is not limited thereto. The display (520) may be formed as a touch screen that detects touch and / or proximity touch (or hovering) input using a part of the user's body (e.g., finger) or an input device (e.g., stylus pen). The display (520) may include at least some of the configurations and / or functions of the display module (160) of FIG. 1.

[0090] According to various embodiments, the display (520) may be at least partially flexible and may be implemented as a foldable display or a rollable display.

[0091] According to various embodiments, the driving motor (530) may include, for example, a DDI (display drive integrated circuit) or a DDI chip as a circuit for controlling the display (520). According to one embodiment, the driving motor (530) may include a TDDI (touch display driver IC) arranged in a COP (chip on panel) or COF (chip on film) manner.

[0092] According to various embodiments, the drive motor (530) may be a DDI package. The DDI package may include a DDI (or DDI chip), a timing controller (T-CON), a graphics RAM (GRAM), or power generating circuits.

[0093] According to various embodiments, the memory (540) may include a volatile memory (e.g., the volatile memory (132) of FIG. 1) and a non-volatile memory (e.g., the non-volatile memory (134) of FIG. 1) to store various data temporarily or permanently. The memory (540) may include at least some of the configuration and / or functions of the memory (130) of FIG. 1 and may store the program (140) of FIG. 1.

[0094] According to various embodiments, the memory (540) may store various instructions that can be executed by the processor (510). Such instructions may include control commands such as arithmetic and logical operations, data movement, and input / output that can be recognized by the processor (510).

[0095] According to various embodiments, the processor (510) may be configured to be operatively, functionally, and / or electrically connected to each component of the electronic device (500) (e.g., display (520), drive motor (530), memory (540)) and capable of performing operations or data processing regarding the control and / or communication of each component. The processor (510) may include at least some of the configuration and / or functions of the processor (120) of FIG. 1.

[0096] According to various embodiments, there may be no limitations on the computational and data processing functions that the processor (510) can implement on the electronic device (500), but various embodiments for recognizing handwriting in an electronic device including a rollable display will be described below. The operations of the processor (510) described below can be performed by loading instructions stored in memory (540).

[0097] According to various embodiments, the processor (510) can display content on the display (520). The processor (510) can display a graphic UI of various applications provided by the electronic device (500), and can display content such as text, pictures, and images. According to one embodiment, the processor (510) can display a stroke by recognizing a point of contact between the display (520) and a writing means (e.g., a stylus). For example, when a user writes using a writing means, the processor (510) can recognize a point of contact between the writing means and the display (520) and display a stroke consisting of at least one dot or line on the display (520). The stroke may be expressed in the form of a line connecting the point of contact between the writing means and the display (520) or a plurality of contact points.

[0098] According to various embodiments, the processor (510) can expand the display (520) in response to a request to expand the display (520). According to one embodiment, the processor (510) can expand the display (520) by controlling a drive motor (530). According to one embodiment, the electronic device (500) may have a component (e.g., a button) for requesting the expansion of the display (520) on one side of the display (520) housing. Alternatively, a graphic UI (e.g., an icon) that can request the expansion of the display (520) may be displayed on the display (520). The processor (510) can expand the display (520) by controlling the drive motor (530) in response to user input regarding the display (520) expansion button or the graphic UI indicating it. For example, the display (520) may be expanded horizontally or vertically depending on the arrangement of the housing.

[0099] According to various embodiments, the processor (510) can divide the strokes constituting the content into at least one word area. According to one embodiment, the processor (510) can determine different areas if the spacing between strokes is greater than or equal to a predetermined length. For example, if the content displayed on the display (520) consists of at least one word, the processor (510) can determine each word as a single word area based on the spacing. The processor (510) can determine the strokes within each word area as a single word.

[0100] According to various embodiments, the processor (510) can perform line analysis of the content. A line is a unit in which the processor (510) performs content analysis, and the processor (510) can recognize text by analyzing at least one line synchronously or asynchronously. A line may include at least one word area. That is, the processor (510) can generate at least one line by mapping at least one word area. The processor (510) can set the line direction differently depending on the arrangement of the word areas. For example, if the word areas are arranged left and right, the line can be set in a horizontal direction, and if the word areas are arranged up and down, the line can be set in a vertical direction.

[0101] According to various embodiments, the processor (510) may generate a line list. The line list may include information about at least one line generated by the processor (510). That is, the processor (510) may generate a line list by mapping information about each line and the word area included in the line.

[0102] According to various embodiments, the processor (510) may determine a reference time (width). The reference time may be a unit of time for the processor (510) to perform content analysis. The processor (510) may obtain information regarding the CPU performance of the electronic device (500) and set the reference time based thereon. For example, if the CPU performance is greater than a set value, the reference time may be set long because a large amount of strokes can be analyzed at once. Conversely, if the CPU performance is less than a set value, the reference time may be set short because it is burdensome to process a large amount of strokes at once. According to one embodiment, the processor (510) may set the reference time based on the time it takes for the display (520) to expand to its maximum. For example, if it takes a first time (e.g., 2 seconds) for the display (520) to expand to its maximum, the processor (510) may determine the number of stroke analyses (e.g., 4 times) while the display (520) expands. The processor (510) can determine a reference time (e.g., 0.5 seconds) according to the number of analyses determined. Below, the time elapsed by the reference time from the first point in time is described as the second point in time.

[0103] According to various embodiments, the processor (510) may determine an area for performing content analysis based on a set reference time. The processor (510) may determine an area for analyzing content based on the length to which the display (520) has expanded during the reference time. The processor (510) may analyze a wider area of ​​content as time progresses. For example, if the display (520) has expanded by a first ratio of the total expanded length during the reference time, the processor (510) may determine an area occupying that first ratio of the content as an area for analysis. In the following description, the area for analysis is described as including a first area for analyzing content at a first time point and a second area for analyzing content at a second time point, but is not limited to the described examples.

[0104] According to various embodiments, the processor (510) may acquire at least one word area on a line-by-line basis that does not extend beyond the analysis area. The processor (510) may not acquire word areas that extend a certain portion both inside and outside the analysis area. For example, at the first time point, the content may include a first word area that is fully contained within the first area, a second word area that extends both inside and outside the first area, and a third word area located outside the first area. In this case, the processor (510) may analyze only the first section that is fully contained within the first area at the first time point, and may not analyze the second word area and the third word area.

[0105] According to various embodiments, the processor (510) can generate text by analyzing strokes included in the acquired word area. The processor (510) can generate text based on a handwriting recognition algorithm (e.g., linear matching, elasting matching, neural network, hidden Markov model). The processor (510) can construct a model library using a handwriting model and generate text by comparing the input handwriting with the stored model.

[0106] According to various embodiments, the processor (510) may display text generated on one side of the content. For example, the processor (510) may display a first text generated based on a word area included in a first area at a first time point on one side of the content, and may display a second text generated based on a word area included in a second area at a second time point on one side of the content.

[0107] According to one embodiment, the processor (510) stores text information about an area analyzed once in memory (540), and when performing content analysis on the same area thereafter, it can quickly display text by referring to the information stored in memory (540). The processor (510) can expand the analysis area at standard time intervals until the analysis of all strokes of the content is completed and convert the handwriting of the content into text.

[0109] FIGS. 6a and 6b are drawings illustrating an electronic device according to various embodiments analyzing a stroke to generate a line list.

[0110] Referring to FIG. 6a, the processor can generate a word area (620) containing at least a portion of a stroke in the content (610). According to various embodiments, the processor can set criteria for generating the word area (620). For example, the processor can set one stroke to be included in one word area (620), and strokes that are separated from neighboring strokes by a set distance (e.g., 3 mm) or more to be included in different word areas (620). The processor can scan the entire content (610) to generate at least one word area (620) containing at least one stroke.

[0111] According to one embodiment, the processor may build a model library to recognize words and generate a word region (620) containing the recognized words. The processor may analyze the entire content (610) and, if the content (610) consists of at least one word, generate at least one word region (620) containing each word.

[0112] Referring to FIG. 6b, the processor can generate a line (630) containing at least one word region (620). The line (630) may be in the shape of a virtual rectangle having a fixed size. The processor can generate the line (630) based on an arrangement of word regions (620). For example, as shown in FIG. 6b, if the word regions (620) are arranged horizontally relative to each other, the line (630) may also be in the shape of a horizontally elongated rectangle.

[0113] According to various embodiments, the processor may generate a line list based on information about the generated line (630). The line list may include information about the word area (620) included in each line (630). According to one embodiment, the processor may store information about the word area (620) belonging to the area to be analyzed at each point in time in the line list, based on the size of the analysis area according to a reference time.

[0114] For example, in content such as Fig. 6b, the processor may determine "Modi, who has ruled India since" as the first line, "his young age" as the second line, and "has maintained his popularity" as the third line. The processor may generate a list of lines by mapping the words included in each line (630).

[0116] FIG. 7 illustrates an analysis area that varies according to the reference time of an electronic device according to various embodiments.

[0117] Referring to FIG. 7, the processor may determine a reference time differently based on the CPU performance of the electronic device (e.g., the electronic device (400) of FIG. 4 or the electronic device (500) of FIG. 5) on the same content (e.g., the content (610) of FIG. 6) or the amount of strokes included in the content. According to one embodiment, the processor may determine a reference time based on CPU performance. For example, if the CPU performance is low, the amount of strokes that can be analyzed at once is small, so the reference time may be determined to be short (702). If the CPU performance is high, a large amount of strokes can be analyzed at once, so the reference time may be determined to be long (704).

[0118] According to one embodiment, the processor can determine a reference time based on the amount of strokes included in the content. If there are many strokes included in the content, the processor can set a short reference time to analyze the strokes in multiple steps (702). Conversely, if there are few strokes included in the content, the processor can set a long reference time to analyze the strokes (704).

[0120] FIG. 8 illustrates an electronic device according to various embodiments recognizing word regions of content by reference time.

[0121] Referring to FIG. 8, the processor can analyze strokes contained in areas of different sizes over time within the same content (e.g., content (610) of FIG. 6). FIG. 8 illustrates an embodiment in which the processor analyzes all strokes of the content four times. The processor can determine a reference time and analyze strokes by generating a word area (e.g., word area (620) of FIG. 6) and a line list.

[0122] According to various embodiments, at a first time point, the processor may determine a first region (802) to be analyzed based on a reference time point. For example, the processor may select a word region that does not go beyond the first region (802) to generate a first text (800) including "Modi" in the first line, "his" in the second line, "has" in the third line, "to" in the fifth line, "in the" in the seventh line, "and" in the eighth line, and "India's" in the ninth line. At a second point in time after the reference time has elapsed, the processor determines a second area (812) and can generate a second text including "Modi, who has" in the first line, "his young age" in the second line, "has maintained" in the third line, "despite his" in the fourth line, "to grow India's" in the fifth line, "creating" in the sixth line, "in the" in the seventh line, "and providing" in the eighth line, and "India's poorest" in the ninth line (810). At the third point in time, the processor determines the third area (822) and can generate a third text including "Modi, who has ruled" in the first line, "his young age" in the second line, "has maintained his" in the third line, "despite his continued" in the fourth line, "to grow India's" in the fifth line, "creating substantial" in the sixth line, "in the distribution of" in the seventh line, "and providing healthcare" in the eighth line, and "India's poorest citizens" in the ninth line (820).At the fourth point in time, the processor determines the fourth area (832) and can generate a fourth text including "Modi, who has ruled India since" in the first line, "his young age" in the second line, "has maintained his popularity" in the third line, "despite his continued efforts" in the fourth line, "to grow India's economy" in the fifth line, "creating substantial advance" in the sixth line, "in the distribution of wealth" in the seventh line, "and providing healthcare to" in the eighth line, and "India's poorest citizens" in the ninth line (830).

[0123] According to various embodiments, the processor stores information about text generated at a first time point, a second time point, and a third time point, and can utilize said information at a subsequent time point. For example, the text "Modi" generated on the first line at the first time point can be utilized as is when generating text for the first line at the second time point to reduce time.

[0124] According to various embodiments, the extension of the display (e.g., the display (520) of FIG. 5) may be terminated at a fourth time point. The processor may set a reference time so that all stroke analysis of the content is completed at the time when the extension of the display is terminated.

[0126] FIG. 9 illustrates an electronic device according to various embodiments recognizing word regions over time using a separate thread for each line.

[0127] Referring to FIG. 9, the processor can expand the size of the analysis area over time. The processor can determine the size of the area to be analyzed based on a predetermined reference time. The processor can text out strokes located within a word area (e.g., the word area (620) of FIG. 6) that does not go outside the analysis area. The processor can analyze strokes located in the first area at a first time point and analyze strokes located in the second area at a second time point that has elapsed by a reference time from the first time point.

[0128] For example, in FIG. 9, the processor may analyze the first area (910) at the first time point and analyze the second area (920) at the second time point. At the first time point, the processor may generate a first text by analyzing "Modi", "his", "has", "despite", "to grow", "in the", "and", and "India's" located within the first area (910). The processor may not perform analysis on word areas that are outside the first area (910). For example, word areas containing "who" in the first line (900a) and "young" in the second line are partially included in the first area (910), but since the remaining area is outside the first area (910), analysis may not be performed at the first time point.

[0129] In the same way, at a second time point, the processor can generate a second text by analyzing "Modi, who has," "his young age," "has maintained," "despite his," "to grow India's," "creating," "in the," "and providing," and "India's poorest" located within the second area (920). According to various embodiments, the processor can display the text analyzed as above by dividing it into lines (e.g., lines (630) of FIG. 6). That is, the processor can display the text of the strokes of the first line (900b) and the text of the strokes of the second line on different lines.

[0130] According to various embodiments, the processor may refer to a line list when performing text analysis. Since the line list contains information about word regions located in each line, the processor may obtain information about words located in each line by referring to the line list.

[0131] According to various embodiments, when a processor analyzes content (e.g., content (610) of FIG. 6), it may utilize information about previously analyzed content. For example, when the processor analyzes a second area (920), it may utilize information about the first text analyzed in the first area (910) as is. To do this, the processor may store information about the first text generated at a first time point in memory (e.g., memory (540) of FIG. 5) and utilize it at a second time point.

[0133] FIG. 10 is a diagram showing that stroke recognition is delayed due to performance delay of an electronic device according to various embodiments.

[0134] According to various embodiments, the processor may perform content analysis (e.g., content (610) of FIG. 6) for each line (e.g., line (630) of FIG. 6) synchronously or asynchronously. If content analysis is performed synchronously, stroke analysis for all lines may be executed in a single thread, and the analysis results for each line may be output at once. On the other hand, if content analysis is performed asynchronously, stroke analysis for all lines may be executed in separate threads, and the analysis results for each line may be output independently.

[0135] According to various embodiments, when a processor performs content analysis synchronously or asynchronously, stroke recognition may be delayed due to performance delay of an electronic device (e.g., the electronic device (400) of FIG. 4 or the electronic device (500) of FIG. 5). FIG. 10 illustrates a case where performance delay occurs when the electronic device performs stroke recognition for a first area (1010) asynchronously at a first time point. At the first time point, analysis of lines 1 through 4 (1000a) proceeds normally, but analysis of lines 5 through 9 (1000b) is delayed due to performance delay. In this case, the processor may display the analysis results for lines 1 through 4 (1000a) on one side of the content, but may not display the analysis results for lines 5 through 9 (1000b).

[0136] According to another embodiment, if the analysis of lines 5 through 9 (1000b) is delayed, the processor may not display any text. If the analysis proceeds asynchronously, the analysis of the entire content may be delayed even if the analysis of any one line is delayed. This recognition delay can be resolved if the analysis of the second area (1020) proceeds normally at a second point in time.

[0138] FIGS. 11a and FIGS. 11b illustrate embodiments in which the display of an electronic device according to various embodiments expands vertically.

[0139] According to various embodiments, the display (e.g., the display (520) of FIG. 5) of an electronic device (e.g., the electronic device (400) of FIG. 4 or the electronic device (500) of FIG. 5) may be extended vertically. The processor may analyze the content (e.g., the content (610) of FIG. 6) as described in the embodiment in FIG. 6 through 10 where the display is extended horizontally. Content identical to that described in FIG. 6 through 10 will be omitted from the description.

[0140] According to various embodiments, the processor may expand the analysis area in the direction in which the display expands. Just as the processor expanded the analysis area from left to right (or from right to left) in an embodiment where the display expands left and right, the processor may expand the analysis area from top to bottom (or bottom to top) based on a reference time. For example, the first area (1130) analyzed at a first time point may be located at the top of the content. The processor may display the first text (1112) generated at the first time point on one side of the content. Subsequently, as time elapses, the analysis area may expand downward. The processor may display the second text (1122) generated at a second time point on one side of the content.

[0141] FIG. 11b illustrates a specific embodiment of the analysis area of ​​the content. For example, a first area in which the processor analyzes the content at a first time point may include a stroke that can be analyzed as "Modi, who has ruled India since his young age" in the first and second lines. The processor may analyze the stroke included in the first area and display it at the top of the content. A second area (1140) in which the processor analyzes the content at a second time point after a reference time has elapsed may include a stroke that can be analyzed as "Modi, who has ruled India since his young age, has maintained his popularity despite his continued efforts" in the first to fourth lines. The processor may analyze the stroke included in the second area and display it at the top of the content.

[0143] FIG. 12 illustrates an embodiment in which the display of an electronic device according to various embodiments displays text without expanding.

[0144] The processor can generate text by analyzing content (e.g., content (610) in FIG. 6) even if the display (e.g., display (520) in FIG. 5) (1200) is not expanded. In FIG. 4 through 11, a reference time was determined based on the time the display (1200) expands, but in the embodiment of FIG. 12, the reference time can be determined without considering the expansion time of the display (1200). The processor can determine the reference time based on the CPU performance of the electronic device and the amount of strokes included in the content.

[0145] The operation of the processor analyzing a stroke according to a reference time and generating text is not described as it is as described in the embodiments of FIGS. 4 to 11 above.

[0146] According to various embodiments, the processor can display text by creating a separate graphic UI (1210) within a part of the content. In the previous embodiment, text was displayed in an extended area of ​​the display (1200), but in the embodiment of FIG. 12, since the display (1200) is not extended, text can be displayed by creating a separate graphic UI (1210) inside the content.

[0147] According to various embodiments, the processor may further display a button that can copy or convert text displayed in the graphic UI (1210). The processor may copy the text displayed in the graphic UI (1210) based on user input to the copy button and then paste the same text onto another screen according to a paste command. The processor may convert the currently displayed content into a generated text file based on user input to the convert button.

[0149] An electronic device according to various embodiments comprises a rollable display, a memory, and a processor operatively connected to the rollable display and the memory, wherein the processor displays content consisting of at least one stroke on the rollable display, expands the rollable display in response to a user's request to expand the display, and while performing the expansion of the rollable display, analyzes at least one stroke included in a first area of ​​the content at a first time point and displays a first text corresponding to the analyzed stroke on one side of the content, and at a second time point analyzes at least one stroke included in a second area including the first area and displays a second text corresponding to the analyzed stroke on one side of the content.

[0150] According to various embodiments, the processor may determine the second time point as the time point that has elapsed by a reference time from the first time point.

[0151] According to various embodiments, the processor may determine the reference time based on the amount of stroke included in the content.

[0152] According to various embodiments, the processor may determine the second region by extending the first region in the direction in which the rollable display extends, based on the length that the rollable display has extended during the reference time.

[0153] According to various embodiments, the rollable display can be extended left and right or up and down.

[0154] According to various embodiments, the processor can divide the strokes constituting the content into at least one word area according to a predetermined criterion.

[0155] According to various embodiments, the processor may map at least one word region included in the content to generate at least one line, and generate a line list based on information about the at least one line.

[0156] According to various embodiments, the processor can obtain at least one word region on a line-by-line basis that does not go beyond the region to be analyzed in the content, based on the line list.

[0157] According to various embodiments, the processor can display text generated based on strokes included in the acquired word area on one side of the content line by line.

[0158] According to various embodiments, the processor may perform stroke analysis for all lines of the content in a single thread, or in separate threads corresponding to each line.

[0159] According to various embodiments, the processor may store information about the first text generated at the first time point in the memory, and generate the second text at the second time point using the information about the first text.

[0161] FIG. 13 is a flowchart of an electronic device according to various embodiments.

[0162] According to various embodiments, in operation 1302, an electronic device (e.g., the electronic device (400) of FIG. 4 or the electronic device (500) of FIG. 5) may control a drive motor (e.g., the drive motor (530) of FIG. 5) to expand a display (e.g., the display (520) of FIG. 5). The electronic device may have a button member including a button projection inserted on one side of the housing to expand the display. Alternatively, a graphic UI (e.g., an icon) that can request the expansion of the display may be displayed on the display. A user may operate the button member to direct the expansion of the display. The electronic device may detect the user's operation on the button and control the drive motor to expand the display. According to one embodiment, the display may be expanded left and right (or up and down).

[0163] According to various embodiments, in operation 1304, the electronic device may analyze strokes to determine word regions (e.g., word regions (620) of FIG. 6). According to one embodiment, the electronic device may determine different regions if the interval between strokes is greater than or equal to a predetermined length. The electronic device may determine strokes within each word region as a single word.

[0164] According to various embodiments, in operation 1306, the electronic device may generate a line list. The electronic device may generate a line (e.g., line (630) of FIG. 6) by performing analysis of a line (e.g., content (610) of FIG. 6) of content. The electronic device may generate at least one line by mapping at least one word region and generate a line list based thereon. The line list may include information about at least one line generated by the electronic device.

[0165] According to various embodiments, in operation 1308, the electronic device may determine a reference time. The reference time is a unit of time in which the electronic device performs content analysis, and the reference time may be set based on information regarding the CPU performance of the electronic device. According to one embodiment, the electronic device may set the reference time based on the time it takes for the display to expand to its maximum.

[0166] According to various embodiments, in operation 1310, the electronic device can determine the analysis area of ​​the content. The electronic device can determine the analysis area of ​​the content based on the length that the display has extended during a reference time. The electronic device can analyze a wider area of ​​the content as time passes. According to one embodiment, the electronic device can extend the analysis area in the direction in which the display extends.

[0167] According to various embodiments, in operation 1312, the electronic device may determine a word area within an analysis area. The electronic device may not acquire a word area that extends a certain portion both inside and outside the analysis area. The electronic device may determine a word area included in the analysis area and generate text by analyzing a stroke included in the word area. According to one embodiment, the electronic device may store information about a stroke analyzed once in memory (e.g., memory (540) of FIG. 5) and then refer to that information to quickly generate text when texting the same stroke later.

[0168] According to various embodiments, the electronic device may perform content analysis for each line synchronously or asynchronously. When content analysis is performed synchronously, stroke analysis for all lines may be executed in a single thread, and the analysis results for each line may be output at once. On the other hand, when content analysis is performed asynchronously, stroke analysis for all lines may be executed in separate threads, and the analysis results for each line may be output independently.

[0169] According to various embodiments, in operation 1314, the electronic device may display analyzed text on one side of the content. According to one embodiment, the electronic device may display text in an area created as the display expands. The size of the area where the generated text is displayed may increase with the expansion of the display.

[0170] According to various embodiments, in operation 1320, the electronic device can analyze the content to determine whether text generation has been completed. If all strokes of the content have been analyzed, the electronic device can terminate text generation. According to one embodiment, since the electronic device determines a reference time based on the time the display is extended, it can complete the analysis of all strokes at the same time the extension of the display ends.

[0172] A handwriting recognition method using a rollable display according to various embodiments includes an operation of displaying content consisting of at least one stroke on a rollable display, and an operation of expanding the rollable display in response to a user's request for display expansion. During the expansion of the rollable display, the method may include an operation of analyzing at least one stroke included in a first area of ​​the content at a first time point, an operation of displaying a first text corresponding to the analyzed stroke on one side of the content, an operation of analyzing at least one stroke included in a second area including the first area at a second time point, and an operation of displaying a second text corresponding to the analyzed stroke on one side of the content.

[0173] According to various embodiments, the second point in time may be a point in time that has elapsed by a reference time from the first point in time.

[0174] According to various embodiments, the operation of determining the reference time based on the amount of stroke included in the content may be included.

[0175] According to various embodiments, the operation of recognizing at least one stroke included in the second region may include the operation of determining the second region by extending the first region in the direction in which the rollable display extends, based on the length in which the rollable display has extended during the reference time.

[0176] According to various embodiments, the rollable display may include an operation of expanding left and right or up and down.

[0177] According to various embodiments, the operation of analyzing the strokes of the content may include the operation of dividing the strokes constituting the content into at least one word area according to a predetermined criterion.

[0178] According to various embodiments, the operation of analyzing a stroke of the content may include mapping at least one word area included in the content to generate at least one line, and generating a line list based on information about the at least one line.

[0179] According to various embodiments, the operation of analyzing the stroke of the content may include the operation of obtaining at least one word area on a line-by-line basis that does not go beyond the area to be analyzed in the content, based on the line list.

[0180] According to various embodiments, the operation of analyzing the strokes of the content may include performing stroke analysis for all lines of the content in a single thread, or performing the analysis in a separate thread corresponding to each line.

Claims

Claim 1 An electronic device comprising: a rollable display including a first part and a second part connected to the first part and configured to be exposed externally according to an expansion operation; a memory; and a processor operatively connected to the rollable display and the memory, wherein the processor is configured to display content consisting of at least one stroke on the first part of the rollable display, expand the rollable display in response to a user's request for display expansion, and, while performing the expansion of the rollable display, analyze at least one stroke included in a first area of ​​the content at a first time point, display a first text corresponding to the stroke included in the analyzed first area on a portion of the second part exposed according to the expansion of the rollable display, analyze at least one stroke included in a second area including the first area at a second time point, and display a second text corresponding to the stroke included in the analyzed second area on the second part. Claim 2 In claim 1, the processor is an electronic device configured to determine the point in time that has elapsed by a reference time from the first point in time as the second point in time. Claim 3 In claim 2, the processor is an electronic device configured to determine the reference time based on the amount of stroke included in the content. Claim 4 In claim 2, the processor determines the second region by expanding the first region in the direction in which the rollable display expands based on the length in which the rollable display has expanded during the reference time, and the electronic device is configured to expand the rollable display left and right or up and down. Claim 5 In claim 1, the processor is an electronic device configured to divide the strokes constituting the content into at least one word area according to a predetermined standard. Claim 6 In claim 5, the processor is an electronic device configured to map at least one word area included in the content to generate at least one line, and to generate a line list based on information about the at least one line. Claim 7 In claim 6, the processor is an electronic device configured to acquire, line by line, at least one word area that does not extend beyond the area to be analyzed in the content based on the line list, and to display text generated based on strokes included in the acquired word area on one side of the content line by line. Claim 8 In claim 1, the processor is an electronic device configured to perform stroke analysis for all lines of the content in a single thread, or in a separate thread corresponding to each line. Claim 9 An electronic device configured such that, in claim 1, the processor stores information about the first text generated at the first time point in the memory, and generates the second text using the information about the first text at the second time point. Claim 10 A handwriting recognition method using a rollable display comprising a first part and a second part connected to the first part and configured to be exposed externally according to an expansion operation, the method comprising: an operation of displaying content consisting of at least one stroke on the first part of the rollable display; and an operation of expanding the rollable display in response to a user's request for display expansion, wherein, while performing the expansion of the rollable display, the method comprises: an operation of analyzing at least one stroke included in a first area of ​​the content at a first time point; an operation of displaying a first text corresponding to the analyzed stroke on a portion of the second part exposed according to the expansion of the rollable display; an operation of analyzing at least one stroke included in a second area including the first area at a second time point; and an operation of displaying a second text corresponding to the analyzed stroke on the second part. Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images