Method for converting handwritten note into character text and electronic device

Abstract

The present application relates to the technical field of terminals and provides a method for converting a handwritten note into a character text and an electronic device. The method comprises: in response to an operation of a user on a display screen, acquiring a selection range; acquiring handwriting in the selection range; recognizing a first color of the handwriting; recognizing the handwriting to obtain text content; according to the number and size of characters in the text content, determining a text format; and, in response to a confirmation operation of the user, displaying a character text on the display screen. Thus, in the process of converting a handwritten note into a character text, the present application can acquire the format and color corresponding to the handwriting, and then generates the text content according to the format and color corresponding to the handwriting, so as to reflect format features generated by the handwriting, thereby improving the efficiency of organizing the handwritten note.

Description

Method and electronic device for converting handwritten notes into text

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on December 29, 2023, with application number 202311871666.X and invention name “A method and electronic device for converting handwritten notes into text”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a method and electronic device for converting handwritten notes into text. Background Art

[0003] With the trend toward paperless office, using a stylus allows for faster recording of content on electronic devices in situations like meetings and classrooms. However, if you want to share notes taken on an electronic device with other devices or organize them into a Word document, you can only do so through illustrations or by manually entering the notes, which is very cumbersome.

[0004] To improve the efficiency of handwriting-to-text conversion, electronic devices can use their internal recognition systems to convert handwritten text into a universally readable font format that the electronic device can display on the screen. However, during the handwriting-to-text conversion process, the text is converted using a universal format, discarding the handwritten formatting, such as font size and color, which affects the generated text.

[0005] Summary of the Invention

[0006] The present application provides a method and electronic device for converting handwritten notes into text, so as to solve the problem that when converting handwritten traces, the formatting such as font size and color is not retained, thereby affecting the generation of text.

[0007] According to a first aspect of an embodiment of the present application, a method for converting handwritten notes into text is provided, which is applied to an electronic device with a display screen, and the method includes: obtaining a selection range in the handwriting area in response to a user's operation on the handwriting area; obtaining a handwriting trajectory within the selection range; identifying a first color of the handwriting trajectory; identifying the handwriting trajectory to obtain text content; determining a text format based on the number and size of characters in the text content; and displaying the text on the display screen in response to a user's confirmation operation; the text includes text content whose font format conforms to the text format and whose font color is a first color.

[0008] Based on the technical solution provided in this application, it is possible to identify information such as color and font size in the handwriting trajectory during the conversion process of the handwriting trajectory, so as to generate text corresponding to the features of the handwriting trajectory, avoid the loss of features in the handwriting trajectory, and improve the efficiency of organizing handwritten notes.

[0009] In one feasible implementation, the number of characters includes the number of characters in the text content, the number of lines of text, and the number of characters in each line of text; the character size includes the length and width of each character in the text content; and the text format includes the font size and line spacing. Thus, by obtaining and determining information such as the number of characters in the text content, formatting such as color and font size can be preserved in the handwriting trace.

[0010] In a feasible implementation, the text format is determined based on the number and size of characters in the text content, including: calculating the first font size corresponding to each line of text in the text content based on the number of lines of text content, the number of characters in each line of text, and the length of each character in each line of text; or calculating the second font size corresponding to each line of text in the text content based on the number of characters in the text content and the length and width of each character; determining the first line spacing based on the spacing between each line of text in the text content; or determining the second line spacing based on the number of lines of text content and the length of each character in each line of text. In this way, the format of the generated text can be restricted by factors such as font size and number, so that the format of the generated text is similar to the format of the handwriting trajectory, thereby improving the efficiency of organizing handwritten notes and reducing the loss of handwriting features during the handwritten note conversion process.

[0011] In a feasible embodiment, after determining the text format based on the number and size of characters in the text content, the method further includes: displaying a first interface on a display screen; the first interface includes a text display area and an interface control, the text display area displays the text content, and the interface control displays the text format. In response to a user's confirmation operation, the text is displayed on the display screen, including: in response to the user's confirmation operation on the first interface, displaying a text box on the display screen; the text box includes text, the size of the text box is the same as the selected range, and the format of the text is the same as the text format displayed in the interface control. In this way, the user can also preview and confirm the recognition results during the recognition process, thereby improving the user experience.

[0012] In a feasible implementation, after the handwriting trace is recognized to obtain the text content, it also includes: determining the character type of each word in the text content according to the text content; the character types include Chinese characters, letters, numbers and symbols. After calculating the font size corresponding to the text content, it also includes: if the character type of the first word is a letter, number or symbol, then the font size of the second word adjacent to the first word is determined to be the font size of the first word; the first word is any word in the text content, and the second word includes words in the text content whose character type is Chinese characters. In this way, by judging the character type of the word, a more appropriate font size can be determined for words that are not Chinese characters, avoiding the problem that letters, numbers, symbols and other words are affected by the size of the words themselves during the writing process, resulting in the font size being too small.

[0013] In one feasible embodiment, in response to a user's confirmation operation, displaying the text on the display screen further includes: modifying the text content in the text display area in response to a first modification instruction from the user on the first interface; and / or modifying the text format in the interface control in response to a second modification instruction from the user on the first interface. In this way, the user can modify the recognition results during the recognition process, making the subsequently generated text more in line with the user's needs and reducing the time required for text editing.

[0014] In one feasible embodiment, if the first color includes at least two colors, after recognizing the handwriting trace to obtain the text content, the method further includes: obtaining the first color corresponding to each character in the text content based on the correspondence between the first color and the handwriting trace. In this way, when the selection range includes multiple colors, the color of each character in the text can be determined based on the colors of different traces, thereby preserving the color characteristics of the handwriting trace.

[0015] In one feasible implementation, recognizing a handwriting trajectory to obtain text content includes: determining the trajectory coordinates corresponding to the handwriting trajectory within a selected range; segmenting the handwriting trajectory into individual characters to obtain individual character text; decomposing the individual character text into strokes based on the trajectory coordinates; and performing stroke recognition on the decomposed strokes to obtain text content. In this way, the handwriting trajectory can be recognized to obtain the text content corresponding to the handwriting trajectory.

[0016] In one feasible embodiment, stroke recognition is performed on the disassembled stroke points to obtain text content, including: determining a feature vector of the handwriting trajectory in each single-word text based on the stroke points and the trajectory coordinates corresponding to the stroke points; obtaining trajectory features based on the feature vectors; and determining the text content corresponding to each single-word text based on the trajectory features of each single-word text. In this way, the single-word text can be recognized by the stroke points, thereby obtaining the text corresponding to each single-word text, and then the text content corresponding to the handwriting trajectory can be obtained based on the recognition results of all the single-word texts.

[0017] In one feasible embodiment, an electronic device includes a note display module and a handwriting recognition module, and a method includes: the handwriting recognition module obtains a selection range within the display range of the note display module in response to a user's operation on a handwriting area; the handwriting recognition module reads the handwriting trace within the selection range and identifies a first color of the handwriting trace; the handwriting recognition module recognizes the handwriting trace to obtain text content, and determines the text format based on the number of characters and the size of the characters in the text content; the handwriting recognition module sends the text content and the text format to the note display module; the note display module obtains a user confirmation operation, and the handwriting recognition module sends the text content in the text format and the first color to the note display module; and the note display module displays the text on the display screen. In this way, the interaction between the note display module and the handwriting recognition module can be used to identify information such as color and font size in the handwriting trace, thereby generating a correspondence between the text and the handwriting trace features, thereby improving the efficiency of organizing handwritten notes.

[0018] According to the second aspect of the embodiments of the present application, an electronic device is provided, comprising: a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; wherein the memory stores computer program code, the computer program code comprises computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method for converting handwritten notes into text as in any of the previous embodiments.

[0019] It can be understood that the beneficial effects that can be achieved by the technical solution provided in the second aspect mentioned above can refer to the beneficial effects in the first aspect and any feasible implementation scheme thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG1 is a schematic diagram showing the effect of converting handwritten content into text;

[0021] FIG2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0022] FIG3 is a schematic diagram of a layered architecture of an electronic device software system according to an embodiment of the present application;

[0023] FIG4 is a schematic diagram of an interaction between an electronic device software system according to an embodiment of the present application;

[0024] FIG5 is a flow chart of a method for converting handwritten notes into text in an embodiment of the present application;

[0025] FIG6 is a schematic diagram of a response to a user operating a display screen in an embodiment of the present application;

[0026] FIG7 is a schematic diagram of a process of handwriting trajectory recognition in an embodiment of the present application;

[0027] FIG8 is a schematic diagram of a single word segmentation in an embodiment of the present application;

[0028] FIG9 is a schematic diagram of another type of word segmentation in an embodiment of the present application;

[0029] FIG10 is a schematic diagram of a process for displaying text in an embodiment of the present application;

[0030] FIG11 is a schematic diagram showing a display of a handwritten note converted into text in an embodiment of the present application;

[0031] FIG12 is a schematic diagram of operating the first interface in an embodiment of the present application;

[0032] FIG13 is a schematic diagram of a system for converting handwritten notes into text according to an embodiment of the present application;

[0033] FIG14 is a schematic diagram of a process of executing a conversion method by a conversion system in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will clearly describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, other embodiments obtained by ordinary technicians in this field without making any creative work are all within the scope of protection of this application.

[0035] Hereinafter, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified with "first," "second," etc., may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0036] In addition, in this application, directional terms such as "upper" and "lower" are defined relative to the orientation of the components in the drawings. It should be understood that these directional terms are relative concepts. They are used for relative description and clarification, and they can change accordingly according to changes in the orientation of the components in the drawings.

[0037] With the trend toward paperless office work, using electronic devices to record content has become a common method for meeting minutes. In scenarios like meetings and classrooms where written notes are required, using a stylus allows for faster recording of content on electronic devices. However, if you want to share notes recorded on an electronic device with other devices or organize them into a Word document, you can only do so through illustrations or by manually entering the recorded content. This can be time-consuming and cumbersome.

[0038] In order to improve the efficiency of converting handwriting to text, electronic devices can set up a handwriting recognition system internally to convert handwritten notes into text in a universal format that can be read by electronic devices and displayed on the display screen. However, since handwritten notes may use font size, font color and other formats to reflect content characteristics during the handwriting process, such as by changing the text color to other colors to emphasize the content, or by increasing or decreasing the text size to reflect characteristics. After the handwritten notes are converted into text that can be read by electronic devices, the font size, font color and other formats in the handwritten notes will be overwritten by a universal format, so that the generated text cannot reflect the content emphasized or marked in the handwritten notes during the writing process, affecting the generation of text.

[0039] It should be understood that the general formats and other text formats in this application are specific formats of text that users can see through electronic devices, such as Songti, Heiti and other formats with limited font shapes, size 4, small 4 and other formats, and black, red and other formats with limited font colors. This application does not limit the specific types of text formats.

[0040] FIG1 is a schematic diagram showing the effect of converting handwritten content into text.

[0041] As shown in (a) of FIG1 , a user may write on the screen of an electronic device using a stylus or other device to form a handwriting track. When the handwriting track needs to be converted, the electronic device may recognize the handwriting track to obtain text corresponding to the handwriting track.

[0042] After recognition is complete, as shown in Figure 1(b), the electronic device displays the recognized text on the screen in a universal format set by the electronic device, thereby converting the handwritten trace into text that the electronic device can recognize. However, during the handwritten trace conversion process, the format of the generated text can only adopt the format pre-set in the electronic device, or after the text conversion is completed, the user must perform further operations on the generated text, which reduces the efficiency of organizing the handwritten trace, prolongs the user's organization time, and affects the user experience.

[0043] To address the above issues, the present invention provides a method for converting handwritten notes into text, which is applicable to electronic devices with display screens. This technical solution extracts the color of the handwritten trace before recognizing it, and retains the font size, spacing, and other formatting inherent in the handwritten trace during the recognition process, thereby preserving the content characteristics, improving the efficiency of organizing the handwritten trace, reducing the user's subsequent operations, and improving the user experience.

[0044] The implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0045] In the embodiments of the present application, a user can use a stylus pen to input handwriting on the touch screen of an electronic device, or can connect the electronic device to an external device such as a handwriting tablet to input handwriting. Therefore, the electronic device in the embodiments of the present application can be any electronic device with a handwriting function or capable of connecting to a handwriting peripheral device, such as a smartphone, a tablet computer, a laptop computer, a personal computer (PC), a personal digital assistant (PDA), and other electronic devices, and the embodiments of the present application do not impose any restrictions on this.

[0046] For example, taking the electronic device as a tablet computer as an example, FIG2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.

[0047] As shown in FIG2 , the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a radio frequency circuit 140, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a display 193, a subscriber identification module (SIM) card interface 194, and a camera 195. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0048] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0049] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0050] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0051] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0052] The memory 120 can be used to store computer executable program code, and the executable program code includes instructions. The memory 120 may include a program storage area and a data storage area. The program storage area may store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the memory 120 may include one or more storage units, for example, it may include volatile memory (volatile memory), such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.; it may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, etc. The processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in the memory 120 and / or instructions stored in a memory provided in the processor.

[0053] The wireless communication function of the electronic device can be implemented through the radio frequency circuit 140, the mobile communication module 150, the wireless communication module 160, the modem and the baseband processor.

[0054] RF circuit 140 may include antenna 141 and antenna 142 for transmitting and receiving electromagnetic wave signals. Each antenna in an electronic device can be used to cover a single or multiple communication frequency bands. In some embodiments, antennas can be used in conjunction with tuning switches, and different antennas can be reused to improve antenna utilization.

[0055] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to electronic devices. The mobile communication module 150 can receive electromagnetic waves through the antenna 141, and filter, amplify and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 141. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0056] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 193. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0057] The wireless communication module 160 may include a wireless fidelity (Wi-Fi) module, a Bluetooth (BT) module, a GNSS module, a near field communication (NFC) module, an infrared (IR) module, and the like. The wireless communication module 160 may be one or more devices integrating at least one of the above modules. The wireless communication module 160 receives electromagnetic waves via the antenna 141, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive signals to be transmitted from the processor 110, modulate the frequencies of the signals, amplify the signals, and convert them into electromagnetic waves for radiation via the antenna 141.

[0058] Display screen 193 is used to display images, videos, and the like. Display screen 193 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, the electronic device can include one or N display screens 193, where N is a positive integer greater than one.

[0059] A touch sensor, also known as a "touch device," can be coupled to the display screen 193, thereby forming a touch screen, also known as a "touch screen," with the touch sensor and the display screen 193. The touch sensor is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 193. In other embodiments, the touch sensor can also be disposed on the surface of the electronic device, at a location different from that of the display screen 193.

[0060] The pressure sensor is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor can also be coupled to the display screen 193. There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. When a touch operation is applied to the display screen 193, the electronic device monitors the intensity of the touch operation based on the pressure sensor. The electronic device can also calculate the position of the touch based on the monitoring signal of the pressure sensor. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0061] In the embodiment of the present application, the touch sensor and / or pressure sensor coupled to the display screen 193 can record and respond to user operations such as clicking, sliding, and writing on the display screen 193. For example, when a user writes on the display screen 193 with a finger or a stylus, the touch sensor and / or pressure sensor can record the user's writing trajectory and display the writing trajectory in real time on the display screen 193, thereby realizing the process of stroke drawing through the electronic device.

[0062] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods. At the same time, the above-mentioned Figure 2 is only an exemplary illustration when the electronic device is in the form of a tablet computer. If the electronic device is a mobile phone, a handheld computer, a PC, a PDA, a wearable device (such as a smart watch, a smart bracelet) or other device forms, the structure of the electronic device may include fewer structures than shown in Figure 2, or may include more structures than shown in Figure 2, and there is no limitation here.

[0063] It is understandable that, in general, the realization of electronic device functions requires not only hardware support but also software cooperation. The software system of the electronic device can adopt a layered architecture, event-driven architecture, micro-core architecture, micro-service architecture, or cloud architecture. Taking the system as an example, the software structure of the electronic device is illustrated.

[0064] Figure 3 is a schematic diagram of the layered architecture of an electronic device software system in an embodiment of the present application. The layered architecture divides the software into several layers, each with clear roles and divisions of labor. The layers communicate with each other through software interfaces (e.g., APIs).

[0065] In some examples, as shown in FIG3 , in an embodiment of the present application, the software of an electronic device is divided into five layers, namely, from top to bottom, the application layer, the framework layer (or application framework layer), the system library and Android runtime (Android runtime), the HAL layer (hardware abstraction layer), and the driver layer (or kernel layer). Among them, the system library and Android runtime can also be called the local framework layer or native layer.

[0066] The application layer may include a series of applications. As shown in FIG3 , the application layer may include applications (APPs) such as camera, gallery, calendar, map, WLAN, Bluetooth, music, video, short message, call, navigation, and instant messaging.

[0067] The framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions or services. For example, the application framework layer may include an activity manager, a window manager, a content provider, an audio service, a view system, a telephony manager, a resource manager, a notification manager, a package manager, etc., but the embodiments of this application do not impose any restrictions on this.

[0068] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0069] Content providers are used to store and retrieve data and make it accessible to applications. This data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0070] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0071] The phone manager is used to provide communication functions for electronic devices. For example, the phone manager can manage the call status of the call application (including initiation, connection, and hang up).

[0072] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0073] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0074] Package Manager in The system is used to manage application packages. It allows applications to obtain detailed information about installed applications and their services, permissions, etc. The package manager is also used to manage events such as application installation, uninstallation, and upgrades.

[0075] The system library can include multiple functional modules. For example: surface manager, media library, OpenGL ES, SGL, etc. The surface manager is used to manage the display subsystem and provides 2D and 3D layer fusion for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGL ES is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing. SGL is the drawing engine for 2D drawing.

[0076] The Android runtime consists of core libraries and the ART virtual machine. The Android runtime is responsible for scheduling and management of the Android system. The core libraries consist of two parts: one for Java-based functions and the other for the Android core library. The application layer and application framework layer run in the ART virtual machine. The ART virtual machine executes Java files from the application layer and application framework layer as binary files. The ART virtual machine is responsible for performing functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0077] The HAL layer is an interface layer located between the operating system kernel and the hardware circuit. Its purpose is to abstract the hardware. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, making it hardware-independent and portable across multiple platforms. The HAL layer provides a standard interface to display device hardware capabilities to the higher-level Java API framework (i.e., the framework layer). The HAL layer contains multiple library modules, each of which implements an interface for a specific type of hardware component, such as the audio HAL audio module, the bluetooth HAL Bluetooth module, the camera HAL camera module (also known as the camera HAL or camera hardware abstraction module), and the sensors HAL sensor module (or sensor service).

[0078] The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, a sensor driver, a battery driver, and the like, though this application does not limit these functions. Specifically, the sensor driver may include drivers for each sensor included in the electronic device, such as a pressure sensor driver. For example, the pressure sensor driver may respond to an instruction or command from the sensor module 180 to acquire detection data and promptly transmit the pressure sensor's detection data to the sensor module 180.

[0079] In an embodiment of the present application, the application layer may include a note display module (NotePad), and the framework layer may also include a handwriting recognition module (PenKit).

[0080] FIG4 is a schematic diagram of an interaction of an electronic device software system in an embodiment of the present application.

[0081] As shown in FIG4 , after receiving a user operation through the note display module, the handwriting recognition module can obtain each handwriting track within the selected range displayed in the note display module.

[0082] Specifically, the handwriting recognition module can obtain the drawn handwriting trajectory through the resource manager during the drawing process and read the handwriting trajectory within the selected range after receiving the user's operation on the display screen 193. In some embodiments of the present application, the handwriting trajectory drawn by the note display module can be stored in the content provider. Then, after receiving the user's operation, the handwriting recognition module can obtain the handwriting trajectory within the selected range from the content provider based on the user's operation and perform subsequent operations on the handwriting trajectory.

[0083] After reading the handwriting trace within the selected range, the handwriting recognition module can perform color recognition on the drawn handwriting trace. After the recognition is completed, the handwriting trace can be identified to obtain the text content corresponding to the handwriting trace, and the text format can be determined based on the number and size of characters in the obtained text content.

[0084] After obtaining the text content and text format corresponding to the handwriting trace, the handwriting recognition module can send the obtained text content and text format to the note display module, which then displays it through the note display module's recognition result display window. When the note display module receives the user's confirmation operation through the recognition result display window, the handwriting recognition module generates a text box based on the text content, text format, and recognized color, and sends it to the note display module, which then displays the text box on display screen 193.

[0085] In some embodiments, the handwriting recognition module may also directly perform a reading operation on all drawn handwriting trajectories after a preset time after the handwriting trajectories are drawn, so as to perform subsequent operations.

[0086] Based on the structure of the electronic device shown in FIG. 2 and the software architecture shown in FIG. 3 , the method of converting handwritten notes into text in this application is exemplified below in combination with the content in FIG. 5 .

[0087] FIG5 is a flow chart of a method for converting handwritten notes into text in an embodiment of the present application.

[0088] In an embodiment of the present application, a user can use an electronic device to recognize a handwritten trace, thereby obtaining information such as its color and font size, and generating text that matches the handwritten trace. As shown in FIG5 , the method includes:

[0089] S100: In response to a user operation on a handwriting area, a selection range is obtained in the handwriting area.

[0090] The electronic device can display the handwriting trajectory through the display screen 193, where the handwriting trajectory can be the trajectory formed by the user writing on the display screen 193, or it can be sent by other electronic devices to an electronic device for trajectory recognition and conversion. In the embodiment of the present application, there is no restriction on the source of the handwriting trajectory.

[0091] Taking the process of a user writing a track and subsequently converting it into text on the same electronic device as an example, when the user actually writes on the display screen 193, he will use an external handwriting device to write on the display screen 193, thereby improving the writing speed. In addition, connecting an external handwriting device to the electronic device can improve the clarity of the handwriting track and improve recognition efficiency.

[0092] In the embodiment of the present application, the handwriting device can connect to the electronic device by establishing a wireless connection with the wireless communication module 160. For example, the handwriting device can be provided with a Bluetooth module, which cooperates with the Bluetooth function of the wireless communication module 160 to achieve the connection between the handwriting device and the electronic device. The handwriting device can also connect to the electronic device via the universal serial bus interface 130. This application does not limit the connection method between the handwriting device and the electronic device.

[0093] It should be understood that in order to capture and display handwriting traces, the user can write in a designated area of ​​the display screen 193. For example, the designated area can be the handwriting area of ​​the display screen 193, and the display area of ​​the handwriting area is smaller than the area of ​​the display screen 193, thereby reserving space for controls in the electronic device and thus achieving control of the handwriting trace.

[0094] FIG6 is a schematic diagram of a response to a user operating a display screen in an embodiment of the present application.

[0095] As shown in Figure 6, operation controls can be displayed above the display screen 193. The operation controls can be used to wake up functions such as drawing, erasing, color modification, lasso selection, and insertion. By clicking on the operation controls, the user can wake up the corresponding functions of the controls, and then the user can implement the above functions through operations.

[0096] Taking lasso selection as an example, the user can activate the function by clicking the corresponding control of lasso selection, and then slide from one point to another in the handwriting area to determine the rectangular box with the two points as diagonals as the selection box 1931.

[0097] It should be noted that the above-mentioned lasso selection function is only a feasible implementation method in the embodiment of the present application. The controls in the embodiment of the present application may also include other functions. The functions that can be awakened by the operating controls in the present application are not limited in the present application.

[0098] S200: Obtaining the handwriting trajectory within the selected range.

[0099] Before converting handwritten notes, the user can operate the display screen 193 to generate an area where the handwritten notes need to be converted. As shown in Figure 6, the user can slide on the display screen 193 to generate a selection box 1931 on the display screen 193. In this embodiment of the present application, the electronic device can generate a rectangular selection box 1931 based on the starting and ending points of the user's slide, so as to read the handwritten trace within the selection box 1931.

[0100] It should be understood that the rectangular selection box 1931 in the embodiment of the present application is only an exemplary description. In the actual selection process, selection boxes 1931 of other shapes can also be obtained through other methods. The present application does not limit the specific shape of the selection box 1931.

[0101] As shown in FIG6 , the range included in the selection box 1931 is the selection range for handwriting trace reading in this application. Therefore, when the user operates the display screen 193 and the selection box 1931 is displayed on the display screen 193, the handwriting trace within the selection range can be obtained. Specifically, all handwriting traces within the selection box 1931 can be extracted to obtain the handwriting trace within the selection range.

[0102] S300: Recognizing the first color of the handwriting trace.

[0103] Because the color of the handwriting trace may interfere with the recognized text content during the handwriting trace recognition process in some embodiments, the color of the handwriting trace is removed during the recognition process through binarization or other methods. Therefore, in the embodiments of the present application, the color of the handwriting trace needs to be extracted before the text content is recognized, so as to facilitate the subsequent color assignment of the converted text.

[0104] Before color extraction, the background can be filtered out first, thereby reducing the influence of the background color on the color of the handwritten track. In some embodiments, the colors of all handwritten tracks within the selected range can be obtained by reading the RGB (Red Green Blue) values ​​of the pixels corresponding to the handwritten track and taking the average value. It should be understood that the above method is only applicable to the case where the selected range includes handwritten tracks of one color. When the selected range includes handwritten tracks of two or more colors, the method of calculating the average value of the pixels will lead to inaccurate color reading. The difference between the RGB values ​​of different pixels can be used to obtain two or more groups of pixels. The number of pixel groups is the same as the number of color types in the handwritten track, and then the average RGB value of each group of pixels is calculated.

[0105] When there are more than one types of handwriting track colors within the selection range, that is, the first color obtained based on handwriting track recognition includes at least two colors, the first color corresponding to each character in the text content can be obtained based on the correspondence between the first color and the handwriting track.

[0106] It should be understood that color extraction through RGB values ​​is only one feasible implementation method in the embodiments of this application. Other methods of identifying colors can also be applied in this application, and this application does not limit this.

[0107] After obtaining the color of the handwriting trajectory, the handwriting trajectory can be identified through the functional module in the electronic device framework layer. For example, in the process of handwriting trajectory recognition, the shape, size and other information of the handwriting trajectory can be analyzed, and classification can be performed using models such as maximum likelihood or Bayesian. It is also possible to extract features from the handwriting trajectory and classify them using feature vectors generated by feature extraction. It is also possible to extract features and realize trajectory recognition through deep learning methods. In the embodiment of the present application, any of the above methods can be used to identify the handwriting trajectory, and the present application does not limit the specific method for identifying the handwriting trajectory.

[0108] It should be understood that the method for recognizing handwriting trajectories is not limited to the above method. Other methods for recognizing handwriting trajectories can also be used in this application. In some embodiments of this application, multiple methods can also be combined to improve the accuracy of handwriting trajectory recognition.

[0109] One of the handwriting trajectory recognition methods is described in detail below with reference to FIG. 7 .

[0110] FIG7 is a schematic diagram of a process of handwriting trajectory recognition in an embodiment of the present application.

[0111] Taking feature extraction of handwriting trajectories as an example, as shown in FIG7 , the process of recognizing handwriting trajectories may include:

[0112] S410: Determine the track coordinates corresponding to the handwritten track within the selected range.

[0113] In the embodiment of the present application, the trajectory coordinates may be the coordinates of each pixel point in the handwriting trajectory relative to a point within the selected range, thereby obtaining the positional relationship of the pixel points in each handwriting trajectory.

[0114] In some embodiments, the large number of pixels in a handwritten trajectory can affect the recognition speed of the trajectory coordinates, and thus the speed of handwritten trajectory recognition. Therefore, a sampling method can be used to obtain some pixels as sampling points and obtain the coordinates of the sampling points, thereby improving the recognition speed of the trajectory coordinates.

[0115] For example, the handwriting trajectory and background can be grayscaled and binarized first, and then the coordinates of the handwriting trajectory pixels within the N×N pixel points are obtained to determine whether they are the pixel points corresponding to the handwriting trajectory, and the average of the trajectory coordinates of the handwriting trajectory pixel points is taken as the trajectory coordinates within the range. Where N is an integer greater than 0. When N=1, no sampling is performed. When N=2, sampling is performed in the four 2×2 pixel points, and the average of the trajectory coordinates of the handwriting trajectory pixel points in the four pixel points is calculated. Therefore, the number of pixels involved in the calculation is less than or equal to N2. The same applies when N is other values.

[0116] The present application does not impose any restrictions on the sampling range, but it should be understood that when the sampling range is too large, pixels in different handwriting trajectories will be repeatedly sampled, affecting the accuracy of subsequent recognition.

[0117] In some embodiments of the present application, if the handwritten trajectory is drawn through the display screen 193 of the electronic device, the electronic device can also generate a recorded time through a pressure sensor when the user draws the handwritten trajectory, obtain and record the drawing time of each trajectory, and distinguish the handwritten trajectories, thereby facilitating the judgment of the trajectory coordinates in the intersecting handwritten trajectories, improving the sampling efficiency of the pixel points, and improving the subsequent stroke decomposition and text recognition efficiency.

[0118] S420: Segment the handwritten trace into single words to obtain single-word text.

[0119] After obtaining the coordinates of each handwritten track, we can determine whether any two tracks belong to the same character based on their positional relationship and track overlap. The positional relationship describes the distance between tracks, while the track overlap indicates the degree of overlap between the projections of different tracks in a certain direction.

[0120] It should be understood that during actual handwriting trajectory recognition, the handwriting trajectory may include trajectories corresponding to Chinese characters, letters, numbers, symbols, and other characters. Compared to Chinese characters, the structure of letters, numbers, and symbols is relatively simple. During the recognition process, the handwriting trajectory corresponding to letters, numbers, and symbols can be segmented based on the trajectory position relationship and overlap to obtain single-word text.

[0121] FIG8 is a schematic diagram of a single-word segmentation in an embodiment of the present application; FIG9 is a schematic diagram of another single-word segmentation in an embodiment of the present application.

[0122] Since a single character in a form of text such as Chinese characters has multiple strokes, there are many handwritten trajectories in a single-word text of Chinese characters. Using only the position relationship of the trajectories and the degree of overlap of the trajectories will result in problems such as missed judgments and false judgments. Therefore, in some embodiments of the present application, as shown in FIG8 , during the process of single-word segmentation of the handwritten trajectories, the handwritten trajectories can be counted in the horizontal direction to obtain a horizontal projection, thereby obtaining the upper and lower limits of each line of text, and then the handwritten trajectories can be counted in the vertical direction to obtain a vertical projection. Combined with the horizontal projection, the handwritten trajectories can be segmented into single words.

[0123] It should be understood that in the aforementioned single-word segmentation process, the handwriting trace is drawn from left to right, that is, the text corresponding to the handwriting trace is arranged horizontally, and since the text corresponding to the handwriting trace is also arranged horizontally in the handwriting trace drawn from right to left, the above-mentioned single-word segmentation method is also applicable to handwriting traces drawn from right to left.

[0124] However, when the drawing direction of the handwriting track is from bottom to top or from top to bottom, the corresponding text in the handwriting track is arranged vertically, and the text corresponding to the handwriting tracks of different columns cannot be classified into the same row. Therefore, when performing single-word segmentation, the above-mentioned single-word segmentation method will be used to segment the text. As shown in Figure 9, in some embodiments of the present application, if the corresponding text in the handwriting track is arranged vertically, the handwriting track can be counted in the vertical direction to obtain a vertical projection, thereby obtaining the left and right limits of each column of text, and then by performing horizontal statistics on each column of handwriting tracks to obtain a horizontal projection, the horizontal projection is combined with the vertical projection to segment the vertically arranged handwriting tracks into single words.

[0125] In some embodiments, since there may be connectivity between characters of Chinese characters and there may be complex internal structures in the trajectories formed by Chinese character characters, which affect subsequent trajectory recognition, before performing vertical direction statistics, the handwritten trajectory can be subjected to a closing operation to widen the gaps between the handwritten trajectories representing Chinese character characters. Among them, the closing operation includes a dilation operation and an erosion operation, and the dilation operation is performed before the erosion operation. By the closing operation, the loss of features of the handwritten trajectory during the single character segmentation process can be avoided, and the accuracy of subsequent recognition can be improved.

[0126] In the embodiments of the present application, after the single character segmentation is completed, the distance between the upper and lower boundaries of each line of the handwritten trajectory obtained by horizontal direction statistics of the handwritten trajectory can be used as the line height of each line of text, and vertical direction statistics are performed on each line of the handwritten trajectory, and the distance between the left and right boundaries of each single character text obtained is used as the character width of each single character text, which is convenient for subsequent calculation of the text font size.

[0127] S430: Disassemble the stroke points of the single character text according to the trajectory coordinates.

[0128] After obtaining the single character text, the stroke points in the single character text can be disassembled through the trajectory coordinates, so as to obtain the stroke direction, stroke length in the single character text, and the relationship between different strokes in the single character text.

[0129] Since the generation speed of the feature vector corresponding to the trajectory is related to the number of collected feature coordinates, the more the number of feature coordinates, the slower the disassembling speed, and in the same trajectory, most of the trajectory coordinates are redundant data, which affects the recognition speed of the electronic device for the handwritten trajectory.

[0130] Exemplarily, the stroke points of the single character text can be obtained correspondingly by identifying the feature points in the handwritten trajectory. For example, the starting point, ending point and inflection point of each trajectory in the single character text can be identified and used as feature points. Through the trajectory coordinates of the feature points, the disassembling of the stroke points in the single character text can be realized. During the disassembling process, the stroke points can be represented and stored, so that the stroke recognition can be performed through the disassembled stroke points to obtain the text in subsequent steps.

[0131] In the embodiments of the present application, any handwritten trajectory includes at least one starting point and one ending point, and the number and position of the inflection points are related to the stroke corresponding to the trajectory. Therefore, a handwritten trajectory can include M inflection points, where M is an integer greater than or equal to 0. For example, the horizontal stroke "一" includes 1 stroke starting point, 1 stroke ending point, and 0 stroke inflection points; the hook stroke "亅" includes 1 stroke starting point, 1 stroke ending point, and 1 stroke inflection point; the horizontal fold and hook stroke "乙" includes 1 stroke starting point, 1 stroke ending point, and 3 stroke inflection points.

[0132] It should be understood that, as shown in Figures 8 and 9, in addition to the starting point, end point and inflection point, the strokes in Chinese characters may also include intersection points. Since intersection points can represent the internal structure of single-word text, intersection points can also be used as feature points for identifying text. Therefore, when decomposing strokes, the intersection points in the handwriting trajectory can also be collected and decomposed.

[0133] Furthermore, some Chinese characters have strokes with a certain curvature but no bends. In the process of decomposing these stroke points, the trajectory coordinates of the positions with larger curvatures in the trajectory can also be used as feature points, and combined with feature points such as the starting point, end point, and inflection point, the stroke points can be decomposed.

[0134] At the same time, since the trajectory in the handwriting trajectory has a certain width and length, if the trajectory coordinates obtained in the above steps are all coordinates representing a pixel point, in some embodiments, in order to accurately decompose the stroke points, multiple trajectory coordinates can be selected as feature points at the starting point and / or end point of the trajectory, so that the strokes can be expressed more delicately through the decomposed stroke points.

[0135] It should be noted that during the stroke decomposition process, the electronic device can store the decomposed stroke positions to facilitate subsequent stroke recognition. For example, the electronic device can represent and store the decomposed stroke positions in the following format.

[0136] PointList{

[0137] x1,y2,action

[0138] x2,y2,action

[0139] ……}.

[0140] It should be understood that the above-mentioned decomposition and representation of stroke points is only a feasible implementation method in this application, and this application does not limit the method of decomposition and representation of stroke points.

[0141] In another embodiment of the present application, if the handwritten trajectory is drawn by the display screen 193 of the electronic device, the electronic device can also generate and record the time recorded by the pressure sensor when the user draws the handwritten trajectory, and obtain and record the drawing time of each trajectory to distinguish the handwritten trajectory. In this embodiment, after the handwritten trajectory is split into single-word texts, the starting point, end point, and inflection point of the handwritten trajectory can be obtained according to the drawing order of the handwritten trajectory in the single-word text, and its trajectory coordinates can be further obtained to obtain the corresponding stroke points.

[0142] S440: Perform stroke recognition on the disassembled stroke points to obtain text content.

[0143] After the stroke points are decomposed, the feature vector can be obtained by connecting the stroke points, and then the corresponding stroke can be determined by the angle and direction of the feature vector. Then, the corresponding text content can be obtained by the stroke information in each single word text.

[0144] Exemplary, after the stroke points of the single-character text are disassembled, the stroke points and the trajectory coordinates corresponding to the stroke points are connected to obtain a feature vector, and then the feature vector of each handwritten trajectory in each single-character text is determined. It should be understood that the feature vector has information features such as direction and length. Therefore, through each feature vector in the single-character text, the trajectory features in the single-character text can be obtained, wherein the trajectory features include the length of the feature vector and the angular relationship between different feature vectors. After obtaining the trajectory features of each single-character text, the text content corresponding to each single-character text can be determined according to the trajectory features.

[0145] As shown in Figure 5, after obtaining the text content by recognizing the handwriting trajectory, it is necessary to determine the text format so that the generated text is closer to the features in the handwriting trajectory, improve the feature retention of the generated text, and then reflect the features in the handwriting trajectory by generating text, thereby improving the efficiency of organizing the handwriting trajectory. Therefore, the method includes:

[0146] S500: Determine the text format according to the number and size of characters in the text content.

[0147] In an embodiment of the present application, in order to determine the text format, it is necessary to calculate the text format by obtaining information such as the number of characters in the text content and the size of the characters. For example, the number of texts may include the total number of characters in the text content. In the aforementioned text recognition process of the handwriting trajectory, single-character segmentation will be performed to obtain single-character text. Therefore, after the text content is recognized and obtained, the number of characters in the text content can be determined based on the number of single-character texts obtained by segmentation.

[0148] The character size may include the length and width of each character in the text content, and the text format may include the font size and line spacing of the characters in the text content. Since the handwriting trace and the corresponding text content displayed on the display screen 193 are both two-dimensional, the length of each character can be obtained by the vertical scale of the character, and the width of each character can be obtained by the horizontal scale of the character.

[0149] Further, in the embodiments of the present application, the size of the text can be determined by the length and width of the single-word text, and then the text format of the text content can be determined. Exemplarily, as shown in FIG. 8, if the trajectory arrangement direction in the handwritten trajectory is horizontal, then by calculating the distance between the upper limit and the lower limit of each line of text, the line height of each line of text in the text content can be obtained, and by calculating the distance between the left limit and the right limit of each line of text, the width of each text in the text content can be obtained.

[0150] Among them, the line height of each line of text can be used as the length of each text in that line, and the upper limit and the lower limit are obtained by statistically analyzing the horizontal direction of each line of handwritten trajectory, and the left limit and the right limit are obtained by statistically analyzing the vertical direction of each line of handwritten trajectory.

[0151] Exemplarily, the size of the single-word text can also be obtained by recording the specific positions of the upper limit, the lower limit, the left limit, and the right limit of each single-word text. As shown in FIG. 8, each single-word text has its corresponding upper, lower, left, and right limits. Therefore, after the single-word text is recognized, the electronic device can be expressed in the following format.

[0152] words{

[0153] {"天",{left,top,right,bottom}}

[0154] {"气",{left,top,right,bottom}}

[0155] {"阴",{left,top,right,bottom}}

[0156] {"转",{left,top,right,bottom}}

[0157] ……}.

[0158] s]]Among them, left represents the left limit of the corresponding single-word text, top represents the upper limit of the corresponding single-word text, right represents the right limit of the corresponding single-word text, and bottom represents the lower limit of the corresponding single-word text. Based on the above method, after the text is recognized by strokes, the corresponding size of the text can be recorded together to facilitate the subsequent determination of the text format.

[0159] It should be understood that the foregoing recording of the single-word text and the limits of the single-word text is only a feasible implementation manner in the present application, and the present application does not limit the recording method.

[0160] In another embodiment of the present application, as shown in Figure 9, if the track arrangement direction in the handwriting track is vertical, the width of each character in the text content can be obtained by calculating the distance between the left and right boundaries of each column of text, and the length of each character in the text content can be obtained by calculating the distance between the upper and lower boundaries of each column of text.

[0161] The left limit and the right limit are obtained by performing vertical statistics on each column of handwriting tracks, and the upper limit and the lower limit are obtained by performing horizontal statistics on each row and column of handwriting tracks.

[0162] It should be understood that in some embodiments of the present application, the number of rows of the handwriting traces can be obtained by counting the handwriting traces horizontally, thereby determining the number of rows of the text content. Alternatively, the number of columns of the handwriting traces can be obtained by counting the handwriting traces vertically, thereby determining the number of columns of the text content.

[0163] After obtaining the number of characters in the text content and the length and width of each character through the text content, the text format can be calculated using the above parameters.

[0164] In some embodiments of the present application, taking the horizontal direction of the handwriting trajectory as an example, the text format determined by calculation includes the font size and line spacing. Since the font sizes between different lines may be different, in the process of font size calculation, each line of text can be used as a unit to calculate the font size of each line of text separately.

[0165] For example, the first font size corresponding to each line of text in the text content can be calculated by the number of lines of text content, the number of characters in each line of text, and the length of the characters in each line of text. The length of the characters in each line of text can be the length value of any character in the line of text.

[0166] It should be understood that because text content may contain characters such as letters, numbers, or symbols, the length values ​​of different characters in the same line of text may vary significantly. Therefore, in some embodiments, to avoid selecting the length value of shorter characters such as symbols or letters as the length of the characters in the line of text, the length of each character in each line of text may be the maximum length value in the line of text.

[0167] Furthermore, the second font size corresponding to each character can be calculated based on the number of characters in the text content and the length and width of each character. The second font size can correspond to multiple font sizes, but each character has at most one second font size corresponding to its font size.

[0168] To calculate the line spacing between texts in different lines, the distance between the lower and upper bounds of adjacent lines of text can be used to obtain the distance between the text-free area between the two adjacent lines of text, thereby determining the distance between the two adjacent lines of text as the first line spacing.

[0169] Alternatively, the second line spacing between different lines of text can be obtained by combining the number of lines in the text content and the length of the characters in each line of text with the length of the selection range. The length of the selection range can be obtained by the vertical extension of the selection box 1931, so that the final text content based on the text format can fill the selection range, so that the generated text retains more characteristics of the handwriting trajectory.

[0170] As shown in Figure 8, during the text format calculation process, each line of text can be horizontally projected, and each character in each line of text can be vertically projected to construct a first rectangle corresponding to each line of text and a second rectangle corresponding to each character. Each second rectangle is set within the first rectangle, and a second rectangle is only set within one first rectangle. The font size of the characters in the first rectangle can be calculated based on the length and width of the first rectangle and the number of second rectangles within the first rectangle. By calculating the font size of the characters in each first rectangle, the font size corresponding to the characters in each line of text within the selected range can be obtained.

[0171] In another embodiment of the present application, taking the handwriting trace as a vertical direction as an example, the text format determined by calculation includes the font size and column spacing. Because the font sizes of different columns may differ, the font size of each column of text can be calculated separately during the font size calculation.

[0172] For example, the third font size corresponding to each column of text in the text content can be calculated by the number of columns of text content, the number of characters in each column of text, and the width of the characters in each column of text. The width of the characters in each column of text can be the width value of any character in the column of text.

[0173] It should be understood that because text content may contain characters such as letters, numbers, or symbols, the width values ​​of different characters in the same column of text may vary significantly. Therefore, in some embodiments, to avoid selecting the width value of smaller characters such as symbols or letters as the width of the characters in the column of text, the width of the characters in each column of text may be the largest width value in the column of text.

[0174] Furthermore, the fourth font size corresponding to each character can be calculated based on the number of characters in the text content and the length and width of each character. The fourth font size can also correspond to multiple font sizes, and each character has at most one fourth font size corresponding to its font size.

[0175] To calculate the column spacing between texts in different columns, the distance between the right and left boundaries of adjacent columns of text can be used to obtain the distance between the text-free area between the two adjacent columns of text, thereby determining the distance between the two adjacent columns of text as the first column spacing.

[0176] Alternatively, a second column spacing between different columns of text can be obtained by combining the number of columns in the text content, the width of the characters in each column, and the width of the selection range. The width of the selection range can be obtained by the horizontal extension of the selection box 1931, so that the final text content based on the text format can fill the selection range, so that the generated text retains more characteristics of the handwriting trajectory.

[0177] As shown in Figure 9, during the text format calculation process, a third rectangle corresponding to each column of text and a fourth rectangle corresponding to each character in each column of text can be constructed by vertically projecting each column of text and horizontally projecting each character in each column of text. Each fourth rectangle is set within the third rectangle, and a fourth rectangle is only set within one third rectangle. The font size of the characters in the third rectangle can be calculated by calculating the length and width of the third rectangle and the number of fourth rectangles within the third rectangle. By calculating the font size of the characters in each third rectangle, the font size corresponding to the characters in each column of text within the selected range can be obtained.

[0178] Because different character types, such as letters and Chinese characters, have different lengths and widths in the same font size, in some embodiments of the present application, the character type of the recognized text content may be determined before determining the text format.

[0179] For example, the character type of each word in the text content can be determined based on the text content. Character types may include Chinese characters, letters, numbers, and symbols. It should be understood that the text content recognized by the handwriting trajectory may include one or more types of characters in the character type, and this application does not limit the specific character type of the characters in the text content.

[0180] After identifying and obtaining the character type corresponding to each word, the font size of different words can be determined according to the character type corresponding to the word. For example, if the character type of the first word is a letter, number or symbol, the font size of the second word adjacent to the first word is determined to be the font size of the first word. The first word is any word in the text content, and the second word includes words in the text content whose character type is Chinese characters. The above method can be used to determine the font size of words such as letters, numbers and symbols in the text content as the font size of adjacent Chinese characters, thereby avoiding the problem of font size recognition errors due to the small size of words such as letters, numbers and symbols.

[0181] In a feasible implementation, the font sizes of Chinese characters and other character types can be determined separately, so that each character has a font size that better matches its handwriting trajectory. This application does not limit the font size determination process for characters of different character types.

[0182] After determining the text format by any of the above methods, the identified text and the obtained text format can be displayed for the user to view. At the same time, the text content and / or text format can be modified by receiving user instructions.

[0183] FIG10 is a schematic diagram of a process for displaying text in an embodiment of the present application.

[0184] As shown in FIG10 , in some embodiments of the present application, the process of displaying text on the display screen 193 may include:

[0185] S610: Displaying a first interface on the display screen.

[0186] FIG11 is a schematic diagram showing a display of a handwritten note converted into text in an embodiment of the present application.

[0187] As shown in FIG11(b), the first interface 1932 includes a text display area 1933 and interface controls 1934. The text display area 1933 can display text content, while the interface controls 1934 can display text formatting. Specifically, the interface controls 1934 shown in FIG11 include font size and color. Because information such as line spacing is also obtained during text format recognition, the interface controls 1934 may also include a control for displaying line spacing.

[0188] It should be understood that the first interface 1932 in the present application may also not include the interface control 1934, that is, not display the recognized text format, thereby preserving the format characteristics of the handwriting trace. In addition to displaying the recognized text content, the first interface 1932 is not limited to the display content of the first interface in the present application.

[0189] S620: In response to the user's confirmation operation, display the text on the display screen.

[0190] The text includes text content whose font format conforms to the text format and whose font color is the first color. It should be understood that if the user modifies the text content, text format and font color through the modification instruction, the generated text is the user-modified version.

[0191] As shown in FIG11( b ) and ( c ), in order to display the text content on the display screen 193, a confirmation control 1935, namely, "Confirm Replace" in the figure, is further provided on the first interface 1932. In response to the user's confirmation operation on the first interface 1932, the text can be displayed on the display screen 193. The response to the user's confirmation operation on the first interface 1932 is the response to the user's click operation on the confirmation control 1935.

[0192] Specifically, as shown in FIG11(d), after receiving a confirmation operation on the first interface 1932, the electronic device may generate a text box 1936 of the same size as the selected range. The text is set in the text box 1936, and the format of the text is the same as the text format displayed in the interface control 1934. In this way, the handwriting traces can be organized by converting the text into text that the electronic device can recognize and replacing the original selected range.

[0193] As shown in FIG5 , the method for converting handwritten notes into text in the embodiment of the present application further includes:

[0194] S400: Recognize the handwriting trace to obtain text content.

[0195] It should be understood that step S400 can be implemented through the above-mentioned steps S410 to S440, and therefore will not be described in detail in the embodiment of the present application.

[0196] S600: In response to the user's confirmation operation, display the text on the display screen.

[0197] It should be understood that step S600 can be implemented through the above-mentioned steps S610 to S620, and therefore will not be described in detail in the embodiment of the present application.

[0198] Based on the implementation of the aforementioned method for converting handwritten notes into text, this application also provides an electronic device interaction method. As shown in FIG11 , taking a tablet computer as an example, the display screen 193 is coupled to at least one of a touch sensor and a pressure sensor, so that the electronic device can receive user instructions through the display screen 193.

[0199] As shown in the aforementioned Figure 6, in order to be able to enter and convert handwritten notes, operation controls can be displayed above the display screen 193. The operation controls can be used to wake up functions such as drawing, erasing, changing color, lasso selection, and inserting. By clicking on the operation control, the user can wake up the corresponding function of the control, and then the user can realize the conversion of handwritten notes through the operation of the above functions.

[0200] As shown in (a) in Figure 11, during the conversion of handwritten notes, the user can first activate the lasso selection function by clicking the control corresponding to the function, and then slide from one point in the handwriting area to another point to determine the rectangular box with the two points as diagonals as the selection box 1931. At the same time, a temporary control 1937 is displayed on one side of the selection box 1931.

[0201] Temporary control 1937 is used to operate the handwriting trace selected by selection box 1931. For example, temporary control 1937 may include operations such as cut, copy, adjust color, convert handwriting to text, and delete. It should be understood that Figure 11 (a) shows only one possible implementation in this application. Temporary control 1937 may include other functions. This application does not limit the content included in temporary control 1937.

[0202] After displaying temporary control 1937, the user can convert the handwriting trace selected by selection box 1931 by clicking on the handwriting-to-text option in temporary control 1937. After receiving the user's operation, the electronic device can convert the handwriting trace according to the aforementioned method of converting handwritten notes into text, which will not be described in detail in this application.

[0203] After the electronic device obtains the text content in step S440 and the text format in step S500, it needs to display the recognized text content and text format through the first interface 1932. As shown in Figure 11(b), after the recognition is completed, the first interface 1932 can be displayed on the display screen 193. The content displayed on the first interface 1932 is the same as in the previous embodiment and will not be described in detail in this application.

[0204] FIG12 is a schematic diagram of operating the first interface in an embodiment of the present application.

[0205] In order to achieve the modifiability of the text content, the text display area 1933 and the interface control 1934 are editable areas, so that they can receive the user's modification instructions. As shown in (a) and (c) of Figure 12, the text content obtained by recognizing the handwriting trajectory can be displayed on the display screen 193 through the text display area 1933. During the display process, the user can click on the text display area 1933 and use an input method such as a virtual keyboard as shown in (c) of Figure 11 to enter a first modification instruction into the text display area 1933. After the electronic device receives the first modification instruction in the text display area 1933 in the first interface 1932, it can modify the text content displayed in the text display area according to the content entered by the user.

[0206] The first modification instruction includes adding and / or deleting text content. Specifically, as shown in FIG12(c), the electronic device can receive the first modification instruction, obtain the text content input by the user, and display it together with the recognized text content in the text display area 1933, thereby achieving the effect of adding text content.

[0207] Furthermore, the electronic device may also receive the first modification instruction to obtain an instruction to delete part of the text content, thereby deleting the corresponding text from the text display area 1933 to achieve the effect of deleting the text content.

[0208] As shown in (a) and (d) of FIG12 , the interface control 1934 may include multiple drop-down boxes. By clicking a drop-down box, the user can bring up a drop-down menu with other font sizes or colors. The user can make a selection in the drop-down menu to modify the text format.

[0209] The interface control 1934 can display the text format. The electronic device can receive and respond to the second modification instruction of the first interface 1932 through the interface control 1934, thereby modifying the text format identified through the aforementioned steps to obtain a text format that better meets user needs and improve the efficiency of text organization, for example, changing the originally identified font size 4 to font size 3.

[0210] In addition to displaying the recognized text format, interface control 1934 can also display the first color of the recognized handwriting trace. Furthermore, interface control 1934 can also receive a third modification instruction to modify the first color in interface control 1934, such as changing the originally recognized purple to red.

[0211] For example, as shown in (a) and (b) of Figure 12, the text content displayed in the text display area 1933 can also receive a long press instruction. The display screen 193 detects the user's long press on the text display area 1933 through a sensor, and can select the pressed text in the text display area 1933 according to the pressing position and sliding method, and a control appears after the user finishes pressing, so that the user can copy, cut, translate, share, select all, etc. on the selected text. It should be understood that the control shown in (b) of Figure 12 is only an example, and the location where the control appears and the content included are not limited in this application.

[0212] In some embodiments, such as in some embodiments of the present application, the first interface 1932 also includes a copy control, and the user can control the copy control to copy the content in the text display area 1933 to the system clipboard, so that the copied content can be directly pasted to the location where the user needs to input in the text input scenario.

[0213] As shown in (e) and (f) of FIG12 , if the electronic device fails to recognize the text in the handwriting trace or the selection range does not include the handwriting trace, the display screen 193 may still display the first interface 1932, but the text display area 1933 and / or the display control 1934 are empty, and the text display area 1933 may display a prompt "No content recognized" to remind the user that recognition is complete but no valid content is recognized in the selection range. At this time, the user can click on the text display area 1933 to eliminate the prompt, and the text display area 1933 can receive the first modification instruction to add and modify text.

[0214] In another embodiment, if the electronic device does not recognize the text in the handwriting trace or the selection range does not include the handwriting trace, the user can abandon the currently selected content by clicking the exit control in the first interface 1932. As shown in (e) and (g) of Figure 12, an exit control may be provided in the upper left corner of the first interface 1932. By clicking the exit control, the user can switch to the interface shown in (g) of Figure 12. If the user clicks "Abandon", the first interface 1932 of the electronic device is closed and the handwriting trace before the selection box 1931 is drawn is displayed. If the user clicks "Cancel", the first interface 1932 continues to be displayed.

[0215] It should be understood that the user can close the first interface 1932 at any time by clicking the exit control on the first interface 1932. The present application does not impose any restrictions on the timing of clicking the exit control. Furthermore, the page shown in FIG. 12(g) may also appear directly if a recognition error occurs, and after the user clicks to abandon, the display screen 193 before drawing the selection box 1931 is displayed, allowing the user to select the selection range again.

[0216] Through the above implementation and as shown in (c) of FIG11 , the user can call out the corresponding function by clicking the text display area 1933 and the interface control 1934 and modify the displayed text content and text format through modification instructions.

[0217] As shown in Figure 11(c), after the user completes the modification, they can confirm the current text content and text format by clicking the confirmation control 1935. As shown in Figure 11(d), the electronic device generates text based on the current text content and text format to replace the handwriting trace, thereby realizing the function of converting handwritten notes into text.

[0218] It should be noted that, in order to realize the above functions, the above electronic device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0219] The embodiment of the present application can divide the functional modules of the above-mentioned electronic device according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0220] Figure 13 is a schematic diagram of a handwritten note conversion system in an embodiment of the present application; Figure 14 is a flow chart of the conversion method executed by the conversion system in an embodiment of the present application.

[0221] In the case of dividing each functional module according to each function, as shown in Figures 13 and 14, an embodiment of the present application also provides a handwritten note conversion text system used in an electronic device, which may include: a note display module 1310 and a handwriting recognition module 1320.

[0222] It should be understood that the electronic device in this embodiment is still exemplified by the electronic device shown in FIG. 2 .

[0223] The note display module 1310 may include a canvas 1311 and a window display unit 1312. The note display module 1310 may display content on the display screen 193 of the electronic device via the canvas 1311. The handwriting recognition module 1320 may include a trajectory drawing unit 1321, a command response unit 1322, a trajectory recognition unit 1323, and a text generation unit 1324.

[0224] As shown in FIG14 , a user can draw handwritten notes in the handwriting area of ​​display screen 193, where the handwriting area of ​​display screen 193 can be canvas 1311. During the user's handwriting drawing process, trajectory drawing unit 1321 can receive pressure signals and / or tactile signals via a pressure sensor and / or touch sensor coupled to display screen 193, thereby drawing a handwriting trajectory. Trajectory drawing unit 1321 can then send the completed handwriting trajectory image to canvas 1311 in note display module 1310, thereby displaying the corresponding handwriting trajectory on display screen 193.

[0225] When handwritten notes need to be converted, the user can select the content in the canvas 1311 and frame a selection range on the canvas 1311. After receiving the selection range, the trajectory drawing unit 1321 can send the selection range and the handwriting trajectory to the instruction response unit 1322, so that the instruction response unit 1322 sends the handwriting trajectory image with the selection range to the canvas 1311, making the selection range clearer.

[0226] When sending the handwriting trace image with the selected range to canvas 1311, instruction response unit 1322 needs to send the handwriting trace within the selected range to trace recognition unit 1323, so that trace recognition unit 1323 can recognize the handwriting trace to obtain the first color, text content, and text format. It should be understood that instruction response unit 1322 sending the handwriting trace to trace recognition unit 1323 can be performed simultaneously with the process of instruction response unit 1322 sending the image to canvas 1311. This application does not limit the order of these two steps.

[0227] Meanwhile, the functional implementation process of the trajectory recognition unit 1323 refers to the above steps S200 to S440, and this application will not elaborate on it here.

[0228] After the trajectory recognition unit 1323 recognizes the text content, text format and first color, the recognized content can be sent to the window display unit 1312, so that the recognized text content, text format, first color and other information can be displayed to facilitate user modification.

[0229] If the user needs to modify the text, they can operate the window display unit 1312 through the display screen 193. The specific operation process is described in the interaction method for the first interface 1932, which is not described in detail in this application. By operating the window display unit 1312, the recognized text content, text format and / or first color can be modified to make the text content and format more in line with the user's needs, thereby improving the efficiency of organizing handwritten notes.

[0230] If the user does not need to modify or has completed the modification, the result can be confirmed by clicking on the display screen 193. Specifically, when the window display unit 1312 receives the user's confirmation operation, the displayed text content, text format, first color and other information can be sent to the text generation unit 1324, and the text generation unit 1324 will generate corresponding text and send it to the canvas 1311 to replace the selected range in the canvas 1311.

[0231] It should be understood that the above-mentioned structural implementation is only an example in the embodiment of the present application. The system can also reduce the connections in the handwriting recognition module 1320 and improve the recognition efficiency by setting a memory in the handwriting recognition module 1320 and storing the sent and received information in the memory.

[0232] Regarding the electronic device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method for converting handwritten notes into text in the above embodiment, and will not be further elaborated here. The relevant beneficial effects thereof can also be referred to the relevant beneficial effects of the above method for converting handwritten notes into text, and will not be repeated here.

[0233] The present application also provides an electronic device comprising: a display screen, a memory, and one or more processors; the display screen, the memory, and the processors are coupled; wherein the memory stores computer program code, the computer program code comprising computer instructions, which, when executed by the processor, causes the electronic device to execute the method for converting handwritten notes into text as provided in the aforementioned embodiments. The specific structure of the electronic device may be as shown in FIG2 .

[0234] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0235] In the several embodiments provided in this application, it should be understood that the disclosed systems / devices and methods can be implemented in other ways. For example, the system / device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of the system or unit, which can be electrical, mechanical or other forms.

[0236] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0237] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0238] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0239] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for converting handwritten notes into text, characterized in that, Applied to an electronic device with a display screen, the method includes: In response to a user's operation on a handwriting area, obtaining a selection range in the handwriting area; the handwriting area is an area on the display screen that can receive the user's drawn trajectory; Obtaining the handwriting trajectory within the selection range; Identifying the first color of the handwriting trajectory; Identifying the handwriting trajectory to obtain text content; Determining a text format according to the number of characters and the character size in the text content; In response to a user's confirmation operation, displaying the text text on the display screen; the text text includes the text content with a font format conforming to the text format and a font color being the first color.

2. The method for converting a handwritten note into a text text according to claim 1, wherein: The number of characters includes the number of characters in the text content, the number of lines of characters, and the number of characters in each line of text; The character size includes the length and width of each character in the text content; The text format includes a font size and a line spacing.

3. The method for converting a handwritten note into a text text according to claim 2, wherein: The determining the text format according to the number of characters and the character size in the text content includes: Calculating a first font size corresponding to each line of the text content according to the number of lines of the text content, the number of characters in each line of text, and the length of the characters in each line of text; or, Calculating a second font size corresponding to each line of the text content according to the number of characters in the text content and the length and width of each character; Determining a first line spacing according to the spacing between each line of text in the text content; or, Determining a second line spacing according to the number of lines of the text content and the length of the characters in each line of text.

4. The method for converting a handwritten note into a text text according to any one of claims 1 to 3, wherein: After determining the text format according to the number of characters and the character size in the text content, further includes: Displaying a first interface on the display screen; the first interface includes a text display area and an interface control, the text display area displays the text content, and the interface control displays the text format; The displaying the text text on the display screen in response to a user's confirmation operation includes: In response to a user's confirmation operation on the first interface, displaying a text box on the display screen; the text box includes the text text, the size of the text box is the same as the selection range, and the format of the text text is the same as the text format displayed in the interface control.

5. The method for converting a handwritten note into a text text according to claim 4, wherein: After identifying the handwriting trajectory to obtain text content, further includes: Determining the character type of each character in the text content according to the text content; the character type includes Chinese characters, letters, numbers, and symbols; After calculating the font size corresponding to the text content, further includes: If the character type of the first character is a letter, a digit, or a symbol, determine the font size of the second character adjacent to the first character as the font size of the first character; the first character is any character in the text content, and the second character includes characters in the text content whose character type is Chinese characters.

6. The method for converting a handwritten note into a text as claimed in claim 4, wherein responding to the user's confirmation operation and displaying the text on the display screen further includes: responding to a first modification instruction of the user for the first interface and modifying the text content in the text display area; and / or responding to a second modification instruction of the user for the first interface and modifying the text format in the interface control.

7. The method for converting a handwritten note into a text as claimed in claim 1, wherein if the first color includes at least two colors, after identifying the handwritten trajectory to obtain the text content, further includes: obtaining the first color corresponding to each character in the text content according to the correspondence between the first color and the handwritten trajectory.

8. The method for converting a handwritten note into a text as claimed in claim 1, wherein identifying the handwritten trajectory to obtain the text content includes: determining the trajectory coordinates corresponding to the handwritten trajectory within the selected range; segmenting the handwritten trajectory into single characters to obtain single-character texts; decomposing the single-character texts according to the trajectory coordinates into stroke point positions; identifying the stroke point positions obtained by decomposition to obtain the text content.

9. The method for converting a handwritten note into a text as claimed in claim 8, wherein identifying the stroke point positions obtained by decomposition to obtain the text content includes: determining the feature vector of the handwritten trajectory in each single-character text according to the stroke point positions and the trajectory coordinates corresponding to the stroke point positions; obtaining the trajectory feature according to the feature vector; determining the text content corresponding to each single-character text according to the trajectory feature of each single-character text.

10. The method for converting handwritten notes into text as claimed in claim 1, characterized in that, The electronic device includes a note display module and a handwriting recognition module, and the method includes: the handwriting recognition module responds to the user's operation on the handwriting area and obtains the selected range within the display range of the note display module; reads the handwritten trajectory within the selected range through the handwriting recognition module and identifies the first color of the handwritten trajectory; the handwriting recognition module identifies the handwritten trajectory to obtain the text content, and determines the text format according to the number of characters and the font size in the text content; the handwriting recognition module sends the text content and the text format to the note display module; obtaining the user's confirmation operation through the note display module, and the handwriting recognition module sends the text content in the text format and the first color to the note display module; The text is displayed on the display screen through the note display module.

11. An electronic device, characterized in that, Comprising: A display screen, a memory, and one or more processors; the display screen, the memory are coupled to the processor; wherein, computer program code is stored in the memory, the computer program code includes computer instructions, when the computer instructions are executed by the processor, the electronic device is caused to execute the method for converting handwritten notes into text as described in any one of claims 1-10.

Images