Text editing method and related equipment

Abstract

The invention discloses a text editing method and related equipment. The method can comprise the steps that the electronic equipment detects and responds to a first operation acting on a first interface, and a first input event is generated; when the first input event is the handwriting pen input event and the input position corresponding to the first input event is in the effective stroke starting area, the electronic equipment creates a writing layer; detecting and responding to a second operation acting on the writing layer, and generating a first input event sequence; recognizing the first input event sequence through a recognition engine to obtain a first text or a first editing mode, and editing the text in the self-defined text control in the first interface based on the first text or the first editing mode. By means of the method, recognition of the handwriting to the text and the pen gesture can be achieved on the application of the user-defined text control, the use scene of the handwriting pen is expanded, a user can flexibly edit the text through the handwriting pen, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the technical field of terminals, and in particular, to a text editing method and related devices. Background Art

[0002] Terminals such as mobile phones and tablet computers (i.e., electronic devices) have developed rapidly and become increasingly popular. These terminals have rich and diverse functions, improving people's quality of life and making people's work and study more efficient. To further improve the user experience, many manufacturers have equipped these terminals with a stylus, enabling users to complete input on the terminal through the stylus, providing more input possibilities.

[0003] Currently, for text controls implemented based on the capabilities of the operating system itself in a terminal, such as an input box (EditText) and a text box (TextView), input through a stylus can be achieved based on the capabilities of the operating system itself. However, there are some applications that use text controls defined by the application itself rather than those implemented based on the capabilities of the operating system itself. For these applications, input through a stylus cannot be achieved based on the capabilities of the operating system itself. Summary of the Invention

[0004] This application provides a text editing method and related devices. According to this text editing method, when an electronic device detects a stylus input operation within a valid starting area of a first application, the first application can create a writing layer and call an identification engine to identify the input event acting on the writing layer to obtain an identification result, and then edit the text in the custom text control corresponding to the valid starting area based on the identification result. This method enables applications using custom text controls to also achieve the recognition of handwritten content and pen gestures, expanding the usage scenarios of the stylus.

[0005] In a first aspect, the present application provides a text editing method. This method can be applied to an electronic device equipped with a display screen. According to this method, the electronic device can display a first interface through the display screen and detect a first operation acting on the first interface. In response to the first operation, the electronic device can generate a first input event. When the first input event is a stylus input event and the input position corresponding to the first input event is within a first area, the electronic device can create a writing layer. After creating the writing layer, the electronic device can detect a second operation acting on the writing layer. In response to the second operation, the electronic device can generate a first input event sequence, and identify the first input event sequence through an identification software development kit (SDK) to obtain a first text or a first editing method, and edit the text in a first control in the first interface based on the first text or the first editing method. Among them, the first control is not a text control provided by the operating system of the electronic device. The first area includes the area where the first control is located. The writing layer is a transparent layer covering the first interface.

[0006] In the solution provided by the present application, the user can write on the user interface displayed on the display screen with a stylus to complete the input. Specifically, when the stylus lands on or hovers above the display screen, the electronic device can detect this operation. Further, the electronic device can generate a corresponding input event. When the input position corresponding to the input event is within a valid starting area, it indicates that the input event is an input event for a custom text control, and the electronic device can create a writing layer. Subsequently, the user can write and draw on the writing layer with the stylus, and the electronic device can detect the corresponding operation and generate a corresponding input event, and identify the corresponding input event through an identification engine to obtain text or an editing method. When the identified text is obtained, the electronic device can add the text to the text included in the custom text control. When the identified editing method is obtained, the electronic device can edit the corresponding text included in the custom text control according to the editing method, such as deleting or selecting part of the text.

[0007] Since applications using custom text controls do not use the text controls provided by the operating system, the input method application cannot obtain the position of the control that needs to be input through the operating system. Therefore, applications using custom text controls cannot implement input for custom text controls through the input method application. Applications using custom text controls can only develop a self-developed input method application based on the position of their custom text controls to implement the conversion of handwritten content to text and gesture recognition. However, the self-developed input method application cannot be applied to other applications using custom text controls because the self-developed input method application also cannot obtain the position of the custom text control in other applications using custom text controls. The solution provided by this application can provide the ability of text recognition and gesture recognition for all applications using custom text controls. These applications only need to call the recognition engine on the system side, without the need to develop a self-developed input method application, further expanding the usage scenarios of the stylus and enabling users to use the stylus for flexible text editing, thus enhancing the user experience.

[0008] In some embodiments of this application, the first interface may be the user interface of Application 1 mentioned in step S1 (as Figure 3 shown).

[0009] In some embodiments of this application, the first interface may be the user interface 104 as Figure 1D shown.

[0010] In some embodiments of this application, the first operation may be the stylus input operation on the user interface of Application 1 mentioned in step S1 (as Figure 3 shown).

[0011] In some embodiments of this application, the first operation may include: when the user uses the stylus, putting its tip on the display screen of the electronic device (as Figure 1D shown).

[0012] In some embodiments of this application, the first operation may include: when the user uses the stylus, hovering it above the display screen of the electronic device.

[0013] In some embodiments of this application, the first input event may be the input event 1 mentioned in step S101 and step S201.

[0014] In some embodiments of this application, the first area may be the effective starting area mentioned below.

[0015] In some embodiments of this application, the input position corresponding to the first input event is within the first area, which can be understood as: the stylus input operation on the user interface of Application 1 mentioned in step S2 acts within the effective starting area.

[0016] In some embodiments of the present application, the second operation may be the stylus input operation on the handwriting layer mentioned in step S3.

[0017] In some embodiments of the present application, the second operation may include the stylus input operation corresponding to input event 2. Similarly, in some embodiments of the present application, the second operation may include the stylus input operation corresponding to input event 3. Similarly, in some embodiments of the present application, the second operation may include the stylus input operation corresponding to input event 4.

[0018] In some embodiments of the present application, the first input event sequence may be the corresponding stylus input events mentioned in step S4.

[0019] In some embodiments of the present application, the first input event sequence may include input event 2.

[0020] In some embodiments of the present application, the first input event sequence may include input event 3.

[0021] In some embodiments of the present application, the first input event sequence may include input event 4.

[0022] It can be understood that the recognition software development kit (i.e., recognition SDK) may be the recognition engine mentioned below.

[0023] In some embodiments of the present application, the first text may include "therefore" in Document 1 as shown in Figure 1F . In some embodiments of the present application, the first text may include "the old man" in Document 1 as shown in Figure 1G .

[0024] It can be understood that the first editing method may be any one or more of the editing methods mentioned below. In some embodiments of the present application, the first editing method may include deletion as shown in Figure 1H and Figure 1I .

[0025] In some embodiments of the present application, the recognition result mentioned in step S5 may include the first text.

[0026] In some embodiments of the present application, the recognition result mentioned in step S5 may include the first editing method.

[0027] In some embodiments of the present application, the first control may be a custom text control. In some embodiments of the present application, the first control may be the custom text control when Application 1 displays Document 1. Specifically, reference may be made to Figures 1C - 1K .

[0028] In combination with the first aspect, in a possible implementation manner, the first interface is the user interface of the first application. Creating a writing layer may specifically include: The electronic device creates a writing layer through the first application. Before the electronic device recognizes the first input event sequence through the recognition SDK, the method may further include: The electronic device may call the recognition SDK through a first interface in the first application, and send the first input event sequence to the recognition SDK through the first application. Wherein, the first interface is a custom recognition interface encapsulated in the first application.

[0029] In the solution provided by this application, the electronic device can generate corresponding original input events in response to the first operation. After being processed by other upper-layer modules subsequently, the first input events can be obtained and transmitted to the first application. After receiving the first input events, the first application can determine whether the first input events are stylus input events, and whether the corresponding input positions are within the effective starting area corresponding to the custom text control. When the first input events are stylus input events and the input positions corresponding to the first input events are within the first area, the first application can create a writing layer and call the recognition SDK through the first interface in the first application (specifically, refer to step S105-step S107 and step S205-step S207). The electronic device can generate corresponding several original input events in response to the second operation. After being processed by other upper-layer modules subsequently, the first input event sequence can be obtained and transmitted to the first application. After receiving the first input event sequence, the first application can send the first input event sequence to the recognition SDK. After receiving the first input event sequence, the recognition SDK can recognize text or pen gestures based on the information carried by the first input event sequence (for example, the relevant information of the contact points mentioned below), and obtain the first text or the first editing method.

[0030] Since the first application itself can obtain the position of the custom text control, the first application can, when detecting a stylus input operation on the custom text control, create a writing layer by itself (create through the drawing module or call the drawing engine to create), and directly call the recognition SDK through the recognition interface, without having to launch the input method application when detecting a stylus input operation (only the input method application specifically customized according to the position of the custom text control in the first application can meet the recognition requirements), and perform writing and recognition in the input method application.

[0031] On the one hand, the entire process of the above method is relatively simple and easy to operate. On the other hand, the above method provides a recognition engine on the system side, and only needs to set a corresponding interface (such as the first interface) for the custom text control to call the recognition engine, which can be widely applied to devices using custom text controls.

[0032] In some embodiments of the present application, the first application may be Application 1.

[0033] In combination with the first aspect, in a possible implementation manner, the first interface is the user interface of the first application. The recognition SDK may include a gesture recognition SDK and a text recognition SDK. The electronic device recognizes the first input event sequence through the recognition SDK to obtain the first text or the first editing method, and edits the text in the first control based on the first text or the first editing method. Specifically, it may include: when the input position corresponding to the first input event is on the text included in the first control, the electronic device may recognize the first input event sequence through the gesture recognition SDK to obtain the first editing method, and determine the text to be edited through the first application, and edit the text to be edited based on the first editing method; however, when the input position corresponding to the first input event is within the blank area included in the first control, the electronic device may recognize the first input event sequence through the text recognition SDK to obtain the first text, and add the first text to the text included in the first control through the first application. The position of the text to be edited is the same as the input position corresponding to the first input event sequence.

[0034] In the solution provided by the present application, the electronic device can first determine whether the user needs to perform gesture recognition or recognition of handwritten content (also referred to as handwritten text), that is, text recognition. When it is determined that the user needs to perform gesture recognition, the gesture recognition SDK is called for recognition, and when it is determined that the user needs to perform text recognition, the text recognition SDK is called. This method can further improve the accuracy of recognition, and will not recognize gesture actions as text, nor will it recognize handwritten text as a gesture.

[0035] In some embodiments of the present application, the first editing method may be deletion.

[0036] In some embodiments of the present application, the text to be edited may be Figure 1H "therefore" in Document 1 as shown.

[0037] In combination with the first aspect, in a possible implementation, after the electronic device recognizes the first input event sequence through the recognition SDK, the method may further include: under the condition of meeting the first preset condition, the electronic device may delete the writing layer; under the condition of meeting the second preset condition, the electronic device may close the recognition SDK. Among them, meeting the first preset condition may include any one or more of the following: the interval duration between the current moment and the last detected user operation on the writing layer is equal to or greater than the first duration; a non-stylus input event is detected. Meeting the second preset condition may include any one or more of the following: the interval duration between the current moment and the last detected user operation on the writing layer is equal to or greater than the second duration; a non-stylus input event is detected; a stylus input event with the corresponding event type being the first type is detected.

[0038] In the solution provided in this application, the electronic device can delete the writing layer and close the recognition SDK under certain conditions. This way can reduce the device power consumption to a certain extent.

[0039] In some embodiments of this application, the first preset condition may be the preset condition 1 mentioned below.

[0040] In some embodiments of this application, the second preset condition may be the preset condition 2 mentioned below.

[0041] In some embodiments of this application, the first duration may be T11, and the second duration may be T12.

[0042] In some embodiments of this application, the first duration and the second duration may be T1.

[0043] In some embodiments of this application, the first duration and the second duration may be T2.

[0044] In some embodiments of this application, the first type may be a pen-lifting event.

[0045] It can be understood that the interval duration between the current moment and the last detected user operation on the writing layer may be the duration of no detected input event mentioned below.

[0046] In combination with the first aspect, in a possible implementation, the first interface is the user interface of the first application. The first input event sequence may include a series of successively generated input events. The electronic device can identify the first input event sequence through the recognition SDK, which may include: the electronic device can identify M1 input events in the first input event sequence through the recognition SDK to obtain a first recognition result, and send the first recognition result to the first application through the recognition SDK; the electronic device can also identify M2 input events in the first input event sequence through the recognition SDK to obtain a second recognition result, and in the case where the second recognition result is different from the first recognition result, send the first recognition result to the first application through the recognition SDK. Wherein, the recognition result is text or an editing method. M1 is a positive integer. M2 is a positive integer.

[0047] In the solution provided by this application, the user makes a single writing on the display screen with a stylus (for example, writes one or more strokes, one or more characters, etc.). Correspondingly, the electronic device can generate multiple input events. These multiple input events can be understood as an input event sequence (for example, the first input event sequence). Since the generation times of the input events in this input event sequence are in a sequential order, after the electronic device receives a part of the input events, it can first perform recognition based on this part of the input events. After subsequently receiving another part of the input events, it can perform recognition based on this part of the input events. If it subsequently receives yet another part of the input events, it can perform recognition based on this part of the input events. That is to say, the electronic device can perform multiple recognitions on the first input event sequence, and each recognition can be directed at one or more input events in the first input event sequence. This method can improve the recognition accuracy while ensuring the recognition speed, so as to obtain a relatively accurate recognition result in a relatively short time.

[0048] In combination with the first aspect, in a possible implementation, the generation time of the M2 input events is later than the generation time of the M1 input events.

[0049] In the solution provided by this application, the electronic device can perform multiple recognitions on the first input event sequence. For example, every time it receives a part of the input events in the first input event sequence, it can perform recognition on this part of the input events, so that multiple recognition results can be obtained. It can be understood that the times when the electronic device obtains these multiple recognition results are different. In the case where these multiple recognition results are different, the electronic device can select the recognition result obtained at the latest as the final recognition result, because the input events generated at a later time can better reflect the content that the user wants to write, and the recognition result obtained in this case has a higher accuracy.

[0050] In combination with the first aspect, in a possible implementation, M2 is greater than M1, and the M2 input events include one or more of the M1 input events. The generation times of the other input events among the M2 input events except the M1 input events are later than the generation times of the M1 input events.

[0051] In the solution provided by this application, the electronic device can identify the first input event sequence multiple times. For example, every time a part of the input events in the first input event sequence is received, the electronic device can identify by combining this part of the events and the previously received input events, so that multiple identification results can be obtained. It can be understood that the times when the electronic device obtains these multiple identification results are different. In the case where these multiple identification results are different, the electronic device can select the identification result obtained at the latest as the final identification result, because the input events generated at multiple times can better reflect the content that the user wants to write, and the identification result obtained in this case is more accurate.

[0052] In combination with the first aspect, in a possible implementation, in the case where the second identification result is different from the first identification result, the second identification result is the first text or the first editing method; in the case where the second identification result is the same as the first identification result, the first identification result is the first text or the first editing method.

[0053] In the solution provided by this application, in the case where the multiple identification results obtained by the electronic device for identifying the first input event sequence are different, the electronic device can select the identification result obtained at the latest as the final identification result, because the input events generated at multiple times can better reflect the content that the user wants to write, and the identification result obtained in this case is more accurate.

[0054] In combination with the first aspect, in a possible implementation, after the electronic device obtains the first text, the method may further include: the electronic device can find associated words based on the first text through a text recognition SDK, and send one or more of the found associated words to the first application. After the electronic device adds the first text to the text included in the first control through the first application, the method may further include: the electronic device can display a second interface through the display screen; detect a third operation on the first associated word in the first display area; in response to the third operation, add the first associated word to the text included in the first control. Wherein, the second interface includes the first control and the first display area, and the first display area can display the one or more found associated words.

[0055] In the solution provided by this application, after the electronic device recognizes the handwritten content written by the user with a stylus and obtains the corresponding text, it can search for relevant associated words and display the found associated words on the display screen for the user to select. This method can further reduce the user's manual writing and make it more convenient for the user to input through the stylus.

[0056] In some embodiments of this application, the second interface may be the user interface 113 as shown in Figure 1K . The first display area may be the display area 1131 as shown in Figure 1K . The first associated word may be "immortal" as shown in Figure 1K .

[0057] It can be understood that the third operation may be user operations such as touch, voice control, gesture, etc., and this application does not make specific restrictions on this.

[0058] In some embodiments of this application, one or more associated words found by the electronic device may include associated word 1.

[0059] In a second aspect, this application provides an electronic device, which includes: one or more memories, and one or more processors; the one or more memories are coupled to the one or more processors, the memories are used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method described in the first aspect or any implementation manner of the first aspect.

[0060] In a third aspect, this application provides a computer storage medium. The computer storage medium includes computer instructions, and when the computer instructions run on the electronic device, the electronic device is enabled to execute the method described in the first aspect or any implementation manner of the first aspect.

[0061] In a fourth aspect, an embodiment of this application provides a chip. The chip can be applied to an electronic device. The chip includes one or more processors, and the processors are used to call computer instructions to enable the electronic device to execute the method described in the first aspect or any implementation manner of the first aspect.

[0062] In some embodiments of this application, the chip system may be an Application Processor (AP), or a System on Chip (SoC) including an AP. The method described in the first aspect or any implementation manner of the first aspect may be implemented by one AP, and the method described in the second aspect or any implementation manner of the second aspect may be implemented by one AP.

[0063] In some other embodiments of the present application, the chip system may include an AP and other modules. The other module may be a modem (also referred to as a baseband processor).

[0064] In a fifth aspect, an embodiment of the present application provides a computer program product containing instructions. When the computer program product runs on an electronic device, it causes the electronic device to execute the method described in the first aspect or any implementation manner of the first aspect.

[0065] It can be understood that the electronic device provided in the second aspect, the computer storage medium provided in the third aspect, the chip provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the method described in the first aspect or any implementation manner of the first aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of any possible implementation manner in the first aspect above, and will not be elaborated here. Description of the Drawings

[0066] Figures 1A - 1K It is a schematic diagram of a set of user interfaces provided by an embodiment of the present application;

[0067] Figure 2 It is a software structure of an electronic device provided by an embodiment of the present application;

[0068] Figure 3 It is a flowchart of a text editing method provided by an embodiment of the present application;

[0069] Figures 4A - 4B It is a schematic diagram of adding a writing layer to Application 1 in a group;

[0070] Figures 5A - 5B It is a schematic diagram of adding a writing layer to Application 1 in another group;

[0071] Figure 6A It is a flowchart of another text editing method provided by an embodiment of the present application;

[0072] Figure 6B It is a flowchart of another text editing method provided by an embodiment of the present application;

[0073] Figure 7 It is a hardware structure of an electronic device provided by an embodiment of the present application. Detailed Embodiments

[0074] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B. The "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality of" means two or more than two.

[0075] It should be understood that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0076] It should be understood that the term "user interface" in the specification, claims and drawings of the present application is a media interface for interaction and information exchange between an application or an operating system and a user. The common form of presentation of the user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations presented in a graphical manner. It can be an interface element such as an icon, window, control, etc. displayed on the display screen (also known as a touch screen) of an electronic device. Among them, the control can include visible interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0077] Referring to "embodiments" in the present application means that specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The appearance of this phrase at various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described in the present application can be combined with other embodiments.

[0078] According to the above, some of the text controls used in the electronic device are text controls provided by the operating system, such as EditText and TextView, while there are also applications in the electronic device that use custom text controls, such as some professional text editing applications. For the text controls provided by the operating system, the electronic device can determine their positions based on the capabilities of the operating system itself and call the input method to support the input of the stylus on these text controls. For the custom text controls of the application (which belong to the text controls not provided by the operating system and can also be understood as the text editors developed by the application itself), the electronic device cannot determine their positions based on the capabilities of the operating system itself, nor can it call the input method to achieve the input of the stylus on these text controls.

[0079] Based on the above, embodiments of the present application provide a text editing method and related devices. According to this text editing method, when the input event acting on the application using the custom text control is a stylus input event and the input position corresponding to this input event is within the effective starting area, the electronic device can create a transparent layer on the user interface corresponding to this application, transmit the recognition point information (text recognition point information or pen gesture recognition point information) corresponding to the input event detected by the electronic device acting on the transparent layer to the recognition engine, and identify the pen gesture or handwritten text through the recognition engine, and then transmit the recognition result to this application for display. Through this method, the recognition of converting handwritten content to text and pen gestures can be achieved on the application using the custom text control, expanding the usage scenarios of the stylus, enabling users to use the stylus for flexible text editing, and improving the user experience.

[0080] First, some text editing scenarios provided by the present application will be introduced below.

[0081] For ease of description, the application using the custom text control in the present application is denoted as Application 1. That is to say, the text control in Application 1 is a custom text control by Application 1, that is, a self-developed text editor, rather than a text control provided by the operating system of the electronic device. In some embodiments of the present application, Application 1 is an application program for text editing on electronic devices such as mobile phones and tablet computers.

[0082] For ease of description, the user operation performed by the stylus in the present application is denoted as a stylus input operation. The stylus input operation may specifically include a stylus starting operation.

[0083] 1. Start the application with the custom text control and display the custom text control

[0084] A user can trigger an electronic device to start Application 1 by means of touch, voice control, gesture, etc. (for example, clicking on the application icon corresponding to Application 1, verbally instructing to start Application 1, gesturing to start Application 1). Correspondingly, the electronic device can detect the user operation. In response to the user operation, the electronic device can start Application 1 and display the main interface of Application 1. It is understandable that the main interface of Application 1 is the user interface that is default displayed after starting Application 1. Further, the user can trigger the electronic device to display a custom text control in Application 1.

[0085] Exemplarily, please refer to Figure 1A , Figure 1A which is a schematic diagram of a user interface with application icons provided by an embodiment of the present application. As Figure 1A shown, the user interface 101 displays a page with application icons. The page may include multiple application icons (for example, weather application icon, calendar application icon, email application icon, settings application icon, app store application icon, photo album application icon, Application 1 icon 1011, etc.). Among them, the Application 1 icon 101 is the icon of Application 1. The Application 1 icon 1011 can be used to trigger the start of Application 1. A page indicator may also be displayed below the above-mentioned multiple application icons to indicate the positional relationship between the currently displayed page and other pages. In some embodiments of the present application, there may be multiple tray icons (for example, camera application icon, browser application icon, messages application icon, dialer application icon) below the page indicator. The tray icons can remain displayed when the page is switched.

[0086] Exemplarily, the electronic device can detect a user operation on the Application 1 icon 1011. In response to the user operation, the electronic device can start Application 1 and display the user interface 102 as Figure 1B shown. The user interface 102 is the main interface of Application 1. The user interface 102 can display files that the user has recently opened. For example, Document 1, Document 2, Document 3, Presentation Document 1, and Workbook 1. Among them, the control corresponding to Document 1 is Control 1021, the control corresponding to Presentation Document 1 is Control 1022, and the control corresponding to Workbook 1 is Control 1023. The user interface 102 may also include Control 1024. Control 1024 can be used to create a new file.

[0087] For ease of description, in this application, the file types corresponding to Document 1, Document 2, and Document 3 are denoted as text documents (e.g., Word), the file type corresponding to Presentation Document 1 is denoted as a presentation document (e.g., PPT), and the file type corresponding to Workbook 1 is denoted as a table (e.g., Excel). It can be understood that Application 1 can also create and edit other types of files (e.g., independent format documents, etc.), and this application places no restrictions on this. Among them, compared with text documents, independent format documents (e.g., PDF) are not easily editable and better retain the file format.

[0088] 2. Display the custom text control

[0089] After the electronic device starts Application 1, the user can trigger the electronic device to display the custom text control in Application 1.

[0090] Exemplarily, the electronic device can detect a user operation acting on the control 1021 included in the user interface 102. In response to this user operation, the electronic device can open Document 1 and display the Figure 1C shown user interface 103. The user interface 103 may include a display area 1031. The display area 1031 can be understood as the display area corresponding to the custom text control in Document 1. The display area 1031 can be used to display the text content in Document 1 (such as Figure 1C shown partial content of the work "A Short Day in Sorrow" by the Tang Dynasty poet Li He). The user interface 103 may further include an edit control 1032, a share control, a search control, and a print control. Among them, the edit control is used to trigger entering the edit mode of Document 1, the share control is used to share Document 1 with friends or other devices, the search control is used to search for the content in Document 1, and the print control is used to trigger printing Document 1.

[0091] It can be understood that when the electronic device displays the user interface 103, Document 1 is in the browsing mode (which can also be called the reading mode). In this case, the user can only view Document 1 and cannot edit Document 1.

[0092] 3. Recognize the input content of the stylus pen for the custom text control

[0093] In some embodiments of this application, only when the file opened in Application 1 is in the edit mode can the user input through the stylus pen for the custom text control in Application 1, while when the file opened in Application 1 is in the browsing mode, the user cannot input through the stylus pen for the custom text control in Application 1.

[0094] In a possible implementation, after the user triggers the electronic device to open Document 1, Document 1 is in the browsing mode by default. In this case, the user needs to trigger the electronic device to switch the mode of Document 1 to the editing mode. Exemplarily, the electronic device can detect a user operation acting on the editing control 1032 included in the user interface 103. In response to this user operation, the electronic device can switch Document 1 from the browsing mode to the editing mode and display the user interface 104 as shown in Figure 1D In this case, the user can edit Document 1.

[0095] In another possible implementation, after the user triggers the electronic device to open Document 1, Document 1 is in the editing mode by default, rather than the Figure 1C browsing mode shown. In this case, after the user triggers the electronic device to open Document 1, the user can directly edit Document 1 (for example, input text in Document 1 through the keyboard, etc.) without triggering Document 1 to enter the editing mode.

[0096] It can be understood that when Document 1 is in the editing mode, the user can input on the electronic device with a stylus. Correspondingly, the electronic device can recognize the content input by the stylus as text or an editing method. Specifically, the electronic device can detect the position of the custom text control in Document 1 through Application 1, determine the effective starting area based on the position of the custom text control, and determine whether the stylus input operation acts within the effective starting area. If the stylus input operation acts within the effective starting area, the electronic device can determine that the generated corresponding stylus input event is an input event for the custom text control. The electronic device can continue to detect the stylus input operation, generate a corresponding stylus input event, and recognize the content of the stylus input event corresponding to the stylus input (which can be simply referred to as the handwritten content or also as the handwritten text) to obtain the corresponding text or editing method. In the case of obtaining text, the electronic device can add the text to Document 1. In the case of obtaining an editing method, the electronic device can edit the text in the custom text control included in Document 1 based on the editing method.

[0097] In some embodiments of the present application, the stylus input operation may be a starting action of the stylus falling on the display screen. In this case, the stylus input operation acts within the effective starting area, which can be understood as: the starting position of the stylus falling on the display screen is within the effective starting area. In some other embodiments of the present application, the stylus input operation may be that the stylus is hovering above the display screen, and the distance between the stylus tip and the display screen is less than D1. Among them, the stylus tip can be abstracted as a point, and the display screen can be abstracted as a surface. The distance between the stylus tip and the display screen refers to the distance from the point abstracted by the stylus tip to the plane abstracted by the display screen, that is, the minimum length from the point abstracted by the stylus tip to a point in the plane abstracted by the display screen, or the vertical distance from the point abstracted by the stylus tip to the plane abstracted by the display screen, which can also be understood as the distance between the point abstracted by the stylus tip and its projection point on the plane abstracted by the display screen. In this case, the stylus input operation acts within the effective starting area, which can be understood as: the projection point of the point abstracted by the stylus tip on the plane abstracted by the display screen is within the effective starting area. It is understandable that the specific value of D1 can be set according to actual needs, and the present application does not limit this. For example, D1 can be 1 cm.

[0098] In some embodiments of the present application, after the electronic device determines that the stylus input event it generates is an input event for a custom text control, it can create a writing layer and provide the user with the ability to write and draw through the writing layer. In this case, the electronic device can display the content that the user subsequently writes and draws on the writing layer with the stylus (i.e., the stylus input content) and recognize the content.

[0099] For example, the electronic device may display Figure 1D In the user interface 104 shown, at this time, the document 1 is in the editing mode, and the user can input on the electronic device through the stylus. Figure 1D As shown, the starting position of the stylus on the display screen is within the display area 1031. The electronic device can determine that the starting position is within the valid starting area, and then the electronic device can create a writing layer. The input made by the user through the stylus on the electronic device is actually displayed through the writing layer, that is, the electronic device can display the content written by the user through the stylus on the writing layer. The user can write the two words "so" on the writing layer with the stylus. Correspondingly, Figure 1E In the user interface 105 shown, the electronic device can display the two characters "so" written by the user (i.e., the handwriting input content). In addition, the electronic device can also recognize the handwriting input content on the writing layer as corresponding text, such as recognizing the content written by the user on the writing layer as the two characters "so", and adding the text to the document 1 (e.g.,Figure 1F as shown. During this process, the user can continue to write on the display screen of the electronic device with a stylus. For example, Figure 1F as shown, the user can continue to write the two characters "old man" on the writing layer with a stylus. Correspondingly, the electronic device can display a user interface 106 as shown in Figure 1F Figure. Similarly, the electronic device can continue to recognize the writing content on the writing layer, obtain the corresponding text. For example, the content that the user continues to write on the writing layer is recognized as the two characters "old man", and this text is added to Document 1. In this case, the electronic device can display a user interface 107 as shown in Figure 1G Figure.

[0100] In some embodiments of the present application, after Document 1 enters the editing mode, the electronic device can display an editing toolbar. Exemplarily, the user interfaces 104, 105, 106, and 107 can also include an editing toolbar 1041. The editing toolbar 1041 can include a finish editing control, a previous step control, a next step control, a font style control, a paragraph style control, a highlight control, and an insert control. Among them, the finish editing control is used to trigger the switching of Document 1 from the editing mode to the browsing mode. The previous step control and the next step control are respectively used to trigger the cancellation of the previous input and the restoration of the previous input. The font style control and the paragraph style control are respectively used to adjust the styles of the font and the paragraph. The highlight control is used to highlight specific content (for example, highlighting text in another color). The insert control is used to insert pictures, text boxes, tables, formulas, etc.

[0101] Exemplarily, the user can draw a broken line on the two characters "therefore" in Document 1 with a stylus. Correspondingly, the electronic device can display a user interface 108 as shown in Figure 1H Figure. During the process that the user draws a broken line on "therefore" with a stylus, the electronic device can detect and recognize the writing content on the writing layer, determine that the editing method corresponding to the writing content is deletion, and determine that the deletion is for the two characters "therefore" according to the position of the writing content. Further, the electronic device can delete "therefore" in the custom text control included in Document 1 and display a user interface 109 as shown in Figure 1I Figure.

[0102] In some embodiments of the present application, whether the file opened in Application 1 is in the browsing mode or the editing mode, the user can input to the custom text control in Application 1 with a stylus.

[0103] In a possible implementation, after the user triggers the electronic device to open Document 1, Document 1 is default in the browsing mode. In this case, the user can directly write on the display screen of the electronic device with a stylus. Once the electronic device detects the writing and drawing operation of the stylus, the mode of Document 1 can be switched to the editing mode. Subsequently, the user can input through the stylus for the custom text control in Document 1 in the editing mode.

[0104] Exemplarily, when the electronic device displays the user interface 103, once the electronic device detects the stylus input operation on the display area 1031 in the user interface 103, the electronic device can switch the mode of Document 1 to the editing mode and display the user interface 104 as Figure 1D shown.

[0105] It can be understood that when Document 1 is in the browsing mode, the user can input on the electronic device with a stylus. Correspondingly, the electronic device can switch the mode of Document 1 to the editing mode and recognize the stylus input content as text or an editing method. Specifically, the electronic device can detect the position of the custom text control in Document 1 through Application 1, determine the effective starting area based on the position of the custom text control, and determine whether the stylus input operation acts within the effective starting area. If the stylus input operation acts within the effective starting area, the electronic device can determine that the generated corresponding stylus input event is an input for the custom text control. The electronic device can switch the mode of Document 1 from the browsing mode to the editing mode, continue to receive the stylus input operation, generate the corresponding stylus input event, recognize the stylus input content corresponding to the stylus input event, and obtain the corresponding text or editing method. In the case of obtaining text, the electronic device can add the text to Document 1. In the case of obtaining an editing method, the electronic device can edit the text in the custom text control included in Document 1 based on the editing method.

[0106] Similarly, in some embodiments of the present application, after the electronic device determines that the generated stylus input event is an input for the custom text control and switches the mode of Document 1 to the editing mode, it can create a writing layer and provide the user with the ability to write and draw through this writing layer. In this case, the electronic device can display the content written and drawn by the user on the writing layer later (i.e., the stylus input content) and recognize this content.

[0107] It can be understood that the size and position of the writing layer created by the electronic device are determined based on the size and position of the user interface corresponding to Application 1. In some embodiments of the present application, the electronic device can display the user interfaces of Application 1 and other applications in split screen. In this case, the writing layer created by the electronic device is within the range of the user interface corresponding to Application 1, its size is not larger than the size of the user interface corresponding to Application 1, and its position does not exceed the position of the user interface corresponding to Application 1 on the display screen.

[0108] Exemplarily, as Figure 1J shown, the electronic device can display the user interface corresponding to Application 1 (i.e., User Interface 111) and the user interface corresponding to the shopping application (i.e., User Interface 112) in split screen. The user can input on the electronic device with a stylus. As Figure 1J shown, the electronic device only displays the writing content on User Interface 111, but does not display the writing content on User Interface 112. That is to say, the writing layer created by the electronic device does not exceed the area where User Interface 111 is located.

[0109] In some embodiments of the present application, after the electronic device recognizes the content input by the stylus as corresponding text, it can also obtain associated words based on the recognized text and display them on the display screen for the user to select.

[0110] Exemplarily, as Figure 1F shown, after the user writes the two characters "old man" on the electronic device with a stylus, the electronic device can not only recognize the content input by the stylus detected, to obtain the corresponding text, that is, "old man", but also obtain associated words based on the recognized text. As Figure 1K shown, after the electronic device obtains the associated words, it can display User Interface 113. User Interface 113 may include a display area 1131. The display area 1131 is used to display the associated words of "old man", such as "is old", "let it be", "immortal".

[0111] It should be noted that Figures 1D - 1J the user operation performed by the stylus shown is the stylus input operation.

[0112] Next, the software structure of the electronic device involved in the embodiments of the present application will be introduced.

[0113] The operating system of an electronic device may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of this application, taking the Android operating system (which can be simply referred to as the Android system) with a layered architecture as an example, the software structure of the electronic device is exemplarily described. It should be noted that although the embodiments of this application are described by taking the Android system as an example, the basic principles thereof are equally applicable to electronic devices based on operating systems such as iOS or Windows.

[0114] Figure 2 This is a schematic diagram of the software structure of an electronic device provided in the embodiments of this application.

[0115] The software structure of the electronic device adopts a layered architecture, that is, the software is divided into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. Taking the Android system running on the AP as an example, in some embodiments of this application, the software structure of the Android system is divided into five layers, from top to bottom are the application layer, the application framework layer (Framework), the Android runtime and system libraries, the hardware abstraction layer (HAL), and the system kernel layer (Kernel).

[0116] Among them, the application layer may include a series of application packages. The application packages may include applications such as a camera, a gallery, a calendar, a call, a map, WLAN, Bluetooth, music, video, and short message. The application layer may also include the System User Interface (SystemUI). The SystemUI is used to display the interface of the electronic device, such as displaying the lock screen interface, displaying the signal icon corresponding to the SIM card, and displaying the call interface. The application layer may also include Application 1. Application 1 may include an identification interface. The identification interface is used to call the identification engine of the application framework layer. In some embodiments of this application, Application 1 may also include a drawing module. The drawing module may be used to create a writing layer for the user to write and draw. It can be understood that Application 1 includes a custom text control, and its related description can refer to the above, and this application will not elaborate here.

[0117] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer.

[0118] The application framework layer may include an identification engine. The identification engine can be used to identify the content input by a stylus to obtain corresponding text or editing methods. In some embodiments of the present application, the identification engine may include a pen gesture identification engine and a text identification engine. In some embodiments of the present application, the identification engine may be a Software Development Kit (SDK).

[0119] In some embodiments of the present application, the application framework layer may include a drawing engine. The drawing engine can be used to create a writing layer and provide writing and drawing tools on the writing layer.

[0120] The application framework layer may include some predefined functions. For example, the application framework layer may include a window manager, an input manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. Among them, the view system may include visible controls. For example, a control for displaying text, a control for displaying pictures, etc. The view system can be used to build an application. The display interface can be composed of one or more views. For example, a display interface including a short message notification icon may include a view for displaying text and a view for displaying pictures.

[0121] The window manager is used to implement the management (such as adding, removing, updating, etc.) and drawing of system and application windows.

[0122] The main function of the input manager is to monitor input devices (such as a stylus, keyboard, mouse, etc.) and timely transfer the generated input events (such as clicking on the screen, pressing a key, swiping, etc.) to the operating system or the application, and then the operating system or the application performs corresponding processing. For example, the user touches a control of an activity in an application. The input manager can convert the user's touch into a touch event, transfer it to the application, and then continue to transfer it to the control, and the control performs corresponding processing.

[0123] The input manager can be understood as a running container for an Input Reader and an Input Dispatcher. It creates two threads to respectively carry the running of the Input Reader and the Input Dispatcher. The input manager may include an Input Reader and an Input Dispatcher. The main function of the InputReader is to read events. The main function of the Input Dispatcher is to distribute events.

[0124] It should be noted that the input manager is only a part of the entire input management service. The entire input management service not only involves the application framework layer (also known as the Java framework layer), but also involves the native layer (also known as the Native framework layer). The main work of the input management service in the application framework layer is to provide implementation methods for the Reader Policy and the Dispatcher Policy, and to cooperate with other system services (such as the window management service, etc.). However, the native layer is at a lower level compared to the application framework layer. The application framework layer generally includes some system services, while the native layer generally includes some native services and some link libraries, etc. For example, if complex operations need to be performed, if implemented through the application framework layer, the efficiency may be very low, but if implemented through the native layer and then communicate with the application framework layer (i.e., the JNI mechanism in Android), the efficiency will be greatly improved. Another example is that when the electronic device runs, it needs to interact with the underlying hardware drivers, and this process also needs to go through the native layer. The native layer can include EventHub. EventHub is created when the InputManager in the native layer is created. The main role of EventHub is to use the inotify and epoll mechanisms of Linux to monitor device events. The device events mentioned here include device plugging and unplugging, as well as various touch and button events, etc. EventHub mainly faces the device nodes in the / dev / input directory. For the input events on the device nodes, they can be monitored and obtained through EventHub's getEvents.

[0125] The Runtime is responsible for the scheduling and management of the system. The Runtime includes the core libraries and the virtual machine. Among them, the core libraries contain two parts: one part is the functional functions that the programming language (such as the Java language) needs to call, and the other part is the core libraries of the system. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the programming files (such as Java files) of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0126] The system libraries can include multiple functional modules. For example, the Surface Manager, Media Libraries, 3D graphics processing libraries (such as OpenGL ES), and 2D graphics engines (such as SGL), etc. The specific meanings and functions of these functional modules can be referred to the relevant technical documents and will not be elaborated here.

[0127] The Hardware Abstraction Layer (HAL) is an interface layer located between the operating system kernel and the upper-layer software, and its purpose is to abstract the hardware. The HAL is an abstract interface for device kernel drivers and is used to implement application programming interfaces that provide access to underlying devices to a higher-level Java API framework. The HAL can provide a standard interface to display the hardware functions of the device to a higher-level Java API framework. The HAL contains multiple library modules (for example, library modules corresponding to cameras, audio, Bluetooth, and sensors). When the system framework layer API requests access to the hardware of a portable device, the operating system will load the library module for this hardware component.

[0128] The kernel layer is the foundation of the Android system. The kernel layer is responsible for functions such as hardware driver programs, networking, power supply, system security, and memory management. The kernel layer is an intermediate layer between hardware and software, and its role is to pass the requests of application programs to the hardware. The kernel layer may include audio drivers, display drivers, camera drivers, and sensor drivers.

[0129] It should be noted that Figure 2 the schematic diagram of the software structure of the electronic device shown is only an example and does not limit the specific module division in different layers of the Android system. Specifically, reference can be made to the introduction of the Android system software structure in conventional technologies. In addition, the method provided in this application can also be implemented based on other operating systems, and specific examples are not given one by one in this application.

[0130] The following introduces a text editing method provided by an embodiment of this application.

[0131] Please refer to Figure 3 , Figure 3 which is a flowchart of a text editing method provided by an embodiment of this application. This text editing method can be applied to an electronic device, and this text editing method may include but is not limited to the following steps:

[0132] S1: Detect a stylus input operation on the user interface of Application 1.

[0133] The electronic device can display the user interface of Application 1. The user can input on Application 1 with a stylus. The electronic device can detect the corresponding stylus input operation and generate a corresponding stylus input event.

[0134] In some embodiments of the present application, the electronic device can detect a change in the state of the display screen (e.g., a change in the capacitance of the display screen) caused by the stylus touching the display screen, and convert the change in the state of the display screen into an input event. It can also be understood that the change in the state of the display screen is encapsulated into an object of a more advanced class. In some embodiments of the present application, the input event may refer to an InputEvent object. InputEvent is a Java class that can include multiple subclasses, such as motion events (MotionEvent) and key events (KeyEvent). Among them, MotionEvent can be used not only to represent touch events, that is, input events generated when the user operates on the device screen (i.e., the display screen) through the skin or a stylus, but also to represent hover events, that is, input events generated when the user hovers or slides the mouse on the device screen, as well as input events generated when the user hovers the stylus on the device screen, and can also be used to represent scroll events, that is, input events generated when the user scrolls the mouse wheel. It should be noted that since the types of devices that can run the operating system of the electronic device (e.g., the Android system) are very rich, the corresponding input sources are naturally very rich, and the specific types of InputEvent are also very rich, not limited to the above-mentioned motion events and key events.

[0135] It should be noted that, in some embodiments of the present application, the stylus input operation on the user interface of Application 1 refers to the stylus starting operation, that is, the user places the stylus on the display screen of the electronic device (before starting to write or draw). Exemplarily, as Figure 1D shown, the stylus input operation on the user interface of Application 1 can be that the user places the stylus on the user interface 104.

[0136] In some other embodiments of the present application, the electronic device can detect that the stylus hovers above the display screen and can also generate a corresponding input event, that is, encapsulate to obtain an object of a more advanced class, such as a hover event.

[0137] It can be understood that the electronic device can display the user interface of Application 1 full-screen (as Figure 1C shown), or display the user interface of Application 1 in split-screen (as Figure 1J shown), and the present application does not limit this.

[0138] S2: In response to the stylus input operation on the user interface of Application 1, when the stylus input operation on the user interface of Application 1 acts within the effective starting area, a writing layer is created and the recognition engine is enabled. The effective starting area is determined based on the position and size of the custom text control in Application 1.

[0139] After the electronic device detects a stylus input operation on the user interface of Application 1, in response to the stylus input operation, the electronic device can determine whether the stylus input operation acts within the effective starting area and generate a corresponding stylus input event. If the stylus input operation acts within the effective starting area, the electronic device can create a transparent layer, i.e., a writing layer, on the basis of the originally displayed user interface of Application 1 and enable the recognition engine.

[0140] Among them, the electronic device can determine whether the stylus input operation acts within the effective starting area, which can also be understood as: the electronic device determines whether the input position corresponding to the generated stylus input event is within the effective starting area.

[0141] It can be understood that the effective starting area is determined based on the position and size of the custom text control in Application 1. The effective starting area includes the area where the custom text control in Application 1 is located. In some embodiments of the present application, the effective starting area is larger than the area where the custom text control in Application 1 is located.

[0142] Exemplarily, as Figure 4A shown, the electronic device can display the user interface of Application 1, and the user interface of Application 1 can include text, pictures, and other elements (for example, tables, hyperlinks, etc.). Among them, the control corresponding to the text can be understood as the custom text control of Application 1, and the controls corresponding to the pictures and other elements can be understood as other types of controls. The user can directly input using a stylus on the custom text control in the user interface of Application 1. Specifically, when the stylus touches the display screen of the electronic device, the electronic device can detect the corresponding stylus input operation and determine the position where the stylus input operation acts (which can also be understood as the input position corresponding to the generated stylus input event), as Figure 4A shown, the input position corresponding to the stylus input event generated when the stylus touches down (or starts writing) is within the effective starting area. In this case, the electronic device can create a writing layer. As Figure 4BAs shown, the writing layer is a transparent layer covering the user interface of application 1. After the electronic device creates the writing layer, the input made by the user through the stylus on the electronic device actually falls on the writing layer, and the electronic device will display the traces of the user's writing and drawing on the writing layer through the stylus, that is, the user can see the traces of writing and drawing on the writing layer, and the traces will not affect the display of the user interface of application 1. Figure 4B As shown, the user writes the word "所" on the writing layer with a stylus, and the electronic device can display the writing content, but it does not affect the display of the user interface of application 1, and the user interface of application 1 still displays as shown. Figure 4A The text, images, and other elements shown.

[0143] In some embodiments of the present application, the area occupied by the writing layer on the display screen of the electronic device does not exceed the area occupied by the user interface of application 1 on the display screen.

[0144] For example, Figure 5A As shown, the electronic device can display the user interface of application 1 and the user interface of other applications in a split screen. The user interface of application 1 may include text, and the user interfaces of other applications may include pictures and other elements (for example, tables, hyperlinks, etc.). The control corresponding to the text can be understood as the custom text control of application 1. The user can use the stylus to directly input the custom text control in the user interface of application 1. Specifically, the stylus falls on the display screen of the electronic device, and the electronic device can detect the corresponding stylus input operation and determine the position of the stylus input operation (which can also be understood as the input position corresponding to the corresponding generated stylus input event), such as Figure 5A As shown in FIG. 1 , the input position corresponding to the stylus input event generated when the stylus is put down is within the valid starting area. In this case, the electronic device can create a writing layer. Figure 5B As shown, since the user interface of application 1 only occupies the right half of the display screen of the electronic device, the writing layer created by the electronic device also occupies the right half of the display screen of the electronic device accordingly. After the electronic device creates the writing layer, the stylus input operation acting on the writing layer is the valid input operation for the custom text control in application 1, and the writing and drawing content acting on the non-writing layer area is invalid, and the electronic device cannot display the writing and drawing content.

[0145] It can be understood that when the electronic device simultaneously displays the user interface of Application 1 and the user interfaces of other applications, the electronic device can not only detect the input operation of the stylus on the writing layer, but also detect the user operations on the user interfaces of other applications (such as touch operations, button operations, mouse operations, voice control operations, gesture operations, etc.). Moreover, the input operation of the stylus on the writing layer and the user operations on the user interfaces of other applications do not affect each other.

[0146] Exemplarily, as Figure 5B shown, since the writing layer is not covered on the user interface of other applications, the writing and drawing operations performed by the user on the user interface of other applications with the stylus will not cause the electronic device to display corresponding writing and drawing traces, nor will it affect the custom text control in Application 1. Instead, it may affect the user interface of other applications, such as causing the sliding of the user interface of other applications, or triggering the display of other user interfaces of other applications, or affecting the controls in the user interface of other applications.

[0147] It can be understood that this application does not limit the sequence of creating the writing layer and enabling the recognition engine by the electronic device.

[0148] In some embodiments of this application, in response to the input operation of the stylus on the user interface of Application 1, when the input operation of the stylus on the user interface of Application 1 acts within the effective starting area, the electronic device can detect whether the recognition engine object has been created. If the electronic device has created the recognition engine object, the electronic device can directly enable the corresponding recognition engine. If the electronic device has not created the recognition engine object, the electronic device can first create the recognition engine object and then enable the corresponding recognition engine.

[0149] In some embodiments of the present application, the recognition engine may include a gesture recognition engine and a text recognition engine. The electronic device may determine whether the stylus input operation acting on the user interface of application 1 corresponds to a gesture action or a handwritten text action. In the case where the stylus input operation corresponds to a gesture action, the electronic device may detect whether a recognition engine object has been created. If the electronic device has created a recognition engine object, the electronic device may directly start the gesture recognition engine. If the electronic device has not created a recognition engine object, the electronic device may first create a recognition engine object, and instruct the recognition engine object to call a gesture recognition method, and then start the gesture recognition engine. Similarly, in the case where the stylus input operation corresponds to a handwritten text action, the electronic device may detect whether a recognition engine object has been created. If the electronic device has created a recognition engine object, the electronic device may directly start the text recognition engine. If the electronic device has not created a recognition engine object, the electronic device may first create a recognition engine object, and instruct the recognition engine object to call a text recognition method, and then start the text recognition engine.

[0150] S3: A stylus input operation acting on the writing layer is detected.

[0151] The user can input on the application 1 through the stylus, and the electronic device can detect the stylus input operation and generate a corresponding stylus input event. The stylus input event can include an input event generated when the pen is put down (i.e., an input event corresponding to the stylus start operation) and an input event generated during subsequent writing and drawing.

[0152] In some embodiments of the present application, the electronic device can create a writing layer based on the handwriting pen starting operation (as shown in step S2). After the electronic device creates the writing layer, the user writes and draws on the writing layer with the stylus (such as the first stroke written on the writing layer after the pen is put down). It should be noted that since the writing layer is a transparent layer, the user cannot perceive the creation of the writing layer. The pen-putting and subsequent writing and drawing actions when the user inputs with the stylus can be coherent, but the electronic device first creates the writing layer after detecting the input event generated when the pen is put down, and the time for creating the writing layer is extremely short, so the subsequent writing and drawing actions naturally fall on the writing layer.

[0153] For example, Figure 1E As shown, the handwriting pen input operation acting on the writing layer may be that the user writes the two words “so” on the user interface 105 with the handwriting pen (actually, writing “so” on the writing layer).

[0154] For example, Figure 1FAs shown, the stylus input operation on the writing layer enables the user to write the two characters "old man" on the user interface 106 with the stylus (actually writing "old man" on the writing layer).

[0155] Exemplarily, as Figure 1J shown, the stylus input operation on the writing layer enables the user to write the two characters "nature" on the user interface 111 with the stylus (actually writing "nature" on the writing layer).

[0156] In some other embodiments of the present application, the electronic device can create a writing layer based on the operation of the stylus hovering above the display screen. After the electronic device creates the writing layer, the user writes and draws on the writing layer with the stylus (such as the first stroke written on the writing layer after putting the pen down).

[0157] S4: In response to the stylus input operation on the writing layer, generate a corresponding stylus input event.

[0158] It can be understood that after the electronic device detects the stylus input operation on the writing layer, in response to the stylus input operation, a corresponding stylus input event is generated.

[0159] In some embodiments of the present application, the stylus input event may include an event type, location information (e.g., X / Y coordinates), etc. Among them, the location information can be used to represent the contact position of the stylus with the display screen or the position of the projection point on the display screen when hovering.

[0160] In some embodiments of the present application, the event type may include a pen-down event, a movement event, and a pen-up event. Among them, the pen-down event refers to the event generated when the stylus touches the screen. In the case of multi-touch, this type of event is only generated when first touching the screen, and will not be generated when touching the screen later. It is the start of a touch screen event sequence. It can be understood that when the finger or stylus no longer touches the screen and then touches the screen again, a pen-down event will be generated, which indicates the start of a new touch screen event sequence. The movement event refers to the event generated when the stylus slides on the screen. In the case of multi-touch, this type of event is generated when not first touching the screen. The pen-up event refers to the event generated when the stylus (the stylus that generates the pen-down event) leaves the screen, which is the end of a touch screen event sequence.

[0161] It is understandable that the user can lift the pen after continuously writing and drawing on the display screen with a stylus. During this process, the electronic device can continuously generate input events, and these continuously generated input events can be generally understood as a touch screen event sequence as a whole. This touch screen event sequence can include an input event with an event type of pen-down event, multiple input events with an event type of move event, and an input event with an event type of pen-up event.

[0162] It is understandable that the user can also lift the pen after continuously writing on the display screen with a stylus, then put the pen down again, continuously write and then lift the pen, and can continue to repeat the writing and drawing actions of putting the pen down again, continuously writing and then lifting the pen. During this process, the electronic device can continuously generate input events, and these continuously generated input events can also be generally understood as a touch screen event sequence as a whole. This touch screen event sequence can include multiple input events with an event type of pen-down event, multiple input events with an event type of move event, and multiple input events with an event type of pen-up event.

[0163] In some embodiments of the present application, the pen-down event can be ACTION_DOWN, the move event can be ACTION_MOVE, and the pen-up event can be ACTION_UP. The specific meanings of ACTION_DOWN, ACTION_MOVE, and ACTION_UP can be referred to relevant technical documents, and the present application will not elaborate here.

[0164] It is understandable that the stylus input event can also include other event types, and the present application does not limit this. For example, ACTION_POINTER_UP, ACTION_POINTER_DOWN, ACTION_CANCEL, etc. The specific meanings of these event types can be referred to relevant technical documents, and the present application will not elaborate here.

[0165] In some embodiments of the present application, the stylus input event can also include a timestamp and / or an input device type (i.e., the stylus). The timestamp can be used to represent the generation time of the stylus input event. The input device types included in the stylus input event are all styluses, which can be represented by characters, numbers, etc.

[0166] S5: Identify the corresponding stylus input event through an identification engine to obtain an identification result.

[0167] It is understandable that the identification result can be text, or can also be an editing method (for example, select, delete, split, combine, etc.).

[0168] In some embodiments of the present application, the electronic device can identify the corresponding stylus input event through an identification engine to determine whether the corresponding stylus input operation is a gesture action or a handwritten text action. If the electronic device determines through the identification engine that the corresponding stylus input operation is a gesture action, the electronic device can use the gesture recognition ability of the identification engine to identify the corresponding stylus input event as a corresponding gesture, and thus determine the corresponding editing method based on the gesture (e.g., a horizontal line, a wavy line, a broken line, etc.). Similarly, if the electronic device determines through the identification engine that the corresponding stylus input operation is a handwritten text action, the electronic device can use the text recognition ability of the identification engine to identify the corresponding stylus input event as corresponding text.

[0169] It can be understood that the present application does not limit the correspondence between gestures and editing methods. Exemplarily, the editing method corresponding to a horizontal line is to select part / all of the text, the editing method corresponding to a wavy line or a horizontal broken line is to delete part / all of the text, the editing method corresponding to drawing a vertical line downwards is to split the text, and the editing method corresponding to drawing a vertical line upwards is to combine the text.

[0170] In some embodiments of the present application, the identification engine can include a gesture recognition engine and a text recognition engine. According to the above, after the electronic device detects a stylus input operation on the user interface of Application 1, it can determine whether the stylus input operation on the user interface of Application 1 corresponds to a gesture action or a handwritten text action, and activate the corresponding identification engine after the determination. If the electronic device has determined that the stylus input operation on the user interface of Application 1 corresponds to a gesture action, after the electronic device obtains the generated corresponding stylus input event, it can identify the generated corresponding stylus input event through the gesture recognition engine to obtain an editing method. Similarly, if the electronic device has determined that the stylus input operation on the user interface of Application 1 corresponds to a handwritten text action, after the electronic device obtains the generated corresponding stylus input event, it can identify the corresponding stylus input event through the text recognition engine to obtain the corresponding text.

[0171] In some embodiments of the present application, the electronic device can filter out a part of the stylus input events from the above-generated corresponding stylus input events, and identify the filtered stylus input events through the identification engine to obtain an identification result.

[0172] S6: Edit the text in the custom text control based on the identification result.

[0173] In the case where the recognition result is text, the electronic device may add the recognized text to the text included in the custom text control. In the case where the recognition result is an editing method, the electronic device may edit the corresponding text in the custom text control in Application 1 according to the recognized editing method.

[0174] It can be understood that in the case where the recognition result is an editing method, the electronic device may determine, from all the text included in the custom text control in Application 1, the text whose coordinates are the same as the position information included in the corresponding stylus input event, and edit the text based on the recognized editing method.

[0175] The following is based on Figure 6A and Figure 6B to introduce a specific implementation manner of the above embodiments.

[0176] First, please refer to Figure 6A , Figure 6A which is a flowchart of another text editing method provided by an embodiment of the present application. The text editing method may include but is not limited to the following steps:

[0177] S101: Application 1 detects an input event 1 for the user interface of Application 1.

[0178] The user may input to the user interface of Application 1 in different ways. For example, the user may click on the custom text control in the user interface of Application 1 with a mouse. For example, the user may touch the custom text control in the user interface of Application 1 with a finger. For another example, the user may click on the custom text control in the user interface of Application 1 with a stylus. For another example, the user may input to the custom text control in the user interface of Application 1 by tapping the keyboard.

[0179] It can be understood that after the electronic device detects the user's input operation on the user interface of Application 1, it may generate a corresponding raw input event (RawEvent). The kernel layer of the electronic device may collect the raw input event and transmit it to the input device node. Modules in the upper layer of the electronic device relative to the kernel layer (for example, EventHub in the Native layer) may listen to the input device node, obtain the raw input event, and process (for example, preprocess and classify) the raw input event to obtain a processed input event (for example, an InputEvent object), such as input event 1. Modules in the even higher layer of the electronic device (for example, InputReader and InputDispatcher in the application framework layer) may read and distribute input event 1 and finally transmit it to Application 1. In this way, it can be understood that Application 1 detects an input event 1 for the user interface of Application 1.

[0180] S102: Application 1 determines whether input event 1 is a stylus input event.

[0181] The input event may carry information about the input device type (e.g., finger, mouse, stylus, etc.). After detecting input event 1, Application 1 can determine whether the input device is a stylus based on the input device type information carried by input event 1. If the input device is a stylus, then input event 1 is a stylus input event; otherwise, input event 1 is not a stylus input event. When input event 1 is a stylus input event, Application 1 can continue to execute step S103, that is, determine whether the input position corresponding to input event 1 is within the valid starting area.

[0182] It can be understood that when the electronic device processes the original input event, different types of InputEvent objects can be generated for different types of input events. For example, when the electronic device detects a user operation on the user interface of Application 1 by a stylus, the subsequent generated InputEvent object is specifically a MotionEvent. In the MotionEvent class, the action that generates this event is called motion, and the entity that generates this action (e.g., finger, mouse, stylus, etc.) is called pointer. A MotionEvent object can contain one or more pointers, and each pointer contains attributes such as identification (id), index, position, size, direction, etc. Pointer can be understood as the input device mentioned above. The input device type mentioned above can also be represented by the id contained in the pointer.

[0183] It should be noted that when input event 1 is a stylus input event, the user operation corresponding to input event 1 can be understood as the stylus starting operation mentioned above.

[0184] S103: Application 1 determines whether the input position corresponding to input event 1 is within the valid starting area.

[0185] When input event 1 is a stylus input event, Application 1 can determine whether the input position corresponding to input event 1 is within the valid starting area. According to the above, the valid starting area is determined based on the position and size of the custom text control in Application 1.

[0186] It can be understood that input event 1 may carry position information. Application 1 can determine the input position corresponding to input event 1 based on the position information carried by input event 1, and determine whether the input position corresponding to input event 1 is within the valid starting area. If the input position corresponding to input event 1 is within the valid starting area, Application 1 can continue to execute step S104, that is, create a writing layer.

[0187] If there are multiple custom text controls, Application 1 can determine whether the input position corresponding to Input Event 1 is within the valid starting area corresponding to the multiple custom text controls. When the input position corresponding to Input Event 1 is within the valid starting area corresponding to one of the custom text controls, Application 1 can determine that Input Event 1 is an input event for this custom text control, that is to say, the user wants to edit the text in this custom text control with a stylus.

[0188] It can be understood that this application does not limit the order of Application 1 executing Step S102 and Step S103.

[0189] S104: Application 1 creates a writing layer.

[0190] When Input Event 1 is a stylus input event and the input position corresponding to Input Event 1 is within the valid starting area, Application 1 can create a writing layer. In some embodiments of this application, Application 1 can create a writing layer through a drawing module. In some other embodiments of this application, Application 1 can create the writing layer of Application 1 by calling a drawing engine. It can be understood that the relevant description of the writing layer can refer to the above, and this application will not elaborate here.

[0191] S105: When the input position corresponding to Input Event 1 is within the text area, Application 1 calls the recognition interface to create a recognition engine object and instructs the recognition engine object to call the pen gesture recognition method.

[0192] When Input Event 1 is a stylus input event and the input position corresponding to Input Event 1 is within the valid starting area, Application 1 can determine whether the input position corresponding to Input Event 1 is within the text area or within the blank area. That the input position corresponding to Input Event 1 is within the text area can be understood as: Input Event 1 acts on the text included in the custom text control in Application 1. In this case, Application 1 can determine that the user operation corresponding to Input Event 1 (for example, the stylus input operation mentioned above) is a pen gesture action, thereby calling the recognition interface to create a recognition engine object and instructing the recognition engine object to call the pen gesture recognition method. It can be understood that the text area refers to the area where the text in the custom text control in Application 1 is located.

[0193] Exemplarily, as Figure 1H shown, Application 1 can detect that the pen-down position of the stylus is on the text in Document 1. In this case, Application 1 can determine that the input position corresponding to Input Event 1 is within the text area.

[0194] It can be understood that the recognition engine object represents an instance of the pen gesture recognition engine. That the application 1 instructs the recognition engine object to call the pen gesture recognition method can be understood as the application 1 calling the pen gesture recognition engine in the recognition engine, which is specifically used to recognize the pen gesture action as an editing method.

[0195] It should be noted that the application 1 can also determine whether the user operation corresponding to the input event 1 is a pen gesture action in other ways, and the present application does not limit this.

[0196] It can be understood that the present application does not limit the order of execution of step S104 and step S105 by the application 1.

[0197] S106: The application 1 registers to listen for the recognition engine object.

[0198] After the application 1 creates the recognition engine object, it can register to listen for the recognition engine object. After the application 1 registers to listen for the recognition engine object, the recognition engine object can return a registration listening result to the application 1. This registration listening result can be used to indicate whether the application 1 has successfully registered to listen for the recognition engine object.

[0199] Since the application 1 instructs the recognition engine to call the pen gesture recognition method, the listening for this recognition engine can be understood as the listening for the pen gesture recognition engine (as Figure 6A shown). It can be understood that the specific implementation method of registering to listen can refer to relevant technical documents, and the present application does not make specific limitations.

[0200] S107: The application 1 starts the pen gesture recognition engine.

[0201] After the application 1 receives the registration listening result returned by the recognition engine object, it can start the pen gesture recognition engine.

[0202] In some embodiments of the present application, the application 1 has created the recognition engine object. In this case, the application 1 does not need to execute step S105 - step S106, and can directly execute step S107, that is, start the pen gesture recognition engine, when the input position corresponding to the input event 1 is in the text area.

[0203] S108: The application 1 obtains the recognition point information 1 based on the input event 2 acting on the writing layer.

[0204] It is understandable that the user operation corresponding to input event 2 can be understood as the subsequent writing and drawing after the stylus pen touches down, and this writing and drawing is performed on the writing layer created by Application 1. Application 1 can filter the information included in input event 2 (such as event type, position information, timestamp, and input device type) to obtain recognition point information 1. It is understandable that input event 2 can include information related to multiple contact points, and recognition point information 1 can include an information set of some or all of these contact points. It is understandable that the information set of each contact point in recognition point information 1 can include some or all of the information of this contact point included in the corresponding input event, and this application does not limit this. In some embodiments of this application, the information set of each contact point in recognition point information 1 can include the event type and position information (such as X / Y coordinates).

[0205] In some embodiments of this application, input event 2 can be an event sequence composed of a series of input events, which includes input events acting on the writing layer within a period of time (such as 1 second). In some embodiments of this application, input event 2 can be an event sequence composed of multiple movement events and one pen-lifting event. In some embodiments of this application, input event 2 can be an event sequence composed of multiple movement events, one or more pen-touching-down events, and one or more pen-lifting events.

[0206] It should be noted that as long as the user continuously writes and draws on the writing layer with the stylus pen, input events will be continuously generated. Therefore, the process by which Application 1 obtains recognition point information based on input events can be understood as a process of continuously obtaining recognition point information based on the real-time obtained event stream.

[0207] S109: Application 1 sends recognition point information 1 to the pen gesture recognition engine.

[0208] After Application 1 obtains recognition point information 1 based on input event 2 acting on the writing layer, it can send recognition point information 1 to the pen gesture recognition engine.

[0209] Correspondingly, the pen gesture recognition engine can receive the recognition point information 1 sent by Application 1.

[0210] S110: The pen gesture recognition engine obtains pen gesture recognition result 1 based on recognition point information 1.

[0211] After the pen gesture recognition engine receives the recognition point information 1 sent by Application 1, it can obtain pen gesture recognition result 1 based on recognition point information 1. The specific implementation method can refer to the above, and this application will not elaborate here.

[0212] It is understandable that the pen gesture recognition result 1 can be an editing method. Different editing methods can be represented in different forms such as characters, numbers, texts, etc., and the present application does not limit this. Exemplarily, the pen gesture recognition result 1 can be 1, and the represented editing method is deletion.

[0213] In some embodiments of the present application, the pen gesture recognition engine can send the pen gesture recognition result obtained based on the recognition point information received within a period of time (e.g., 0.5 seconds) to the application 1 each time, rather than sending the pen gesture recognition result obtained based on the recognition point information of a single input event to the application 1 each time.

[0214] S111: The pen gesture recognition engine sends the pen gesture recognition result 1 to the application 1.

[0215] After the pen gesture recognition engine obtains the pen gesture recognition result 1 based on the recognition point information 1, it can send the pen gesture recognition result 1 to the application 1.

[0216] Correspondingly, the application 1 can receive the pen gesture recognition result 1 sent by the pen gesture recognition engine.

[0217] S112: The application 1 edits the text included in the custom text control in the application 1 based on the pen gesture recognition result 1.

[0218] The application 1 can determine which text in the custom text control in the application 1 the input event 2 targets based on the location information carried by the input event 2, and after receiving the pen gesture recognition result 1 sent by the pen gesture recognition engine, edit the text based on the pen gesture recognition result 1. The specific implementation method can refer to the above (such as step S6), and the present application will not elaborate here.

[0219] Exemplarily, as Figure 1H shown, the application 1 can determine that the input event 2 targets the two characters "so" included in the custom text control in the application 1 based on the location information carried by the input event 2. After the application 1 receives the pen gesture recognition result 1, it can determine that the editing method is deletion, and the application 1 can delete the two characters "so" included in the custom text control in the application 1. Correspondingly, the user interface of the application 1 can become the user interface 109 as Figure 1I shown.

[0220] In some embodiments of the present application, after the application 1 receives the pen gesture recognition result 1 sent by the pen gesture recognition engine, instead of immediately editing the text included in the custom text control in the application 1 based on the pen gesture recognition result 1, it needs to wait for a period of time (e.g., 1 second). If no new pen gesture recognition result is received during this period, the application 1 can edit the text included in the custom text control in the application 1 based on the pen gesture recognition result 1.

[0221] S113: Application 1 obtains the recognition point information 2 based on the input event 3 acting on the writing layer.

[0222] S114: Application 1 sends the recognition point information 2 to the pen gesture recognition engine.

[0223] It can be understood that the specific implementation manners of steps S113 and S114 can refer to steps S108 and S109, and are not elaborated herein in this application.

[0224] S115: The pen gesture recognition engine adjusts the pen gesture recognition result based on the recognition point information 2 to obtain the pen gesture recognition result 2.

[0225] After receiving the recognition point information 2 sent by Application 1, the pen gesture recognition engine can determine a new pen gesture recognition result based on the recognition point information 2, that is, the pen gesture recognition result 2.

[0226] S116: The pen gesture recognition engine sends the pen gesture recognition result 2 to Application 1.

[0227] After the pen gesture recognition engine adjusts the pen gesture recognition result to obtain the pen gesture recognition result 2, it can send the pen gesture recognition result 2 to Application 1. In this case, Application 1 does not edit the text included in the custom text control in Application 1 based on the pen gesture recognition result 1, but edits the text included in the custom text control in Application 1 based on the pen gesture recognition result 2.

[0228] According to the above, in some embodiments of this application, after receiving the pen gesture recognition result 2, Application 1 can wait for a period of time. If no new pen gesture recognition result is received during this period, Application 1 can edit the text included in the custom text control in Application 1 based on the pen gesture recognition result 2.

[0229] It can be understood that steps S113 - S116 are optional steps.

[0230] S117: Application 1 determines that the duration of not detecting an input event is not less than T1.

[0231] Application 1 can monitor the duration between the current moment and the moment when the last input event was detected (i.e., the duration of not detecting an input event), and determine whether this duration is not less than T1. If this duration is not less than T1, Application 1 can execute steps S118 and S119.

[0232] It can be understood that T1 can be set according to actual needs, and this application does not limit this. For example, T1 can be 5 seconds.

[0233] In some embodiments of the present application, if Application 1 detects a non-stylus input event (i.e., an input event where the input device type is not a stylus), then Application 1 can delete the writing layer and turn off the pen gesture engine.

[0234] In some embodiments of the present application, if Application 1 detects a stylus input event with an event type of pen lift event, the electronic device can turn off the pen gesture recognition engine. In this case, Application 1 can not delete the writing layer, but only hide the writing layer. After detecting a stylus input event with an event type of pen down event, Application 1 can restore the writing layer.

[0235] S118: Application 1 deletes the writing layer.

[0236] After Application 1 determines that the duration of no detected input event is not less than T1, it can delete the writing layer.

[0237] In some embodiments of the present application, under the condition of meeting the preset condition 1, Application 1 can delete the writing layer. Among them, meeting the preset condition 1 can specifically include any one or more of the following: the duration of no detected input event is not less than T11; detecting a non-stylus input event. Of course, meeting the preset condition 1 can also include other contents, and the present application does not limit this.

[0238] It can be understood that T11 can be set according to actual needs, and the present application does not limit this.

[0239] S119: Application 1 turns off the pen gesture recognition engine.

[0240] After Application 1 determines that the duration of no detected input event is not less than T1, it can turn off the pen gesture recognition engine.

[0241] In some embodiments of the present application, under the condition of meeting the preset condition 1, Application 1 can turn off the pen gesture recognition engine. Among them, meeting the preset condition 2 can specifically include any one or more of the following: the duration of no detected input event is not less than T12; detecting a non-stylus input event; detecting a stylus input event with an event type of pen lift event. Of course, meeting the preset condition 2 can also include other contents, and the present application does not limit this.

[0242] It can be understood that T12 can be set according to actual needs, and the present application does not limit this. T11 and T12 can be the same or different.

[0243] It should be noted that the preset condition 1 and the preset condition 2 are not necessarily the same.

[0244] It is understandable that after Application 1 instructs the pen gesture recognition engine to shut down, the pen gesture recognition engine can return a corresponding result to Application 1 to inform Application 1 whether the pen gesture recognition engine has been successfully shut down.

[0245] It is understandable that Figure 6A The recognition point information 1 and recognition point information 2 shown are pen gesture recognition point information.

[0246] It should be noted that in some embodiments of the present application, Application 1 can directly send an input event (for example, input event 2, input event 3, etc.) to the pen gesture recognition engine, and the pen gesture recognition engine can recognize the input event to obtain a corresponding pen gesture recognition result. It is understandable that the input object mentioned here can be an InputEvent object, which carries information related to the contact point between the stylus and the display screen.

[0247] Secondly, please refer to Figure 6B , Figure 6B which is a flowchart of another text editing method provided by an embodiment of the present application. The text editing method may include but is not limited to the following steps:

[0248] S201: Application 1 detects input event 1 for the user interface of Application 1.

[0249] S202: Application 1 determines whether input event 1 is a stylus input event.

[0250] S203: Application 1 determines whether the input position corresponding to input event 1 is within the effective starting stroke area.

[0251] S204: Application 1 creates a writing layer.

[0252] It is understandable that the specific implementation manners of steps S201 - S204 can refer to steps S101 - S104, and the present application will not elaborate here.

[0253] S205: When the input position corresponding to input event 1 is in a blank area, Application 1 calls the recognition interface to create a recognition engine object and instructs the recognition engine object to call the text recognition method.

[0254] When the input event 1 is a stylus input event and the input position corresponding to the input event 1 is within the effective starting area, Application 1 can determine whether the input position corresponding to the input event 1 is in the text area or in the blank area. That the input position corresponding to the input event 1 is in the blank area can be understood as: the input event 1 does not act on the text included in the custom text control in Application 1, but acts on the blank area included in the custom text control in Application 1. In this case, Application 1 can determine that the user operation corresponding to the input event 1 (for example, the stylus input operation mentioned above) is a handwritten text action, thereby calling the recognition interface to create a recognition engine object and instructing the recognition engine object to call the text recognition method.

[0255] Exemplarily, as Figure 1E shown, Application 1 can detect that the pen-down position of the stylus is within the blank area in Document 1. In this case, Application 1 can determine that the input position corresponding to the input event 1 is in the blank area.

[0256] It can be understood that Application 1 instructing the recognition engine object to call the text recognition method can be understood as Application 1 calling the text recognition engine in the recognition engine, which is specifically used to recognize the handwritten content as text.

[0257] It should be noted that Application 1 can also determine whether the user operation corresponding to the input event 1 is a handwritten text action through other means, and this application does not limit this.

[0258] It can be understood that this application does not limit the sequence of Application 1 executing step S204 and step S205.

[0259] S206: Application 1 registers to listen for the recognition engine object.

[0260] S207: Application 1 starts the text recognition engine.

[0261] It can be understood that the specific implementation manners of steps S206 - S207 can refer to steps S106 - S107, and this application will not elaborate here.

[0262] S208: Application 1 requests the text recognition engine to obtain association words.

[0263] Optionally, after Application 1 activates the text recognition engine, it can request the text recognition engine to obtain associative words. In some embodiments of the present application, after Application 1 activates the text recognition engine, it can send a specific parameter to the text recognition engine, and this specific parameter is used to indicate whether to obtain associative words. Exemplarily, if Application 1 sends "Word=true" to the text recognition engine, it indicates that Application 1 requests the text recognition engine to obtain associative words. However, if Application 1 sends "Word=false" to the text recognition engine, it indicates that Application 1 does not need to obtain associative words.

[0264] It should be noted that Application 1 can request the text recognition engine to obtain associative words simultaneously with or before sending the recognition point information (or input event) to the text recognition engine for the first time.

[0265] S209: Application 1 obtains recognition point information 3 based on input event 4 acting on the writing layer.

[0266] S210: Application 1 sends recognition point information 3 to the text recognition engine.

[0267] It is understandable that Figure 6B The shown recognition point information 3 is text recognition point information.

[0268] It is understandable that the specific implementation manners of steps S209 - S210 can refer to steps S108 - S109, and the present application will not elaborate herein.

[0269] S211: The text recognition engine obtains text recognition result 1 based on recognition point information 3.

[0270] After receiving the recognition point information 3 sent by Application 1, the text recognition engine can obtain text recognition result 1 based on recognition point information 3, and its specific implementation manner can refer to the above, and the present application will not elaborate herein.

[0271] It is understandable that the text recognition result 1 can be specific text. Exemplarily, the text recognition result 1 can be "therefore".

[0272] In some embodiments of the present application, the text recognition engine can send the text recognition result obtained based on the recognition point information received within a period of time (for example, 0.5 seconds) to Application 1 each time, rather than sending the text recognition result recognized based on the recognition point information of a single input event to Application 1 each time.

[0273] S212: The text recognition engine sends text recognition result 1 to Application 1.

[0274] After the text recognition engine obtains text recognition result 1 based on recognition point information 3, it can send text recognition result 1 to Application 1.

[0275] Correspondingly, Application 1 can receive the text recognition result 1 sent by the text recognition engine.

[0276] S213: Application 1 adds the text recognition result 1 to the custom text control in Application 1.

[0277] After Application 1 receives the text recognition result 1 sent by the text recognition engine, it can directly add the text recognition result 1 to the corresponding custom text control in Application 1 (i.e., the custom text control affected by Input Event 1).

[0278] S214: The text recognition engine determines the associated word 1 based on the text recognition result 1.

[0279] After the text recognition engine obtains the text recognition result 1, it can determine the associated word 1 based on the text recognition result. In some embodiments of the present application, the text recognition engine can search for words, phrases, or punctuation marks used after the text recognition result 1 in the thesaurus to obtain the associated word 1. In some embodiments of the present application, the text recognition engine can also search for words, phrases, or punctuation marks used after the text recognition result 1 based on the user's previous usage habits to obtain the associated word 1.

[0280] S215: The text recognition engine sends the associated word 1 to Application 1.

[0281] After the text recognition engine determines the associated word 1, it can send the associated word 1 to Application 1.

[0282] Correspondingly, Application 1 can receive the associated word 1 sent by the text recognition engine and display the associated word 1 for the user to select.

[0283] Exemplarily, as Figure 1F and Figure 1K shown, Application 1 can add the text recognition result "old man" to Document 1 and display several associated words, which can include a comma, "is old", "let it be", and "immortal".

[0284] S216: Application 1 determines that the duration of no detected input event is not less than T2.

[0285] Application 1 can monitor the duration between the current moment and the moment when the last input event was detected (i.e., the duration of no detected input event) and determine whether this duration is not less than T2. If this duration is not less than T2, then Application 1 can execute Step S217 and Step S218.

[0286] It can be understood that T2 can be the same as T1 or different from T1. T2 can be set according to actual needs, and the present application does not limit this. For example, T2 can be 5 seconds.

[0287] It should be noted that Application 1 can also determine whether to delete the writing layer and turn off the text recognition engine in other ways, and this application does not limit this.

[0288] S217: Application 1 deletes the writing layer.

[0289] After Application 1 determines that the duration of not detecting an input event is not less than T2, it can delete the writing layer.

[0290] S218: Application 1 turns off the text recognition engine.

[0291] After Application 1 determines that the duration of not detecting an input event is not less than T2, it can turn off the text recognition engine.

[0292] It can be understood that the specific implementation manners of steps S216 - S218 can refer to steps S117 - S119, and this application will not elaborate here.

[0293] It should be noted that in some embodiments of this application, Application 1 can directly send an input event (for example, input event 4) to the text recognition engine, and the text recognition engine can recognize the input event to obtain the corresponding text recognition result. It can be understood that the input object mentioned here can be an InputEvent object, which carries relevant information about the contact point between the stylus and the display screen.

[0294] In some embodiments of this application, if the gesture recognition engine or the text recognition engine runs into an error (an exception occurs and sequential recognition cannot be performed), then the gesture recognition engine or the text recognition engine can return an error message to Application 1. After receiving the error message, Application 1 can clear the recognition results obtained by the gesture recognition engine or the text recognition engine before, and configure parameters (for example, initialization parameters), and then instruct the gesture recognition engine or the text recognition engine to recognize again.

[0295] The following introduces the hardware structure of the electronic device involved in the embodiments of this application.

[0296] Please refer to Figure 7 , Figure 7 which is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of this application.

[0297] As Figure 7As shown, the electronic device may include: a processor, an external memory interface, an internal memory, a Universal Serial Bus (USB) interface, a charging management module, a power management module, a battery, Antenna 1, Antenna 2, a mobile communication module, a wireless communication module, a sensor module, keys, a motor, an indicator, a camera, a display screen, and a Subscriber Identity Module (SIM) card slot, etc. Among them, the audio module may include a speaker, a receiver, a microphone, a headphone interface, etc., and the sensor module may include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0298] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. It is understandable that the illustrated components can be implemented in hardware, software, or a combination of software and hardware. In some embodiments of the present application, the electronic device may include more components than those illustrated. Exemplarily, the electronic device may include other types of sensors. In still other embodiments of the present application, the electronic device may include fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The interface connection relationships between the modules illustrated in the embodiments of the present application are only illustrative and do not constitute a structural limitation on the electronic device.

[0299] The processor may include one or more processing units. For example, the processor may include an Application Processor (AP), a Modem (which may also be referred to as a baseband processor), a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a video codec, a Digital Signal Processor (DSP), and / or a Neural-network Processing Unit (NPU), etc. Among them, the AP is the processor responsible for running the operating system and application programs. The Modem is the processor responsible for processing various communication protocols.

[0300] The wireless communication function of the electronic device can be implemented through Antenna 1, Antenna 2, the mobile communication module, the wireless communication module, and the Modem, etc. The Modem can interact with the base station through an antenna (for example, Antenna 1, Antenna 2, etc.). In some embodiments, Antenna 1 of the electronic device is coupled to the mobile communication module, and Antenna 2 is coupled to the wireless communication module, enabling the electronic device to communicate with the network and other devices through wireless communication technologies.

[0301] The electronic device can implement the display function through the GPU, the display screen, and the application processor, etc.

[0302] The GPU is a microprocessor for image processing, connected to the display screen and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor may include one or more GPUs, which execute program instructions to generate or change the display information. The display screen is used to display images, videos, etc. In some embodiments, the electronic device may include one or more display screens.

[0303] The camera is used to capture static images or videos. The ISP is used to process the data fed back by the camera. Light is transmitted through the lens to the camera sensor, the optical signal is converted into an electrical signal, and the camera sensor transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The electronic device may include one or more cameras.

[0304] The internal memory may include one or more RAMs and one or more non-volatile memories (Non-Volatile Memory, NVM). The random access memory can be directly read and written by the processor, and can be used to store the operating system or the executable programs of other running programs (for example, machine instructions), and can also be used to store the data of users and application programs, etc. The non-volatile memory can also store executable programs and store the data of users and application programs, etc., and can be pre-loaded into the random access memory for the processor to directly read and write.

[0305] In the embodiments of the present application, the code for implementing the methods described in the embodiments of the present application can be stored on the non-volatile memory. When running Application 1, the electronic device can load the executable code stored in the non-volatile memory into the random access memory.

[0306] The external memory interface can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device.

[0307] The electronic device can implement the audio function through the audio module, the speaker, the receiver, the microphone, the headphone interface, and the application processor, etc.

[0308] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A text editing method, characterized in that, Applied to an electronic device equipped with a display screen, the method includes: Display a first interface through the display screen; the first interface includes a first control; the first control is not a text control provided by the operating system of the electronic device; Detect a first operation acting on the first interface; In response to the first operation, generate a first input event; When the first input event is a stylus input event and the input position corresponding to the first input event is within a first area; the first area includes the area where the first control is located; the writing layer is a transparent layer covering the first interface; Detect a second operation acting on the writing layer; In response to the second operation, generate a first input event sequence; Identify the first input event sequence through an identification software development kit (SDK) to obtain a first text or a first editing method; Edit the text in the first control based on the first text or the first editing method.

2. The method according to claim 1, wherein The first interface is the user interface of a first application; The creating of the writing layer includes: creating the writing layer through the first application; Before the identifying the first input event sequence through the identification SDK, the method further includes: Invoke the identification SDK through a first interface in the first application; the first interface is a custom identification interface encapsulated in the first application; Send the first input event sequence to the identification SDK through the first application.

3. The method according to claim 1 or 2, characterized in that, The first interface is the user interface of a first application; the identification SDK includes a pen gesture recognition SDK and a text recognition SDK; identifying the first input event sequence through the identification SDK to obtain the first text or the first editing method, and editing the text in the first control based on the first text or the first editing method includes: When the input position corresponding to the first input event is on the text included in the first control, identify the first input event sequence through the pen gesture recognition SDK to obtain the first editing method, and determine the text to be edited through the first application, and edit the text to be edited based on the first editing method; the position of the text to be edited is the same as the input position corresponding to the first input event sequence; When the input position corresponding to the first input event is within the blank area included in the first control, identify the first input event sequence through the text recognition SDK to obtain the first text, and add the first text to the text included in the first control through the first application.

4. The method according to any one of claims 1 to 3, characterized in that, After the identifying the first input event sequence through the identification SDK, the method further includes: Delete the writing layer when a first preset condition is met; Close the identification SDK when a second preset condition is met; Among them, the satisfaction of the first preset condition includes any one or more of the following: the time interval between the current moment and the last detected user operation on the writing layer is equal to or greater than the first duration; a non-stylus input event is detected; The satisfaction of the second preset condition includes any one or more of the following: the time interval between the current moment and the last detected user operation on the writing layer is equal to or greater than the second duration; a non-stylus input event is detected; a stylus input event with a corresponding event type of the first type is detected.

5. The method according to any one of claims 1-4, characterized in that The first interface is the user interface of the first application; the first input event sequence includes a series of sequentially generated input events; The recognition of the first input event sequence by the recognition software development kit (SDK) includes: Recognizing M1 input events in the first input event sequence through the recognition SDK to obtain a first recognition result, and sending the first recognition result to the first application through the recognition SDK; the recognition result is text or an editing method; M1 is a positive integer; Recognizing M2 input events in the first input event sequence through the recognition SDK to obtain a second recognition result, and in the case where the second recognition result is different from the first recognition result, sending the first recognition result to the first application through the recognition SDK; M2 is a positive integer.

6. The method according to claim 5, characterized in that, The generation time of the M2 input events is later than the generation time of the M1 input events.

7. The method according to claim 5, wherein M2 is greater than M1, and the M2 input events include one or more of the M1 input events, and the generation time of the other input events in the M2 input events except the M1 input events is later than the generation time of the M1 input events.

8. The method according to claim 6 or 7, characterized in that, In the case where the second recognition result is different from the first recognition result, the second recognition result is the first text or the first editing method; in the case where the second recognition result is the same as the first recognition result, the first recognition result is the first text or the first editing method.

9. The method according to claim 7, wherein After obtaining the first text, the method further includes: Searching for associated words based on the first text through the text recognition SDK, and sending one or more found associated words to the first application; After adding the first text to the text included in the first control through the first application, the method further includes: Displaying a second interface through the display screen; the second interface includes the first control and a first display area; the one or more associated words are displayed in the first display area; Detecting a third operation on a first associated word in the first display area; In response to the third operation, adding the first associated word to the text included in the first control.

10. An electronic device, characterized in that, The electronic device includes a display screen, one or more memories, and one or more processors; the display screen is coupled to the one or more memories and the one or more processors, the display screen is configured to display a picture, the memory is configured to store computer program code, the computer program code includes computer instructions, and the processor invokes the computer instructions to execute the method according to any one of claims 1-9 above.

11. A computer-readable storage medium, characterized in that, For storing computer instructions, when the computer instructions run on an electronic device, the electronic device is caused to execute the method according to any one of claims 1-9 above.

Images