Method for displaying input content and electronic device

Abstract

The present application relates to the technical field of terminals, and discloses a method for displaying input content and an electronic device. In response to a first operation on a text note editing interface, a handwriting area is added to the editing interface. A user can input content in the handwriting area by means of handwriting. In response to a second operation of the user, the handwriting area is deleted, and a first handwriting display area on the editing interface displays the content input by means of handwriting. In response to a third operation in the first handwriting display area, the handwriting area is displayed on the editing interface again, and the user can continue to input content in the handwriting area by means of handwriting. During this period, the text content in the editing interface is always displayed. Interface transitions are reduced, bringing smooth use experience to users. Moreover, before the editing interface is generated, handwritten input content and text content do not correspond to different layers, thereby avoiding the problem of display misalignment after form changes of a tablet computer caused by text content and handwritten input content being located on different layers.

Description

Method and electronic device for displaying input content Technical Field

[0001] The embodiments of the present application relate to the field of terminal technology, and more particularly to a method for displaying input content and an electronic device. Background Art

[0002] Generally speaking, terminal devices all support note-taking applications, which users can use to record text, pictures, audio and other information.

[0003] As note-taking apps become increasingly powerful, the user experience is also becoming richer. For example, users can enter text using the keyboard, copy and paste content from the clipboard, add hyperlinks, add images, audio, and more, and even write text, graphics, and more in a handwritten format.

[0004] Handwriting input is convenient and fast, allowing users to enter text, graphics, and other content as needed, making it popular. However, handwriting input in note-taking apps still has some flaws. Optimizing the handwriting interaction process in note-taking apps and improving the user experience is a challenge that needs to be addressed.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a method and electronic device for displaying input content, which optimize the display method of handwritten input content of a note-taking application and improve the user experience of inputting content by handwriting.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, a method for displaying input content is provided, which is applied to an electronic device, for example, a terminal device. Taking the terminal device as an example, the method includes: the terminal device displays a first interface; the first interface includes a first text area, the first text area including first text content; in response to a first operation on a first option in the first interface, creating a first view and displaying a second interface; the second interface includes a first handwriting area and a second text area, the second text area including the first text content, and the first view corresponding to the first handwriting area. In response to a first handwriting input in the first handwriting area, the first handwriting input content is displayed in the first handwriting area. In response to a second operation in the second interface (for example, a click operation on an area outside the first handwriting area in the second interface), the first view is deleted and a third interface is displayed; the third interface includes a first handwriting display area and a third text area, the first handwriting display area including the first handwriting input content, and the third text area including the first text content. In response to a third operation in the first handwriting display area of ​​the third interface, creating a second view and displaying a fourth interface; the fourth interface includes the first handwriting area and the second text area, and the second view corresponds to the first handwriting area.

[0009] In this method, in response to a first user action on a text note editing interface, a handwriting area is added to the text note editing interface. The user can enter content by handwriting in this handwriting area. In response to a second user action, such as a click outside the handwriting area, the handwriting area is deleted, and the first handwriting display area on the text note editing interface displays the content entered by the user by handwriting. In response to a third user action on the first handwriting display area, the handwriting area is re-displayed on the editing interface, and the user can continue to enter content by handwriting in the handwriting area. During this time, the first text content remains displayed on the editing interface. This provides a smoother user experience and reduces the user experience degradation caused by switching between the editing interface and the handwriting input interface. Furthermore, the editing interface includes a handwriting display area and a text area, with handwritten content displayed in the handwriting display area and text content displayed in the text area. Before the editing interface is generated, the handwriting input content and text content are not assigned to different layers, thus avoiding the problem of text content and handwriting input content being on different layers, resulting in display misalignment after the tablet computer changes shape.

[0010] In one possible implementation, in response to the second operation on the second interface, a second image is further created, the second image corresponding to the first handwriting display area in the third interface. The first handwriting display area includes the first handwriting input content. Optionally, the second image can be a picture.

[0011] That is, in the second interface, the first handwritten input content is saved as an image, which is also considered rich text content and is placed on the same layer as other text content. This avoids the display misalignment problem caused by the handwritten input content and text content being on different layers.

[0012] In combination with the first aspect, in a possible implementation, the third operation includes a second handwriting input, and in response to the second handwriting input in the first handwriting display area of ​​the third interface, the second handwriting input content is displayed in the first handwriting area of ​​the fourth interface.

[0013] In this method, the user can start handwriting input directly on the handwriting picture (first handwriting display area). In response to the user's handwriting input on the handwriting picture, the handwriting picture is deleted and a view corresponding to the handwriting area (second view) is created.

[0014] In one possible implementation, the fourth interface is generated based on a preset layer (i.e., a pre-drawn transition layer) and the layer containing the second view. Optionally, the fourth interface includes the preset layer and the layer containing the second view; the preset layer is located above the layer containing the second view. In response to a second handwriting input in the first handwriting display area of ​​the third interface, the transition layer is set to visible. Second handwriting input content is generated in the preset layer and displayed in the first handwriting area of ​​the fourth interface.

[0015] Because the transition layer is pre-drawn and not created after receiving the second handwriting input, the second handwriting input content drawn on the transition layer can be displayed before the second view is created. This allows the handwriting corresponding to the second handwriting input to be displayed on the screen through the transition layer before the view corresponding to the first handwriting area in the fourth interface is successfully created. Because the transition layer is pre-drawn, drawing the second handwriting input content on the transition layer is quick, effectively shortening the time interval between the user's handwriting input and the display of the second handwriting input content, improving the tracking of the user's handwriting input.

[0016] After the second view is created, the second view can receive input events corresponding to the user's handwriting input, process the input events, and generate the second handwriting input content on the layer where the second view is located. After the second handwriting input content is generated on the layer where the second view is located, and if no handwriting input is currently occurring, the transition layer is set to invisible. The second handwriting input content can be displayed on the screen through the second view. Subsequently, the second view can process the input events generated by the user's handwriting input in the first handwriting area and display the handwriting input content in the first handwriting area.

[0017] In combination with the first aspect, in a possible implementation, when the second interface is displayed, in response to a morphological change of the terminal device (for example, the morphological change includes: rotation, folding or unfolding, etc.), the first view is deleted; after deleting the first view, a third view is created and the fifth interface is displayed; the fifth interface includes a second handwriting area and a fourth text area, the third view corresponds to the second handwriting area, the second handwriting area includes the first handwriting input content, and the fourth text area includes the first text content.

[0018] In this method, when an editing interface (e.g., a second interface) including a handwriting area undergoes a morphological change, the view corresponding to the handwriting area before the morphological change is deleted, and a new view corresponding to the handwriting area is generated based on the screen layout after the morphological change. The regenerated view (the third view) is laid out according to the layout rules after the morphological change.

[0019] In response to the terminal device changing its form, before deleting the first view, a first image is generated based on the first view. Optionally, the first image can be a picture. In this way, a third view can be generated based on the first image and the screen display area after the terminal device changes its form. The first image records the display position and size of the first handwritten area on the screen.

[0020] In combination with the first aspect, in a possible implementation, when the third interface is displayed, in response to a morphological change of the terminal device, a sixth interface is displayed; the sixth interface includes a second handwriting display area and a fifth text area, the second handwriting display area includes first handwriting input content, and the fifth text area includes first text content; in the process from the morphological change of the terminal device to the display of the sixth interface, no view is deleted or created.

[0021] In this method, the third interface includes a first handwriting display area and a third text area. After the terminal device changes its form, the sixth interface is generated by performing an interface layout on each view in the third interface based on the screen display area. The first handwriting display area corresponds to the second handwriting display area, and the third text area corresponds to the fifth text area. No view is deleted or created.

[0022] In conjunction with the first aspect, in one possible implementation, the third operation includes a click operation, and the fourth interface is the same as the second interface. Optionally, the fourth interface is the same as the second interface, or the fourth interface is the second interface. The status bar does not belong to the interface; that is, if two interfaces differ only in the status bar, they are the same.

[0023] In this method, the user can click on the handwriting image (first handwriting display area) to display the first handwriting area on the editing interface, so that the user can continue to input handwriting in the first handwriting area.

[0024] In conjunction with the first aspect, in one possible implementation, the third interface further includes a third handwriting area, the third handwriting area includes third handwriting input content, the fourth interface further includes a third handwriting display area, the third handwriting display area includes third handwriting input content, and in response to a third operation in the first handwriting display area of ​​the third interface, a third image is created, the third image corresponding to the third handwriting display area. Optionally, the third image can be a picture.

[0025] In this method, the editing interface includes a handwriting area and a handwriting picture (handwriting display area). In response to the user's third operation on the handwriting picture, the handwriting area is displayed in the form of a handwriting picture.

[0026] In combination with the first aspect, in a possible implementation, the first handwriting display area and the first handwriting area are the same display area on the screen.

[0027] In this method, the user is given the experience that the handwriting display area and the handwriting area are the same area, avoiding the user's experience of the handwriting area jumping.

[0028] In conjunction with the first aspect, in a possible implementation, the first text content includes at least one of text, pictures, audio, tables, and hyperlinks. In other words, the text area may include rich text content.

[0029] In a second aspect, an electronic device is provided that has the functionality to implement the method described in the first aspect. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functionality described above.

[0030] In a third aspect, an electronic device is provided, comprising: a processor, a memory, and a display screen, wherein the memory is used to store computer-executable instructions. When the electronic device is running, the processor executes the computer-executable instructions stored in the memory so that the electronic device executes the method described in any possible implementation of the first aspect above.

[0031] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on the computer, the computer can execute the method described in any possible implementation of the first aspect.

[0032] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the method described in any possible implementation of the first aspect.

[0033] Among them, the technical effects brought about by any possible implementation of the second to fifth aspects can refer to the technical effects brought about by different implementations of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] FIG1 is a schematic diagram of a user interface of a note-taking application on a tablet computer;

[0035] FIG2 is a schematic diagram of a scenario example of a handwriting interaction method in a note-taking application;

[0036] FIG3 is a schematic diagram of a scenario example of a handwriting interaction method in a note-taking application;

[0037] FIG4A is a schematic diagram of a scenario example of a method for displaying input content provided by an embodiment of the present application;

[0038] FIG4B is a schematic diagram of a scenario example of a method for displaying input content provided in an embodiment of the present application;

[0039] FIG4C is a schematic diagram of a scenario example of a method for displaying input content provided in an embodiment of the present application;

[0040] FIG5A is a schematic diagram of a scenario example of a method for displaying input content provided by an embodiment of the present application;

[0041] FIG5B is a schematic diagram of a scenario example of a method for displaying input content provided in an embodiment of the present application;

[0042] FIG6 is a schematic diagram of a scenario example of a method for displaying input content provided by an embodiment of the present application;

[0043] FIG7A is a schematic diagram of a scenario example of a method for displaying input content provided by an embodiment of the present application;

[0044] FIG7B is a schematic diagram of a scenario example of a method for displaying input content provided in an embodiment of the present application;

[0045] FIG8 is a schematic diagram of a hardware structure of a terminal device applicable to the method for displaying input content provided in an embodiment of the present application;

[0046] FIG9 is a schematic diagram of a software architecture of a terminal device applicable to the method for displaying input content provided in an embodiment of the present application;

[0047] FIG10A is a schematic diagram of a process of adding a handwriting area to a text note editing interface in a method for displaying input content provided by an embodiment of the present application;

[0048] FIG10B is a schematic diagram of a process of converting a handwritten area into a handwritten image in a method for displaying input content provided by an embodiment of the present application;

[0049] FIG11A is a schematic diagram of a process of rotating a terminal device in a method for displaying input content provided by an embodiment of the present application;

[0050] FIG11B is a schematic diagram of a terminal device orientation in a method for displaying input content provided in an embodiment of the present application;

[0051] FIG12 is a flow chart of a method for displaying input content provided in an embodiment of the present application;

[0052] FIG13 is a schematic diagram of a scenario example of a method for displaying input content provided in an embodiment of the present application;

[0053] FIG14 is a flow chart of a method for displaying input content provided in an embodiment of the present application;

[0054] FIG15 is a schematic diagram of the structural composition of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: the situation where A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are a kind of "or" relationship.

[0056] References to "one embodiment" or "some embodiments" etc. described in this specification mean that the specific features, structures or characteristics described in conjunction with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connected" includes direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0057] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0058] The method for displaying input content provided in the embodiment of the present application can be applied to electronic devices that support handwriting input. For example, the electronic device can be a terminal device. The above-mentioned terminal device can include a mobile phone, a tablet computer, a laptop computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a smart home device (such as a smart TV, a smart screen, a large screen, a smart speaker, a smart air conditioner, etc.), a personal digital assistant (PDA), a wearable device (such as a smart watch, a smart bracelet, etc.), a vehicle-mounted device, a virtual reality device, etc., and the embodiment of the present application does not impose any restrictions on this.

[0059] In one example, a note-taking application is installed on a terminal device. For example, taking a tablet as an example, FIG1 illustrates a user interface diagram of the note-taking application on the tablet. It is understood that the note-taking application may have different names on different terminal devices, such as "Notebook" or "Notepad."

[0060] 1 , tablet computer 100 displays a user interface 101 of a note-taking application, which includes an "Add" button 102. In response to a user clicking on "Add" button 102, a "Text Note" option 1021, a "Handwritten Note" option 1022, and the like are displayed on user interface 101. The user can click on "Text Note" option 1021 to create a text note, and tablet computer 100 displays the text note editing interface. The user can enter text, symbols, images, audio, hyperlinks, and the like on the text note editing interface. The user can also enter content (e.g., text, graphics, and the like) by handwriting on the text note editing interface. The user can click on "Handwritten Note" option 1022 to create a handwritten note. The user can enter text, graphics, and the like by handwriting on the handwritten note user interface (canvas).

[0061] Exemplarily, as shown in FIG1 , in response to a click on the “Text Note” option 1021 , the tablet computer 100 displays a text note editing interface 103 . Optionally, the user can create a text note by clicking on the “Text Note” option 1021 . The editing interface 103 includes a text area, in which the user can enter rich text content such as text, symbols, images, hyperlinks, and audio. In one implementation, the editing interface 103 includes a virtual keyboard 1031 , and the user can enter rich text content such as text, numbers, and symbols into the text area of ​​the editing interface 103 by clicking on virtual keys in the virtual keyboard 1031 . In another implementation, the terminal device includes an input device, such as a keyboard, and the user can enter rich text content such as text, numbers, and symbols into the text area of ​​the editing interface 103 via the keyboard.

[0062] In one implementation, the editing interface 103 includes a handwriting option 1032. After the user clicks the handwriting option 1032, he can perform handwriting input in the handwriting area of ​​the editing interface 103; after the tablet computer 100 receives the user's handwriting input in the handwriting area, the handwriting area will display the user's handwriting trajectory (text, numbers, symbols, graphics, etc.); the handwriting trajectory generated by the user's handwriting input will be retained in the handwriting area and will not disappear when the handwriting trajectory is drawn.

[0063] For example, in some implementations, the editing interface 103 further includes other controls. For example, the editing interface 103 further includes a "Collapse" button 1033. In response to clicking the "Collapse" button 1033, the virtual keyboard 1031 is no longer displayed on the editing interface 103. For example, the editing interface 103 further includes a text tool 1034. The user can modify the font of the text in the text area by clicking the text tool 1034. For example, the editing interface 103 further includes an "Add" button 1035. In response to the user clicking the "Add" button 1035, options such as "Gallery," "Table," and "Hyperlink" are displayed on the editing interface 103. The user can add an image to the text area of ​​the editing interface 103 by clicking the "Gallery" option; the user can add a table to the text area of ​​the editing interface 103 by clicking the "Table" option; the user can add a hyperlink to the text area of ​​the editing interface 103 by clicking the "Hyperlink" option. For example, the editing interface 103 further includes a “recording” button 1036 , and the user can add a recording file to the text area of ​​the editing interface 103 by clicking the “recording” button 1036 .

[0064] Handwriting input in text notes is convenient and fast, and there are many ways to implement it.

[0065] In one implementation, referring to (a) of FIG2 , the editing interface of the text note is generated based on at least two layers, wherein layer A includes rich text content such as text, pictures, tables, hyperlinks, etc., and layer B includes handwritten input content.

[0066] Exemplarily, referring to (b) of FIG2 , the tablet computer 100 displays an editing interface 201 for text notes. The text area of ​​the editing interface 201 includes text content such as words and pictures, and also includes a mark graphic for handwriting input, such as the black shadow block in (b) of FIG2 .

[0067] In this implementation, before the editing interface of the text note is generated based on at least two layers, the handwritten input content and the text content are located in different layers (for example, a layer can be a View or Layer), and the handwritten input content can be superimposed on the text content. However, when the terminal device undergoes a change in form, such as when a tablet computer is rotated, such as when a foldable screen mobile phone is folded or unfolded, the text content and the handwritten input content may be displayed misaligned. For example, as shown in Figure 2 (b), the mark graphic is located above the text "note content", "list", and "style"; after the tablet computer is rotated, as shown in Figure 2 (c), the mark graphic and the text "note content", "list", and "style" are misaligned. This is because the text content and the handwritten input content are located in different layers, and the two layers are independent. The tablet computer independently layouts each layer. Under different layout methods, the text content and the handwritten input content may be misaligned.

[0068] In another implementation, the tablet displays a text note editing interface, into which the user can enter rich text content such as text and audio. When the user needs to use handwriting input, the tablet displays the text note handwriting input interface, which does not include rich text content, and the user can enter handwriting input content into the handwriting input interface by handwriting. After the user stops handwriting input, the tablet no longer displays the handwriting input interface and displays an updated editing interface, which includes an image generated based on the handwriting input content.

[0069] For example, referring to FIG3 , tablet computer 100 displays a text note editing interface 301, which includes an image 3021 and a text area 3022. Image 3021 is generated from a handwriting trace, and text area 3022 includes text content. In one example, in response to a user clicking image 3021, tablet computer 100 displays a handwriting input interface 303, which includes a handwriting area 3031, where the user can enter handwriting. As shown in FIG3 , handwriting area 3031 is relatively large, occupying almost the entire display area of ​​handwriting input interface 303, which does not include a text area.

[0070] After the user completes handwriting input, for example, by clicking "Save" button 3032 in handwriting input interface 303, tablet computer 100 displays an updated editing interface 301. Updated editing interface 301 includes image 3023, which is generated based on the user's handwriting input content in handwriting area 3031. It should be noted that the dashed boxes in editing interface 301 and handwriting input interface 303 shown in FIG3 are only used to indicate the area range and are not displayed in actual applications.

[0071] In this implementation, the handwritten input content (handwritten trace) is displayed as an image on the editing interface. Before the editing interface is generated, the handwritten input content and the text content are placed on the same layer, avoiding the problem of text content and handwritten input content being misaligned after the tablet is rotated. However, handwriting input requires operation in the handwriting input interface, which causes a jump between the editing interface and the handwriting input interface, resulting in a poor user experience.

[0072] The present application provides a method for displaying input content. Instead of performing handwriting input in a handwriting input interface, handwriting operations can be performed in a handwriting area of ​​a text note editing interface. After completing handwriting input, the handwriting input content in the handwriting area is saved as an image and displayed in the updated editing interface. Because the handwriting input content is displayed as an image, the handwriting input content and the text content are located on the same layer, avoiding the problem of misalignment between the text content and the handwriting input content after the tablet computer changes shape; and reducing the poor user experience caused by the jump between the editing interface and the handwriting input interface.

[0073] In one example, a tablet computer displays a text note editing interface 1, which includes a text area 1. A user can enter rich text content such as text, symbols, images, tables, hyperlinks, and audio in text area 1. In response to a user adding a handwriting area (e.g., clicking a handwriting button), the tablet computer displays a text note editing interface 2, which includes a text area 2 and a handwriting area. The user can enter rich text content such as text, symbols, images, tables, hyperlinks, and audio in text area 2, and the user can write handwritten input content into the handwriting area. Text area 2 corresponds to text area 1, and the rich text content displayed in text area 2 is the same as that displayed in text area 1. The starting positions of text area 2 and text area 1 may be different, and the areas displayed on the screen of text area 2 and text area 1 may also be different. Optionally, the initial size (e.g., width and height) of the handwriting area in editing interface 2 is a preset value, and the size of the handwriting area can be dynamically adjusted based on the handwriting input content. The size of the handwriting area can also change in response to the user adjusting the boundaries of the handwriting area. In one example, with the horizontal direction as the X-axis and the direction perpendicular to the ground as the Y-axis, the width of the handwriting area represents the length of the handwriting area in the X-axis direction, and the height of the handwriting area represents the length of the handwriting area in the Y-axis direction.

[0074] Exemplarily, referring to FIG4A(a), the tablet computer 100 displays a text note editing interface 401. The editing interface 401 includes a title bar, an editing area, a toolbar, etc. The title bar includes the title of the text note, the editing time, and classification options. In the editing interface 401, the entire editing area is a text area, such as text area 11, in which the user can enter rich text content such as text, symbols, images, tables, hyperlinks, and audio. Exemplarily, text area 11 includes the text a. The bottom toolbar includes multiple controls, such as a handwriting option 4021, a text tool 4022, an "Add" button 4023, and a "Record" button 4024. It should be noted that the dotted box in the editing interface 401 shown in FIG4A(a) is only used to indicate the area range and is not displayed in actual application. Optionally, the editing interface 401 can be the editing interface 103 shown in FIG1. ​​In this case, in response to the user clicking the "Collapse" button 1033, the virtual keyboard 1031 is not displayed. Among them, the handwriting option 4021 can be the handwriting option 1032 in Figure 1, the text tool 4022 can be the text tool 1034 in Figure 1, the "Add" button 4023 can be the "Add" button 1035 in Figure 1, and the "Recording" button 4024 can be the "Recording" button 1036 in Figure 1.

[0075] In one example, as shown in (b) of FIG4A , in response to a user clicking on the handwriting option 4021, the tablet computer 100 displays an editing interface 403 for text notes, and the editing interface 403 includes a title bar, an editing area, a bottom toolbar, etc.; wherein, the title bar in the editing interface 403 is the same as the title bar in the editing interface 401; the editing area includes a handwriting area 12 and a text area 11'; the user can write handwriting input content to the handwriting area 12 by handwriting, and the user can enter rich text content such as text, symbols, pictures, tables, hyperlinks, audio, etc. in the text area 11'; the bottom toolbar includes a handwriting tool button, and the user can click the handwriting tool button to select the line thickness, pen shape, color, etc. used for handwriting input in the handwriting area 12. In one implementation, after the user clicks the "Add" button 4023, the tablet computer 100 adds the View of the handwriting area 12 in the View corresponding to the editing area, thereby adding a handwriting area to the editing area of ​​the editing interface; accordingly, the display area of ​​the text area in the editing area becomes smaller. For example, the display area of ​​the text area 11' in (b) of Figure 4A is smaller than the display area of ​​the text area 11 in (a) of Figure 4A; the rich text content included in the text area 11' is the same as the rich text content included in the text area 11. In one implementation, the View corresponding to the text area 11' is the View of the text area 11. Optionally, in the editing interface, the handwriting tool button and the handwriting area of ​​the bottom toolbar appear at the same time; that is, when the editing area of ​​the editing interface includes a handwriting area, the bottom toolbar includes a handwriting tool button.

[0076] It should be noted that the dotted frame in the editing interface 403 shown in FIG4A(b) is only used to indicate the area range and is not displayed in actual application.

[0077] For example, as shown in FIG4A(c), the user can input text, numbers, graphics, etc. in the handwriting area 12 by handwriting.

[0078] After the user finishes handwriting input in the handwriting area 12, a save operation may be performed; for example, the save operation includes clicking an area outside the handwriting area in the editing interface, etc. For example, as shown in FIG4A(c), the user clicks on the text area 11'.

[0079] In one implementation, as shown in FIG4B , in response to a user's save operation on the handwriting area 12 (e.g., a click operation within the text area 11'), the tablet computer 100 displays an editing interface 404 for the text note. The editing area of ​​the editing interface 404 includes a picture 13 and a text area 11". The picture 13 is generated based on the handwriting area 12. In one example, the picture 13 is the same size as the handwriting area 12, and the positions where the two are displayed on the screen are also the same; the display area and display position of the text area 11" are the same as those of the text area 11'. The elements in the picture 13 are generated based on the handwriting input content (handwriting trajectory) within the handwriting area 12, and the text area 11" includes the text a.

[0080] In another implementation, as shown in FIG4C , in response to the user's save operation on the handwriting area 12 (e.g., a click operation within the text area 11 ′), the tablet computer 100 displays an editing interface 404 for the text note, and the editing area of ​​the editing interface 404 includes a picture 13 and a text area 11 ″. The picture 13 has the same starting display position (the upper left vertex position of the picture) as the handwriting area 12, and the display area of ​​the picture 13 is different from that of the handwriting area 12; for example, the height of the picture 13 is different from that of the handwriting area 12. Exemplarily, there is less handwriting input content within the handwriting area 12, and the height covered by the handwriting track within the handwriting area is less than the height of the handwriting area. The height of the picture 13 is determined based on the height of the area covered by the handwriting input content within the handwriting area. The elements in the picture 13 are generated based on the handwriting input content within the handwriting area 12. Accordingly, the height of the text area 11 ″ is different from that of the text area 11 ′, and the text area 11 ″ includes the text a.

[0081] 4B and 4C , editing interface 404 does not include a handwriting area, and accordingly, does not include a handwriting tool button. Exemplarily, as shown in FIG4B and 4C , editing interface 404 includes a handwriting option 4051, a text tool 4052, an "add" button 4053, and a "record" button 4054. Handwriting option 4051 may be handwriting option 4021 in FIG4A (a), text tool 4052 may be text tool 4022 in FIG4A (a), "add" button 4053 may be "add" button 4023 in FIG4A (a), and "record" button 4054 may be "record" button 4024 in FIG4A (a).

[0082] For the sake of convenience, in the embodiments of the present application, the image generated based on the handwritten input content (handwriting trajectory) of the handwriting area is referred to as a handwriting image. For example, Figure 13 is a handwriting image. In the embodiments of the present application, the display area occupied by the handwriting image is referred to as the handwriting display area.

[0083] Optionally, in one implementation, in response to the user's save operation on the handwriting area 12, the tablet computer 100 saves information of the handwriting trajectory within the handwriting area 12, for example, the information of the handwriting trajectory within the handwriting area 12 is saved in a binary file; the tablet computer 100 also determines the display position and size of the picture 13 based on the display position and size of the handwriting area 12; the tablet computer 100 also generates elements in the picture 13 based on the handwriting trajectory within the handwriting area 12.

[0084] It can be understood that the dotted frame in the editing interface 404 shown in FIG. 4B and FIG. 4C is only used to indicate the range of the picture 13; in actual application, the dotted frame may not be displayed.

[0085] In this example, the text note editing interface remains displayed while the user is inputting handwriting, and the rich text content in the text area is always displayed on the editing interface, providing a smooth user experience. Furthermore, the handwritten input content can be saved as an element of Image 13; Image 13 and Text a are located on the same layer and will not be misaligned due to rotation of the tablet 100.

[0086] In some embodiments, a user can perform an editing operation on a handwritten image (e.g., click on the handwritten image) to continue handwriting input within the handwriting area. In one example, the editing operation includes clicking on the handwritten image. For example, referring to FIG5A , tablet computer 100 displays editing interface 404, which includes image 13. In response to a user clicking on image 13, tablet computer 100 displays editing interface 403; image 13 in editing interface 404 corresponds to handwriting area 12 in editing interface 403. Optionally, in one implementation, after receiving a user click on image 13, tablet computer 100 determines the position and size of the corresponding handwriting area (handwriting area 12) based on the display position and size of image 13. It also generates a handwriting trajectory for the corresponding handwriting area (handwriting area 12) based on the handwriting trajectory information of the handwriting area saved when generating image 13. For example, the handwriting trajectories "1," "2," "3," and "4" are displayed within handwriting area 12. The user can also continue handwriting input within handwriting area 12. Illustratively, in response to the user's handwriting input in the handwriting area 12 , the tablet computer 100 displays an updated editing interface 403 , on which the handwriting trace “ 5 ” is displayed in the handwriting area 12 .

[0087] In this way, the user experience is that the position of the handwriting area does not change, and the text content of the text area is always displayed on the editing interface, making the user experience more convenient and smooth.

[0088] In some embodiments, the user does not need to perform an editing operation on the handwritten image, but can instead start handwriting input directly on the handwritten image. The tablet computer receives the user's handwriting input on the handwritten image, deletes the handwritten image, generates a handwriting area corresponding to the handwritten image, and displays the handwritten input content within the handwriting area. For example, referring to FIG5B , the tablet computer 100 displays an editing interface 404, which includes image 13; in response to the user's handwriting input on image 13, the tablet computer 100 displays an editing interface 403, which includes a handwriting area 12.

[0089] Among them, picture 13 corresponds to handwriting area 12. In one example, the display position and size of handwriting area 12 are consistent with the display position and size of picture 13. The handwriting input content (handwriting trajectory) displayed in handwriting area 12 includes two parts; one part of the handwriting trajectory is the same as the elements in picture 13, for example, the handwriting trajectory "1", "2", "3", and "4". In one implementation, this part of the handwriting trajectory is generated based on the handwriting trajectory information saved when generating picture 13; the other part of the handwriting trajectory is generated based on the user's handwriting input on picture 13.

[0090] In this way, the user does not need to trigger the generation of the handwriting area through an editing operation (such as clicking on the handwriting picture) first, but can start handwriting input directly on the picture, which is convenient for the user to write.

[0091] In some embodiments, a tablet computer displays a first editing interface for a text note. The first editing interface includes a first handwriting image and a second handwriting area. Optionally, the first editing interface also includes a text area. In response to a user inputting handwriting on the first handwriting image, the tablet computer displays a second editing interface. The second editing interface includes the first handwriting area and the second image. Optionally, the second editing interface also includes a text area. The first handwriting area corresponds to the first image, and the second handwriting area corresponds to the second image. The text content in the text area on the second editing interface is the same as the text content in the text area on the first editing interface.

[0092] In one example, the position and size of handwriting area one are determined based on the position and size of picture one, and the handwriting input content within handwriting area one is generated based on the user's handwriting input on handwriting picture one and the information of the handwriting trajectory of handwriting area one saved by the tablet computer; the display position and size of picture two are determined based on the position and size of handwriting area two, and the elements of picture two are generated based on the handwriting trajectory within handwriting area two.

[0093] Exemplarily, as shown in FIG6(a), tablet computer 100 displays an editing interface 405 for a text note. Editing interface 405 includes picture 14 and handwriting area 15. Picture 14 is a handwritten picture, and the elements of picture 14 are generated based on the handwritten input content. For example, picture 14 is picture 13 in FIG4B. That is, the elements in picture 14 are generated based on the handwritten input content in handwriting area 12 (handwriting traces "1", "2", "3", and "4"), and the elements in picture 14 are the same as the handwriting traces in handwriting area 12. Handwriting area 15 includes handwriting trace b. Optionally, editing interface 405 also includes text area 1a.

[0094] In one example, a user performs handwriting input on picture 14 (or clicks on picture 14), and tablet computer 100 receives the user's handwriting input on picture 14 (or the operation of clicking on picture 14). Exemplarily, as shown in FIG6(b), in response to the user's handwriting input on picture 14 (or the operation of clicking on picture 14), tablet computer 100 displays editing interface 406, which includes handwriting area 16 and picture 17.

[0095] Image 17 is generated based on handwriting area 15. In one example, the display position and size of image 17 are determined based on the display position and size of handwriting area 15; for example, the display position and size of image 17 are consistent with the display position and size of handwriting area 15. The elements in image 17 are identical to the handwriting trace in handwriting area 15. For example, the element in image 17 is handwriting trace b.

[0096] Handwriting area 16 is generated based on image 14. In one example, the size (width and height) of handwriting area 16 is the same as the size (width and height) of image 14, and the display position of handwriting area 16 is the same as the display position of image 14. Optionally, handwriting area 16 is generated in response to a user's handwriting input on image 14, and the handwriting trace displayed in handwriting area 16 is generated based on two handwriting traces: one part is generated based on the user's handwriting input on image 14, and the other part is the same as the handwriting trace of the element in image 14. Optionally, handwriting area 16 is generated in response to a user clicking on image 14, and the handwriting trace displayed in handwriting area 16 is the same as the handwriting trace of the element in image 14.

[0097] In some embodiments, when a user uses a terminal device, the terminal device may change shape, such as a tablet computer rotating, or a foldable screen mobile phone folding or unfolding.

[0098] In one example, when the tablet computer is displaying a text note editing interface, the tablet computer rotates. It is understandable that after the tablet computer rotates, the text note editing interface will be rearranged according to the change in the screen display area.

[0099] If the editing interface includes a handwriting area, the display position and size of the handwriting area will change as the screen display area changes.

[0100] Exemplarily, referring to FIG7A , when the tablet computer 100 is in portrait orientation, the editing interface 403 is displayed. The editing area of ​​the editing interface 403 includes a handwriting area 12 and a text area 11 ′; for example, the text area 11 ′ includes the character a; optionally, the title bar of the editing interface 403 includes a "title" control, an "unclassified" control, etc.; and the bottom toolbar of the editing interface 403 includes a plurality of handwriting tool buttons, etc. The tablet computer is rotated from portrait to landscape orientation, and the tablet computer 100 displays the editing interface 407 in landscape orientation. In one implementation, the editing interface 407 is obtained by rearranging the editing interface 403 based on the screen display area in landscape orientation. Exemplarily, as shown in FIG7A , the text area 11 ′ in the editing interface 403 corresponds to the text area 18 in the editing interface 407, and the handwriting area 12 in the editing interface 403 corresponds to the handwriting area 19 in the editing interface 407.

[0101] Optionally, when the editing interface is rearranged in response to rotation of the tablet computer, the text a, the "title" control, the "unclassified" control, the handwriting toolbar, etc. can be laid out using interface layout rules in conventional technologies.

[0102] Regarding handwriting area 12, in one implementation, upon detecting that tablet computer 100 has begun to rotate, Picture 1 is determined based on the position and size of handwriting area 12. Optionally, the position of Picture 1 in editing interface 403 is the same as the position of handwriting area 12 in editing interface 403, and the size of Picture 1 is the same as the size of handwriting area 12. In this way, the position and size of handwriting area 12 in editing interface 403 can be saved via Picture 1. For example, the position of the upper left corner of Picture 1 is determined based on the position of the upper left corner of handwriting area 12, and the size (e.g., width and height) of Picture 1 is determined based on the size (e.g., width and height) of handwriting area 12.

[0103] After detecting that the tablet computer 100 has completed rotation, the position and size of the handwriting area included in the text note editing interface when in landscape mode are determined based on Image 1. In one implementation, the position of the top left corner of the handwriting area (handwriting area 19 in FIG. 7A ) included in the editing interface when in landscape mode is determined based on the top left corner of Image 1. It will be appreciated that the layout of the editing interface differs when the tablet computer 100 is in portrait and landscape mode, and the position of the handwriting area 19 in the editing interface when in landscape mode may differ from the position of Image 1 in the editing interface when in portrait mode. In one implementation, the size of the handwriting area 19 is determined based on the size of Image 1. In one example, the height of the handwriting area 19 is the same as the height of Image 1, and the width of the handwriting area 19 is the width of the screen display area when the tablet computer 100 is in landscape mode. Thus, by preserving the position and size of the handwriting area in Image 1, the position and size of the handwriting area after the tablet computer is rotated can be determined based on the position and size of the handwriting area before the tablet computer is rotated and the changes in the tablet computer's screen display area, thereby avoiding the problem of misaligned display of handwritten input content after the tablet computer is rotated.

[0104] In one example, the image 1 is used to determine the position and size of the handwriting area 19 , and the image 1 is not displayed on the screen of the tablet computer 100 ; for example, during the rotation of the tablet computer 100 , a rotation effect is displayed on the screen.

[0105] In the method for displaying input content provided by the embodiment of the present application, a handwriting area can be added to the editing interface of the text note, and the handwriting area can be used to receive handwriting input content; after the user finishes the handwriting input, the editing interface is updated, and the updated editing interface no longer includes the handwriting area, but displays a handwritten picture generated based on the handwriting input content in the handwriting area; in this way, if the terminal device undergoes morphological changes such as rotation, folding, and unfolding, the handwritten picture and rich text content (such as text, audio, hyperlinks, etc.) participate in the rearrangement of the user interface, avoiding the problem of misalignment between the handwriting input content and the rich text content. When the user performs handwriting input on the handwriting picture again, the editing interface is updated again, and the updated editing interface includes a handwriting area generated based on the handwriting picture. The handwriting picture and the handwriting area are switched at the same position, and the rich text content is always displayed in the editing interface, giving the user a smooth writing experience.

[0106] If the editing interface includes a handwritten image, the display position and size of the handwritten image will change as the screen display area changes.

[0107] For example, as shown in FIG7B , the tablet computer 100 displays an editing interface 404 of a text note when in portrait orientation. The editing area of ​​the editing interface 404 includes a picture 13 and a text area 11 ″, and the text area 11 ″ includes text a.

[0108] Tablet computer 100 rotates from portrait to landscape orientation, displaying editing interface 408 in landscape orientation. In one implementation, editing interface 408 is obtained by rearranging editing interface 404 based on the screen display area in landscape orientation. For example, as shown in FIG7B , text area 11′ in editing interface 404 corresponds to text area 20 in editing interface 408, and image 13 in editing interface 404 corresponds to image 21 in editing interface 408.

[0109] The editing interface 408 is generated by laying out the various views (e.g., text area 11', image 13, etc.) in the editing interface 404 according to the screen display area of ​​the tablet computer 100 when the tablet computer 100 is in landscape orientation. During the process of rotating the tablet computer 100 to display the editing interface 408, no views are deleted or created.

[0110] The specific implementation of the method for displaying input content provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0111] The method for displaying input content provided in the embodiment of the present application can be applied to a terminal device that supports handwriting input. Figure 8 shows a schematic structural diagram of a terminal device. Among them, the terminal device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a camera 191, a display screen 192, etc. The sensor module 180 may include a pressure sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an acceleration sensor, a gyroscope sensor, etc.

[0112] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the terminal device 100. In other embodiments, the terminal device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0113] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0114] The controller can be the nerve center and command center of the terminal device 100. The controller can generate operation control signals according to instruction operation codes and timing signals to complete the control of instruction fetching and execution.

[0115] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0116] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0117] It is understood that the interface connection relationship between the modules illustrated in this embodiment is only for illustrative purposes and does not constitute a structural limitation on the terminal device. In other embodiments, the terminal device may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0118] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the terminal device's wireless charging coil. While charging the battery 142, the charging management module 140 can also provide power to the terminal device via the power management module 141.

[0119] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the internal memory 121, the external memory, the display 192, the camera 191, and the wireless communication module 160.

[0120] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0121] The terminal device 100 implements display functions through a GPU, display screen 192, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 192 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0122] Display screen 192 is used to display images, videos, etc. Display screen 192 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini-LED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0123] In the embodiment of the present application, the display screen 192 can be used to display a user interface, such as a desktop interface, a note application interface, etc.

[0124] The terminal device 100 can realize the shooting function through the ISP, camera 191, video codec, GPU, display screen 192 and application processor.

[0125] The camera 191 is used to capture static images or videos. In some embodiments, the terminal device may include 1 or N cameras 191, where N is a positive integer greater than 1.

[0126] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0127] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, audio, video, and other files can be stored on the external memory card.

[0128] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the terminal device by running the instructions stored in the internal memory 121. For example, in an embodiment of the present application, the processor 110 can execute instructions stored in the internal memory 121, and the internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and an application required for at least one function. The data storage area can store data created during the use of the terminal device, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0129] The pressure sensor is used to sense pressure signals and can convert pressure signals into electrical signals. In some embodiments, the pressure sensor can be set on the display screen 192. There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a device including at least two parallel plates with conductive material. When force acts on the pressure sensor, the capacitance between the electrodes changes. The terminal device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the display screen 192, the terminal device 100 detects the intensity of the touch operation based on the pressure sensor. The terminal device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor.

[0130] A touch sensor, also known as a "touch panel," can be provided on display screen 192. The touch sensor and display screen 192 form a touch screen, also known as a "touch screen." The touch sensor is used to detect touch operations applied to or near the touch sensor. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 192.

[0131] In some embodiments, a touch operation performed by a user on the screen may be detected by a pressure sensor or a touch sensor, and text, graphics, etc. may be generated according to the touch operation.

[0132] The accelerometer can detect the magnitude of acceleration of the terminal device 100 in all directions (generally three axes). When the terminal device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the terminal device's posture, which is applicable to applications such as landscape and portrait screen switching and pedometers.

[0133] The gyroscope sensor can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (ie, x, y, and z axes) can be determined by the gyroscope sensor.

[0134] The buttons 190 include a power button, a volume button, etc. The buttons 190 may be mechanical buttons or touch buttons.

[0135] In the embodiment of the present application, the terminal device is a terminal device that can run an operating system and install applications. Optionally, the operating system running on the terminal device can be system, system, System, etc.

[0136] The software system of the terminal device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. Taking the system as an example, the software structure of the terminal device 100 is exemplified.

[0137] FIG9 is a block diagram of the software structure of the terminal device 100 according to an embodiment of the present invention.

[0138] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, The system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system library, and kernel layer.

[0139] The application layer may include a series of application packages, such as camera, gallery, calendar, call, map, navigation, etc. For example, the application layer may include a note application.

[0140] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0141] As shown in FIG9 , the application framework layer may include a window manager service (WMS), an input management service, a content provision service, a view system, a resource management service, and the like.

[0142] The window management service is used to manage window programs. For example, it can add, delete, and modify windows. The window management service can also obtain the display size, determine whether there is a status bar, lock the screen, and take screenshots.

[0143] The input management service is used to manage input events, such as distributing input events.

[0144] Content provisioning services are used to store and retrieve data and make it accessible to applications. Such data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0145] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can be composed of one or more views. For example, a note-taking application might include a text note interface, which might include views for displaying text and images.

[0146] Resource management services provide various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0147] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for System scheduling and management.

[0148] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0149] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0150] The system library can include multiple functional modules, such as the surface management module (surface manager), the surface flinger (SF) module, the input flinger module, the media library (Media Libraries), the 3D graphics processing library (such as OpenGL ES), and the 2D graphics engine (such as SGL).

[0151] The surface management module is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0152] The composition rendering module is responsible for compositing and rendering layers based on the interface display data (e.g., the number of layers included in the interface and the display elements of each layer). Based on the display data, the composition rendering module determines the composition instructions (e.g., OpenGL instructions, HWComposer instructions) corresponding to each display data, invokes the composition instructions to generate each layer of the application interface, and sends them for display.

[0153] The input synthesis module is used to generate corresponding input events according to the signals generated by the input device, perform event synthesis, etc.

[0154] The sensor module is used to generate corresponding sensor data based on the signals collected by the sensor and process the sensor data.

[0155] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0156] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0157] A 2D graphics engine is a drawing engine for 2D drawings.

[0158] The kernel layer is the layer between hardware and software. The kernel layer may include display drivers, sensor drivers, keyboard drivers, etc.

[0159] In one example, after receiving a user operation on the screen of the terminal device 100, the sensor driver sends an operation event corresponding to the user operation to the input synthesis module, the input synthesis module sends the operation event to the input management service, and the input management service sends the operation event to the WMS. The WMS determines the application corresponding to the operation event based on the window display position of each application. For example, if the user operation acts on the user interface (editing interface) of the note application, the WMS sends the operation event to the note application. The note application performs corresponding processing based on the operation event, such as updating the interface data of the editing interface. The note application notifies the WMS that the interface data of the editing interface has been updated. The synthesis rendering module synthesizes and renders according to the interface data of the updated editing interface according to the window management rules of the WMS to generate the layer of the updated editing interface; and calls the synthesis instruction to display the layer of the updated editing interface.

[0160] The following describes in detail the method for displaying input content provided by an embodiment of the present application in conjunction with the functional modules shown in FIG9 .

[0161] In some embodiments, a handwriting area may be added to the text note editing interface.

[0162] For example, taking the scenario example shown in FIG4A as an example, FIG10A shows a flow chart of adding a handwriting area in the editing interface. As shown in FIG10A , the method includes:

[0163] S1001. After the note application is started, the tablet computer displays a text note editing interface 1; when the note application is started, an activity 1 (Activity 1) is created.

[0164] In one example, when the note application is started, Activity 1 is created. After the note application is started, the tablet computer displays the user interface of the note application, such as the first editing interface of the text note. For example, the first editing interface is the editing interface 401 shown in FIG4A(a).

[0165] As you can understand, all elements in the user interface of a note-taking application are created based on View and ViewGroup. View and ViewGroup are combined together in a tree structure to form the user interface of the note-taking application (e.g., the text note editing interface).

[0166] For example, taking the editing interface 401 in (a) of Figure 4A as an example, the root node in the View tree structure corresponding to Activity 1 is decorView. The child nodes of decorView include ViewContainer 1, and the child nodes of ViewContainer 1 include WebView 1. WebView 1 is the View corresponding to text area 11. Furthermore, child nodes such as image files and audio files can be added to WebView 1; this allows rich text content such as images and audio to be added to text area 11.

[0167] Optionally, the child nodes of ViewContainer 1 also include ViewGroup 1 corresponding to the title bar, ViewGroup 2 corresponding to the bottom toolbar, etc. For example, referring to FIG4A (a), the bottom toolbar includes a handwriting option 4021, a text tool 4022, an "Add" button 4023, a "Record" button 4024, etc. Correspondingly, the child nodes of ViewGroup 2 include a View corresponding to the handwriting option 4021, a View corresponding to the text tool 4022, a View corresponding to the "Add" button 4023, a View corresponding to the "Record" button 4024, etc.

[0168] S1002: In response to a user clicking a handwriting button, the sensor driver sends a click event to an input synthesis module.

[0169] The bottom toolbar of the editing interface 401 includes a handwriting option 4021. The user can click the handwriting option 4021 to create a handwriting area on the editing interface 401.

[0170] In one example, a sensor (e.g., a touch sensor or a touch panel) detects a user's click operation on the screen, and the sensor driver generates a click event based on the click operation and reports the click event to the input synthesis module; optionally, the click event includes the location of the click operation, the intensity of the click operation, etc. It is understandable that click operations include down operations and up operations. In one implementation, the sensor driver generates a down event based on the down operation and an up event based on the up operation; the sensor driver sending the click event to the input synthesis module includes sending a down event and an up event.

[0171] S1003: The input synthesis module sends a click event to the input management service, and the input management service sends the click event to the WMS.

[0172] S1004. The WMS sends the click event to the note application.

[0173] In one implementation, the WMS determines that the foreground application is the note application based on the window currently focused, and sends the click event to the note application for processing.

[0174] S1005. The note application generates a canvas view (HandView) corresponding to the handwriting area, and adds the canvas view (HandView) to Activity 1 (Activity 1).

[0175] In one implementation, the note-taking application receives a click event and, based on the click event, obtains the click operation's location, click operation intensity, and other information. For example, the click event is distributed layer by layer along the View tree to the View corresponding to handwriting option 4021. The View corresponding to handwriting option 4021 consumes the click event.

[0176] In one example, the consumption of the click event includes: generating a canvas view (HandView), and adding the canvas view (HandView) to ViewContainer 1. In this way, the child nodes of ViewContainer 1 include WebView 1, canvas view (HandView), etc. WebView 1 corresponds to text area 11, and canvas view (HandView) corresponds to handwriting area 12. Optionally, the display position of HandView is determined according to the position of the cursor on the display screen, and the size of HandView is a preset value. It can be understood that View includes HandView, WebView, etc., that is, HandView is a type of View.

[0177] Optionally, in one implementation, corresponding to the generation of HandView, an image file 1 is also generated, and the image file 1 includes information about the picture 1, such as the size of the picture 1, the elements of the picture 1, etc. Exemplarily, the size of the picture 1 is the same as the display size of the HandView, and the elements of the picture 1 are empty (i.e., the picture 1 is a blank image). The image file 1 is added to the WebView 1 to add a blank picture to the text area, and the size of the blank picture is the same as the display size of the HandView. The display position of the blank picture is also the same as the display position of the HandView. In this way, it can be achieved that as the handwriting area is added to the editing area, the display position of the text in the text area is changed. Exemplarily, as shown in (b) of FIG. 4A , after the handwriting area 12 is added to the editing area, the display position of the text a moves downward.

[0178] S1006 . The note application stores the updated data of Activity 1 into the Graphic Buffer of the system library, and notifies the WMS that Activity 1 has been updated.

[0179] In one implementation, the note application transmits the updated data of Activity 1 to the Graphic Buffer of the system library through ViewRootlmpl.java, that is, stores the updated data of Activity 1 in the Graphic Buffer of the system library.

[0180] The note-taking app also notifies WMS that Activity 1 has been updated.

[0181] S1007. When the next synchronization signal (Vsync) arrives, the synthesis and rendering module obtains the configuration information of the current image frame from the WMS, and synthesizes and renders the updated activity 1 (Activity 1) data stored in the graphics buffer (Graphic Buffer) based on the configuration information to generate the layer of the updated editing interface.

[0182] S1008. The synthesis rendering module sends the updated layer of the editing interface to the display driver, and the display driver displays the updated editing interface (editing interface 2) on the display screen according to the updated layer of the editing interface.

[0183] Exemplarily, the second editing interface is the editing interface 403 shown in FIG4A(b).

[0184] In some embodiments, after the user finishes handwriting input in the handwriting area, the handwriting area is saved in the editing interface in the form of a picture.

[0185] Exemplarily, as shown in FIG10B , the method further includes:

[0186] S1009: In response to a click operation performed by the user outside the handwriting area of ​​the second editing interface, the sensor driver reports a second click event to the input synthesis module.

[0187] 4B , tablet computer 100 displays editing interface 403 , which includes handwriting area 12 . The touch sensor detects a click operation by the user on the display screen, which is performed on an area outside handwriting area 12 in editing interface 403 .

[0188] The sensor driver generates a second click event based on the click operation and reports the second click event to the input synthesis module. Optionally, the second click event includes the location of the click operation, the intensity of the click operation, etc. It is understood that click operations include down operations and up operations. In one implementation, the sensor driver generates a down event based on the down operation and an up event based on the up operation; the sensor driver sending the second click event to the input synthesis module includes sending the down event and the up event.

[0189] S1010: The input synthesis module sends a second click event to the input management service, and the input management service sends the second click event to the WMS.

[0190] S1011. The WMS sends a second click event to the note application.

[0191] S1012. The note application generates image file 1 according to the handview in activity 1, adds image file 1 to activity 1, and deletes the handview from activity 1.

[0192] The note application generates an image file 1 according to the canvas view (HandView) in Activity 1 (Activity 1). The image file 1 includes information of the image 1, such as the size of the image 1, elements of the image 1, etc. Exemplarily, the image file is in JPG format.

[0193] In one example, the size of Image 1 is the same as the display size of HandView, and the elements of Image 1 are generated based on the handwriting trajectory on HandView. For example, referring to Figure 4B, the canvas view (HandView) is the View corresponding to handwriting area 12, Image 1 is Image 13, and Image 13 is the same size as handwriting area 12. The elements of Image 13 are the handwriting trajectory within handwriting area 12.

[0194] Add image file 1 to WebView 1 and add image file 1 to the text area. Image file 1's dimensions match the HandView's display size and position. Optionally, delete the blank image file from WebView 1. Furthermore, remove HandView from ViewContainer 1. This replaces the handwriting area in the editing interface with image file 1.

[0195] In one implementation, before deleting the HandView, a binary file is generated based on the handwriting trace on the HandView, and the binary file stores the information of the handwriting trace on the HandView. In this way, when the handwriting area is subsequently restored in the editing interface, the handwriting trace in the handwriting area can be restored based on the binary file.

[0196] S1013. The note application stores the updated data of Activity 1 into the Graphic Buffer of the system library, and notifies the WMS that Activity 1 has been updated.

[0197] S1014. When the next synchronization signal (Vsync) arrives, the synthesis and rendering module obtains the configuration information of the current image frame from the WMS, and synthesizes and renders the updated activity 1 (Activity 1) data stored in the graphics buffer (Graphic Buffer) based on the configuration information to generate the layer of the updated editing interface.

[0198] S1015. The synthesis rendering module sends the updated layer of the editing interface to the display driver, and the display driver displays the updated editing interface (editing interface three) on the display screen according to the updated layer of the editing interface.

[0199] Exemplarily, the third editing interface is the editing interface 404 shown in FIG. 4B .

[0200] In some embodiments, after the terminal device undergoes morphological changes such as rotation, folding, and unfolding, the user interface will be rearranged, and the various controls in the editing interface, such as text, pictures, handwriting areas, etc., will change their display positions according to the screen display area after the morphological change.

[0201] For example, FIG11A shows a flow chart of a terminal device rotating in a method for displaying input content provided by an embodiment of the present application. As shown in FIG11A , the method includes:

[0202] S1101. Start the note application.

[0203] In one example, in response to a user clicking an application icon, the note application is launched.

[0204] S1102. The note application notifies the WMS to start an activity (startActivity) 1.

[0205] In one implementation, the note application calls the startActivity function to send a notification to the WMS to start Activity 1 (Activity 1).

[0206] S1103 . The note application notifies the WMS to create Activity 1 . In the function of creating Activity 1 (onCreate) 1 , the note application registers a direction listener with the WMS.

[0207] In one example, the note application calls the onCreate function to notify the WMS to create Activity 1 (Activity 1).

[0208] In one implementation, during onCreate, the note-taking application registers an orientation listener (onOrientationChangeListener) with the WMS. After registering the orientation listener, the WMS sends a notification message to the note-taking application when the orientation of the terminal device changes.

[0209] Optionally, in other embodiments, the note-taking application registers a posture listener with the WMS; the WMS sends a notification message to the note-taking application after the posture of the terminal device changes.

[0210] S1104: The sensor driver periodically reports the posture or orientation information of the terminal device to the WMS.

[0211] In one implementation, the orientation of the terminal device includes up, down, left, and right. The orientation of the head of the terminal device (the part where the camera is provided) is taken as the orientation of the terminal device. Exemplarily, referring to FIG11B , the X-axis, Y-axis, and Z-axis of the geodetic coordinate system are used as references, wherein the positive direction of the X-axis points to due east, the positive direction of the Y-axis points to due north, and the positive direction of the Z-axis, the positive direction of the X-axis, and the positive direction of the Y-axis satisfy the right-hand rule. When the angle between the orientation of the terminal device and the positive direction of the Z-axis is less than a preset threshold, the orientation of the terminal device is upward; when the angle between the orientation of the terminal device and the negative direction of the Z-axis is less than a preset threshold, the orientation of the terminal device is downward; when the angle between the orientation of the terminal device and the positive direction of the X-axis is less than a preset threshold, the orientation of the terminal device is rightward; when the angle between the orientation of the terminal device and the negative direction of the X-axis is less than a preset threshold, the orientation of the terminal device is leftward.

[0212] For example, the direction of the terminal device can be represented by a direction flag, where a direction flag of 0 represents upward, a direction flag of 1 represents rightward, a direction flag of 2 represents downward, and a direction flag of 3 represents leftward.

[0213] Optionally, the orientation of the terminal device may be determined based on the posture of the terminal device.

[0214] In one implementation, the sensor driver periodically reports the posture information of the terminal device to the Window Management Service (WMS).

[0215] In one implementation, the sensor driver determines the orientation of the terminal device according to the posture of the terminal device, and the sensor driver periodically reports the orientation information of the terminal device to a window management service (WMS).

[0216] S1105. The WMS determines, based on the posture or orientation information of the terminal device, that the orientation of the terminal device has changed, that is, determines that the terminal device has rotated.

[0217] In one implementation, the WMS receives posture information of the terminal device and determines the orientation of the terminal device according to the posture information.

[0218] In one implementation, if it is determined that the orientation information received this time is different from the orientation information received last time, the WMS determines that the terminal device has rotated.

[0219] S1106. After determining that the terminal device has rotated, the WMS sends configuration change information to the note application.

[0220] Because the note-taking application has registered an orientation listener with the window management service, upon determining that the terminal device has rotated, the window management service sends a configuration change message to the note-taking application. In one example, the WMS calls the onConfigureChanged() function to send the configuration change message to the note-taking application, which carries the current orientation information of the terminal device.

[0221] At this time, the terminal device displays the second editing interface of the text note. For example, referring to FIG7A , the second editing interface is the editing interface 403, and the editing interface 403 includes the handwriting area 12. At this time, Activity 1 includes a canvas view (HandView).

[0222] S1107. The note application generates image file 1 according to the handview in activity 1, adds image file 1 to activity 1, and deletes the handview from activity 1.

[0223] The specific implementation of the note application can refer to the relevant description in S1012 and will not be repeated here.

[0224] S1108 . The note application stores the updated data of Activity 1 into the Graphic Buffer of the system library, and notifies the WMS that Activity 1 has been updated.

[0225] S1109. When the next synchronization signal (Vsync) arrives, the synthesis and rendering module obtains the configuration information of the current image frame from the WMS, and synthesizes and renders the updated activity 1 (Activity 1) data stored in the graphics buffer (Graphic Buffer) based on the configuration information to generate the layer of the updated editing interface.

[0226] Optionally, during the rotation of the terminal device, the layer of the editing interface is not displayed on the screen of the terminal device, and a rotation animation is displayed instead. For example, the rotation animation can be generated based on a screenshot of the screen display interface when the terminal device starts to rotate. In one example, the layer of the rotation animation is displayed above the layer of the editing interface.

[0227] S1110. The note application generates a handview 2 based on the image file 1 in the activity 1, adds the handview 2 to the activity 1, and deletes the image file 1 from the activity 1.

[0228] It is understandable that after the terminal device is rotated, the orientation of the terminal device and the screen display area will change accordingly. The note application rearranges Activity 1 according to the interface layout rules corresponding to the current orientation of the terminal device. Exemplarily, the editing interface 403 includes a text area 11, a handwriting area 12, a title bar, a bottom toolbar, etc.; correspondingly, Activity 1 includes WebView 1, an image file 1, a ViewGroup 1 corresponding to the title bar, a ViewGroup 2 corresponding to the bottom toolbar, etc. Among them, WebView 1, ViewGroup 1 corresponding to the title bar, ViewGroup 2 corresponding to the bottom toolbar, etc., can be rearranged according to the interface layout rules after the terminal device is rotated and the solutions in conventional technologies are adopted, which will not be repeated in the embodiments of this application.

[0229] For image file one, image file one includes information of picture one, such as the display position and size of picture one; the display position and size of picture one are determined based on the display position and size of the handwriting area 12. In one implementation, the note application generates canvas view 2 (HandView 2), and HandView 2 corresponds to the handwriting area after the terminal device is rotated. In one example, the display position and size of HandView 2 are determined based on the display position and size of picture one. For example, the position of the upper left vertex of picture one is determined as the display position of the upper left vertex of HandView 2, and the height of picture one is determined as the height of HandView 2. In one example, a handwriting track on HandView 2 is generated based on the information of the handwriting track saved on HandView. In this way, the display position, size, and display content (handwriting track) of HandView 2 are determined.

[0230] Furthermore, HandView 2 is added to ViewContainer 1 of Activity 1. In this way, the child nodes of ViewContainer 1 include WebView 1, canvas view 2 (HandView 2), etc.

[0231] Optionally, delete the image file 1 and delete the handwriting track information saved on the HandView.

[0232] S1111. The note application stores the updated data of Activity 1 into the Graphic Buffer of the system library, and notifies the WMS that Activity 1 has been updated.

[0233] S1112. When the next synchronization signal (Vsync) arrives, the synthesis and rendering module obtains the configuration information of the current image frame from the WMS, and synthesizes and renders the data of the updated activity 1 (Activity 1) stored in the graphics buffer (Graphic Buffer) based on the configuration information to generate the layer of the updated editing interface.

[0234] S1113. The synthesis rendering module sends the updated layer of the editing interface to the display driver, and the display driver displays the updated editing interface (editing interface three) on the display screen according to the updated layer of the editing interface.

[0235] 7A , after the tablet computer 100 is rotated, the editing interface 407 (editing interface three) is displayed. The editing interface 407 includes a handwriting area 19 (HandView 2 ). The handwriting area 19 is a handwriting area corresponding to the handwriting area 12 .

[0236] In some embodiments, when a handwriting area is displayed on an editing interface in the form of an image, the editing interface is updated in response to a user's handwriting input in the handwriting area (in the form of an image), and the handwriting image is replaced with an editable handwriting area. For example, referring to FIG5B , tablet computer 100 displays editing interface 404, where editing interface 404 includes image 13; in response to a user's handwriting input on image 13, tablet computer 100 displays editing interface 403, where editing interface 403 includes handwriting area 12. For example, referring to FIG6 , tablet computer 100 displays editing interface 405, where editing interface 405 includes image 14 and handwriting area 15; in response to a user's handwriting input on image 14, tablet computer 100 displays editing interface 406, where editing interface 406 includes handwriting area 16 and image 17; wherein handwriting area 16 corresponds to image 14, and image 17 corresponds to handwriting area 15.

[0237] When the handwriting area in the editing interface is displayed as an image, the user can start handwriting input directly on the handwriting area in the image (handwriting image). In response to the user's handwriting input in the handwriting area in the image, the terminal device updates the editing interface, and the updated editing interface includes the handwriting area corresponding to the handwriting image.

[0238] In one example, before the editing interface is updated, the text area of ​​the editing interface includes Picture 1 (for example, Picture 13), which is a handwritten picture. Accordingly, ViewContainer 1 in Activity 1 includes WebView 1, WebView 1 includes image file 1, and image file 1 includes information about Picture 1. In response to the user's handwriting input on Picture 1, the editing interface is updated. After the editing interface is updated, Picture 1 is deleted and a handwriting area is added. Accordingly, Image file 1 is deleted in WebView 1 of Activity 1, and HandView is added to ViewContainer 1. HandView is the View corresponding to the handwriting area.

[0239] Before the editing interface is updated, the handwriting area has not yet been generated. That is, the HandView has not yet been added to ViewContainer 1. Therefore, the canvas view (HandView) corresponding to the handwriting area will not receive input events corresponding to handwriting input. This will cause the first handwriting input stroke to be lost, and the first handwriting stroke cannot form a handwriting track within the handwriting area.

[0240] The method for displaying input content provided by an embodiment of the present application, with reference to Figure 12, before the editing interface is updated, that is, before the HandView corresponding to the handwriting area is generated, the web view (WebView) receives the input event on the handwriting picture, and the web view stores the received input event in the blocking queue, that is, the web view is the event production sub-thread of the blocking queue. After the HandView corresponding to the handwriting area is generated, the canvas view (HandView) can receive and process the input event. As the event consumption sub-thread of the blocking queue, the canvas view (HandView) takes out the input event from the blocking queue. The canvas view processes the input events in the chronological order of the input events and draws the handwriting according to the input events, so that the user's handwriting input on the handwriting picture (such as the first stroke of the handwriting input) can be avoided from being lost.

[0241] In the scenario where the user performs handwriting input directly on a handwritten image, only after the canvas view of the handwriting area is drawn can the canvas view of the handwriting area process the input event corresponding to the received handwriting input, draw the handwriting handwriting according to the input event, and generate the handwriting input content. For example, referring to Figure 12, the web page view receives the input event corresponding to the handwriting input on the handwritten image, and saves the input event corresponding to the handwriting input on the handwritten image to the blocking queue. After the canvas view of the handwriting area is drawn, the canvas view, as an event consumer sub-thread, takes out the input event from the blocking queue and draws the handwriting according to the input event. Since it takes a certain amount of time to draw the canvas view of the handwriting area, there will be a certain amount of time between receiving the user's handwriting input on the handwritten image (for example, the first stroke of the handwriting input) and generating the first stroke of the handwriting input. This will lead to the problem of poor hand tracking of the first stroke of the user's handwriting input, making the user experience that the first stroke of the handwriting is displayed some time after the first stroke is completed.

[0242] In the method for displaying input content provided by an embodiment of the present application, the note application pre-draws a transition layer, and the transition layer is located at the top layer of the editing interface of the note application. Before the handwriting area is added to the editing interface (before the HandView corresponding to the handwriting area is successfully created), the transition layer is visible. Handwriting is drawn on the transition layer according to the handwriting input on the handwriting picture. In this way, before the handwriting area is successfully created, the handwriting corresponding to the handwriting input on the handwriting picture can be displayed on the screen through the transition layer. After the handwriting area is successfully created, the canvas view of the handwriting area can process the input event corresponding to the handwriting input, and draw the handwriting on the canvas view according to the input event of the handwriting input. The transition layer is set to invisible, and the handwriting drawn on the canvas view is displayed on the screen.

[0243] Exemplarily, referring to Figure 13, the tablet computer 100 displays an editing interface 405, which includes a picture 14, which is a handwritten picture; the editing interface 405 also includes a handwriting area 15. In response to receiving the user's handwriting input on the picture 14, the transition layer is set to be visible. The note application draws handwriting on the transition layer according to the handwriting input on the picture 14, and the handwriting is displayed on the screen through the transition layer. In response to receiving the user's handwriting input on the picture 14, the handwriting area 16 is added to the updated editing interface (editing interface 406). After the canvas view corresponding to the handwriting area 16 is created, the note application draws handwriting on the canvas view of the handwriting area according to the handwriting input. The transition layer is set to invisible, and the handwriting can be displayed on the screen through the canvas view of the handwriting area.

[0244] In this way, before the handwriting area is successfully created, the handwriting is drawn on the transition layer. After the handwriting area is successfully created, the transition layer becomes invisible and the handwriting is drawn on the canvas view. Because the transition layer is pre-drawn, drawing handwriting on the transition layer takes less time, effectively shortening the time between the user's handwriting input and the display showing the handwriting, improving the tracking of the user's handwriting input.

[0245] For example, taking the scenario shown in FIG6 as an example, FIG14 shows a flow chart of a method for displaying input content provided by an embodiment of the present application when a user starts handwriting input on a handwritten picture. As shown in FIG14 , the method includes:

[0246] S1401. In response to a user's handwriting input on handwriting picture 1, the input management service sends an input event to the window management service.

[0247] When a user enters handwriting on handwritten image 1, the terminal device displays editing interface 4, which includes text area 1 and handwritten image 1. Taking the scenario shown in FIG6 as an example, tablet computer 100 displays editing interface 405, which includes image 14, which is a handwritten image. Editing interface 405 also includes handwriting area 15.

[0248] Illustratively, input events may include a down event, an up event, a move event, etc. When the sensor detects a down operation, the sensor driver generates a down event and sends the down event to the input management service, which then sends the down event to the window management service. When the sensor detects a up operation, the sensor driver generates an up event and sends the up event to the input management service, which then sends the up event to the window management service. When the sensor detects a move operation, the sensor driver generates a move event and sends the move event to the input management service, which then sends the move event to the window management service.

[0249] S1402. The window management service sends an input event to the note application.

[0250] In one implementation, the window management service determines that the foreground application is the note application based on the currently focused window and sends the click event to the note application for processing. In one example, the window management service calls the onTouchEvent function, which sends the input event to the note application through ViewRootlmpl.java.

[0251] S1403: The note application saves the received input event.

[0252] Each time the note-taking application receives an input event, it calls the obtainevent function to save the input event to the blocking queue.

[0253] In one implementation, the root node in the View tree structure corresponding to the Activity corresponding to the editing interface of the note application receives the input event, and distributes the input event downward step by step according to the hierarchy in the View tree structure. In one example, the input event is sent to the web view (WebView) corresponding to the text area, and the web view (WebView) stores the input event in a blocking queue as an event generation sub-thread. In this way, before the handwriting area is successfully created, the input event corresponding to the handwriting input on the handwriting picture is stored in the blocking queue; after the handwriting area is successfully created, the canvas view (HandView) corresponding to the handwriting area can obtain the input event from the blocking queue and draw the handwriting according to the input event, thereby avoiding the loss of handwriting input on the handwriting picture (such as the first stroke of handwriting input).

[0254] On the other hand, a user's input action on the display screen may include multiple input events. For example, the process of a user inputting a piece of handwriting on the display screen by handwriting may include a down event, multiple move events, and an up event; wherein, the input of the starting point of a piece of handwriting corresponds to a down event, and the input of the end point of a piece of handwriting corresponds to an up event. For example, a user's two-finger pressing action on the display screen corresponds to two consecutive down events. For example, a user's two-finger dragging action on the display screen corresponds to a down event, a move event, a down event, a move event... The note application saves each input event after receiving it, so that the user's input action can be judged based on multiple input events.

[0255] S1404: If a press event is received, the note application determines the location of the press event. If the location of the press event is on handwritten image 1, S1405 is executed; if the location of the press event is not on handwritten image 1, the web view processes the press event.

[0256] In one example, a terminal device displays an editing interface 4 of a note-taking application. Editing interface 4 includes a text area 1, which is implemented using a web view (WebView) 1. Editing interface 4 also includes a handwritten image 1, which is implemented using an image file 1; optionally, image file 1 is a child node of WebView 1. Editing interface 4 also includes a handwritten area 2, which is implemented using a canvas view (HandView) 3. In one example, HandView 3 and WebView 1 are both child nodes of ViewContainer 1.

[0257] After receiving the press event, the note application calls the function getCurrentClickedImage to traverse the handwritten images in the text area and determine which handwritten image the press event affects.

[0258] If it is determined that the action position of the press event is on the handwritten image, continue to wait for the move event and execute S1405; if it is determined that the action position of the press event is not on the handwritten image, the web page view processes the press event.

[0259] S1405: If a motion event is received, the note-taking application determines the location of the motion event. If the motion event is located on handwritten image 1 and the number of motion events on handwritten image 1 is greater than a preset value, S1406-S1407, S1411-S1412, and S1421 are executed.

[0260] The note application determines the effective position of the motion event. For example, the note application calls the function getCurrentClickedImage to traverse the handwritten images in the text area and determines on which handwritten image the effective position of the motion event is located.

[0261] If it is determined that the motion event is located on handwritten image 1, the number of motion events continuously received on handwritten image 1 is determined. If it is determined that the number of motion events continuously received on handwritten image 1 is greater than a preset value (e.g., the preset value is 2), steps S1406-S1407, S1411-S1412, and S1421 are executed. For example, the function onClickImageAtGraffitiMode is called to execute steps S1406-S1407, S1411-S1412, and S1421.

[0262] If it is determined that the action location of the move event is not on the handwritten image 1, the web page view processes the move event.

[0263] S1406. The note application sets the transition layer to be visible.

[0264] The note-taking app pre-draws a transition layer. In one implementation, the size of the transition layer is the size of the editing area within the editing interface. The transition layer is located above all layers in the editing interface.

[0265] The transition layer is invisible by default. After receiving a press event and a preset number of move events on handwriting image 1, indicating that the user is inputting handwriting on handwriting image 1, the note-taking app sets the transition layer to be visible (setSwitchSmoothHandWritingViewVisible), so that the handwriting drawn on the transition layer can be displayed on the screen.

[0266] S1407. The note application creates handwriting area one based on handwriting picture one.

[0267] In one example, the note-taking application calls the switchHandWritingView function to create handwriting area 1. Optionally, the note-taking application also generates handwriting image 2 based on handwriting area 2. For example, in the scenario shown in FIG6 , handwriting area 16 is created based on handwriting image 14. Correspondingly, handwriting image 17 is generated based on handwriting area 15.

[0268] In one implementation, the note application calls the function saveAndExitAsync to close the canvas view corresponding to handwriting area 2 (handwriting area 15), and calls the function addHandWritingLayout to create the canvas view corresponding to handwriting area 1 (handwriting area 16) based on the handwriting picture 1 (picture 14). In one example, the note application can close the canvas view corresponding to handwriting area 2 and create the canvas view corresponding to handwriting area 1 asynchronously, that is, there is no need to wait for the action of closing the canvas view corresponding to handwriting area 2 to be completed before creating the canvas view corresponding to handwriting area 1. Optionally, the note application also deletes the image file corresponding to picture 1 (picture 14) and generates an image file corresponding to picture 2 (picture 17) based on handwriting area 2 (handwriting area 15).

[0269] After the handwriting area is created successfully, the canvas view corresponding to the handwriting area can receive the input event corresponding to the handwriting input and draw the handwriting track (handwriting input content) according to the user's handwriting input (input event).

[0270] S1411. After the handwriting area 1 is successfully created, the canvas view corresponding to the handwriting area 1 reads the input events in the blocking queue as a consumer sub-thread of the blocking queue. The note application draws the handwriting on the canvas view according to the input events, and then deletes the processed input events in the blocking queue.

[0271] In one example, after handwriting area 1 is successfully created, that is, after the HandView corresponding to handwriting area 1 is generated, the note-taking application calls the function transferTouchEventsToHandWritingView. In this function, the HandView corresponding to handwriting area 1 acts as a consumer thread of the blocking queue to read the input events in the blocking queue (mTouchEventQueue). HandView calls the function drawTransferredEvents to consume the input events. For example, HandView draws the handwriting based on the input events. Afterwards, the note-taking application deletes the consumed input events from the blocking queue.

[0272] S1412. If the last input event in the blocking queue is a release event, and all input events in the blocking queue have been processed, the note application sets the transition layer to invisible.

[0273] Receiving a release (up) event indicates that the first stroke is completed. If all input events in the blocking queue have been processed, that is, mTouchEventQueue is empty, it means that the handwriting input on the handwriting picture has completed the handwriting trajectory drawing, which means that no handwriting input is currently occurring. After the last input event in the blocking queue is a release event and all input events in the blocking queue have been processed, handwriting area one has been successfully created, and the canvas view corresponding to handwriting area one can process the input event and draw the handwriting according to the input event, and the user's handwriting input will not be lost. The note application sets the transition layer to invisible (setSwitchSmoothHandWritingViewInVisible), and the handwriting drawn on the canvas view corresponding to handwriting area one can be displayed on the screen.

[0274] S1421: The transition layer reads input events in the blocking queue and draws handwriting tracks according to the input events.

[0275] In one example, a note-taking app calls the onTouchEvent function, within which the transition layer calls the drawOneTransferredEvents function to consume the input event. Specifically, the transition layer reads the input event from the blocking queue (before it is removed from the blocking queue by the canvas view) and draws the handwriting trajectory based on the input event.

[0276] Understandably, because creating a canvas view corresponding to a handwriting area takes a long time, the canvas view consumes the same input event in the blocking queue later than the transition layer. Therefore, the transition layer consumes the input event before the canvas view. By the time the transition layer reads the input event from the blocking queue, the canvas view has not yet dequeued it.

[0277] Furthermore, before the handwriting area is created, the handwriting generated by the handwriting input is displayed on the screen through a transition layer. This pre-drawn transition layer reduces the delay compared to drawing the handwriting trajectory on the canvas view, providing users with a more responsive experience.

[0278] It is understandable that the terminal device provided in the embodiment of the present application includes a hardware structure and / or software module for performing each function in order to realize the above functions. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiment of the present application.

[0279] The embodiment of the present application can divide the functional modules of the terminal device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, there may be other division methods.

[0280] In one example, please refer to FIG15 , which shows a possible structural diagram of a terminal device involved in the above embodiment. The terminal device 2000 includes: a processing unit 2010 , a storage unit 2020 , a display unit 2030 and a sensor unit 2040 .

[0281] The processing unit 2010 is used to control and manage the actions of the terminal device 2000. The storage unit 2020 is used to store the program code and data of the terminal device 2000. The processing unit 2010 calls the program code stored in the storage unit 2020 to execute the various steps in the above method embodiment. The display unit 2030 is used to display the user interface of the terminal device 2000; for example, the application interface of the note-taking application. The sensor unit 2040 is used to detect user input, such as detecting user handwriting input; and is also used to detect changes in the terminal device's shape, such as detecting rotation, folding, and unfolding of the terminal device.

[0282] Of course, the unit modules in the terminal device 2000 include, but are not limited to, the processing unit 2010, storage unit 2020, display unit 2030, and sensor unit 2040. For example, the terminal device 2000 may also include a communication unit, a power supply unit, an audio unit, etc. The communication unit is used for the terminal device 2000 to communicate with other electronic devices; the power supply unit is used to supply power to the terminal device 2000; and the audio unit is used to play audio.

[0283] The processing unit 2010 may be a processor or controller, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The storage unit 2020 may be a memory, etc. The display unit 2030 may be a display screen, etc. The sensor unit 2040 may include a touch sensor, a pressure sensor, an acceleration sensor, a gyroscope sensor, etc.

[0284] For example, the processing unit 2010 is a processor (such as the processor 110 shown in Figure 8), the storage unit 2020 can be a memory (such as the internal memory 121 shown in Figure 8), the display unit 2030 can be a display screen (such as the display screen 192 shown in Figure 8), and the sensor unit 2040 can be a sensor module (such as the sensor module 180 shown in Figure 8). The terminal device 2000 provided in the embodiment of the present application can be the terminal device 100 shown in Figure 8. Among them, the above-mentioned processor, memory, display screen, sensor, etc. can be connected together, for example, via a bus. The processor calls the program code stored in the memory to execute the various steps in the above method embodiment.

[0285] An embodiment of the present application also provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected through lines. For example, the interface circuit can be used to receive signals from other devices (such as the memory of an electronic device). For another example, the interface circuit can be used to send signals to other devices (such as a processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the terminal device can execute the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiment of the present application.

[0286] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes the various functions or steps executed by the terminal device in the above-mentioned method embodiment.

[0287] An embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to execute the functions or steps executed by the terminal device in the above method embodiment.

[0288] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0289] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0290] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0291] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0292] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0293] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for displaying input content, applied to an electronic device, characterized in that: The method comprises: Displaying a first interface; the first interface includes a first text area, and the first text area includes first text content; In response to a first operation on a first option in the first interface, a first view is created and a second interface is displayed; the second interface includes a first handwriting area and a second text area, the second text area includes the first text content, and the first view corresponds to the first handwriting area; In response to a first handwriting input in the first handwriting area, the first handwriting input content is displayed in the first handwriting area; In response to a second operation on the second interface, the first view is deleted and a third interface is displayed; the third interface includes a first handwriting display area and a third text area, the first handwriting display area includes the first handwriting input content, and the third text area includes the first text content; In response to a third operation in the first handwriting display area of ​​the third interface, a second view is created and a fourth interface is displayed; the fourth interface includes the first handwriting area and the second text area, and the second view corresponds to the first handwriting area.

2. The method according to claim 1, characterized in that The third operation includes a second handwriting input, and the method further includes: In response to the second handwriting input in the first handwriting display area of ​​the third interface, the second handwriting input content is displayed in the first handwriting area of ​​the fourth interface.

3. The method according to claim 2, characterized in that The method further comprises: In response to a second handwriting input in the first handwriting display area of ​​the third interface, setting a preset layer to be visible; After the second view is created, the preset layer is set to be invisible; when the preset layer is visible, the fourth interface is generated based on the preset layer.

4. The method according to claim 3, characterized in that Preset layer, the method further includes: In response to the second handwriting input in the first handwriting display area of ​​the third interface, the second handwriting input content is generated in the preset layer, and the preset layer is located above the layer where the second view is located.

5. The method according to claim 4, characterized in that Setting the preset layer to be invisible includes: After the second handwriting input content is generated in the layer where the second view is located, and when no handwriting input is occurring, the preset layer is set to be invisible.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: When the second interface is displayed, in response to a change in the form of the electronic device, deleting the first view; After deleting the first view, a third view is created and a fifth interface is displayed; the fifth interface includes a second handwriting area and a fourth text area, the third view corresponds to the second handwriting area, the second handwriting area includes the first handwriting input content, and the fourth text area includes the first text content.

7. The method according to claim 6, characterized in that In response to the electronic device changing its form, before deleting the first view, the method further includes: generating a first image based on the first view; The creating of the third view includes: The third view is generated according to the first image and the screen display area of ​​the electronic device after the shape change occurs.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: When the third interface is displayed, in response to the electronic device undergoing a morphological change, a sixth interface is displayed; the sixth interface includes a second handwriting display area and a fifth text area, the second handwriting display area includes the first handwriting input content, and the fifth text area includes the first text content; in the process from the electronic device undergoing a morphological change to the display of the sixth interface, no view is deleted or created.

9. The method according to claim 1, characterized in that The third operation includes a click operation, and the fourth interface is the same as the second interface.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: In response to a second operation on the second interface, a second image is created; the second image corresponds to the first handwriting display area in the third interface.

11. The method according to any one of claims 1 to 10, characterized in that: The third interface further includes a third handwriting area, the third handwriting area includes third handwriting input content, the fourth interface further includes a third handwriting display area, the third handwriting display area includes the third handwriting input content, and the method further includes: In response to a third operation in the first handwriting display area of ​​the third interface, a third image is created, where the third image corresponds to the third handwriting display area.

12. The method according to any one of claims 1 to 11, characterized in that The first handwriting display area and the second handwriting area are the same display area on the screen.

13. The method according to any one of claims 1 to 12, characterized in that: The first text content includes at least one of text, pictures, audio, tables, and hyperlinks.

14. The method according to any one of claims 1 to 13, characterized in that The second operation on the second interface includes: a click operation on an area outside the first handwriting area in the second interface.

15. The method according to claim 7, characterized in that The morphological change includes: rotation, folding or unfolding.

16. An electronic device, characterized in that: include: one or more processors; Display screen; Memory; One or more computer programs are stored in the memory, and the one or more computer programs include instructions. When the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 15.

17. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 15 is implemented.

18. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images