Handwritten content display method and electronic equipment

Abstract

The invention discloses a handwritten content display method and electronic equipment, relates to the technical field of terminals, and can reduce the display time delay of handwritten content in the process that a user writes the content on a content interface, improve the display chirality and improve the use experience of the user. In the scheme, the electronic equipment can create a system-level transparent image layer, such as a global image layer, which is independent of the image layer where the content interface is located and overlaid on the image layer where the content interface is located when a user carries out handwriting operations such as scribing, writing, circling and marking on the content interface, and carries out report point prediction based on the handwriting operations of the user and a historical writing track; and drawing a corresponding writing track on the global layer based on the touch report point corresponding to the handwriting operation of the user and the predicted report point.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a method for displaying handwritten content and an electronic device. Background Art

[0002] With the rapid development of electronic equipment hardware technology, software technology and application display technology, in order to provide users with a better user experience, more and more electronic devices support users to input information such as drawing, drawing, marking, and annotating on the content interface (such as application interface) displayed on them.

[0003] Take the user drawing on the content interface as an example. When receiving the user's drawing operation on the content interface, the electronic device usually extracts the point information of the drawing operation and generates an input event. It then performs subsequent interface drawing, rendering and synthesis according to the input event and finally displays it. In this case, the information drawn by the user on the content interface at a certain moment will often take a period of time to be finally displayed on the content interface. The above display lag will bring a bad user experience. Especially for continuous touch screen interaction processes such as drawing, drawing lines, and marking, users can also clearly perceive very small display lags, which will greatly affect the hand tracking of the device display. For example, Figure 1 As shown, when the user has drawn to point B, the electronic device may update and display the content drawn by the user at point A. Therefore, how to reduce the display delay of the content handwritten by the user on the content interface is an urgent problem to be solved. Summary of the invention

[0004] The present application provides a method for displaying handwritten content and an electronic device, which can reduce the display delay of handwritten content when a user writes handwritten content on a content interface, improve the hand tracking of the display, and enhance the user experience.

[0005] In order to achieve the above purpose, this application adopts the following technical solutions:

[0006] In a first aspect, a method for displaying handwritten content is provided, the method being applied to an electronic device, such as a first electronic device, the method comprising: when the first electronic device displays a content interface, in response to an operation by a user triggering a handwriting function, creating a global layer superimposed on an application layer corresponding to the content interface, wherein the global layer is a transparent layer; the first electronic device determines a corresponding touch point according to the handwriting operation of the user on the content interface; the first electronic device draws a corresponding writing track on the global layer according to the touch point; the first electronic device synthesizes the above-mentioned application layer and the global layer and displays a second interface, wherein the second interface includes the content interface and the writing track.

[0007] Exemplarily, the content interface may be an interface of any application or applet that supports user handwriting, such as a memo interface, a drawing interface, a handwriting interface, and the like.

[0008] Exemplarily, the first electronic device can draw a writing track in a global layer through a global drawing service. Exemplarily, the above handwriting operation can include but is not limited to the user's operation on the content interface through a finger or a touch auxiliary tool, and the touch auxiliary tool can include but is not limited to any of the following tools: a stylus, a touch pen, and a touch ball.

[0009] Exemplarily, the handwriting operation may include but is not limited to any of the following operations performed by the user on the content interface: drawing, writing, circling, and marking.

[0010] In the solution provided in the first aspect above, when the user performs handwriting operations such as drawing, writing, circling, and marking on the content interface, the first electronic device can create a system-level transparent layer, such as a global layer, which is independent of the layer where the content interface is located and superimposed on the layer where the content interface is located, and draws the corresponding writing track on the global layer based on the touch reporting point corresponding to the user's handwriting operation. In the above process, since only the writing track is drawn on the global layer and the drawn content is simple, the time required for each link in the drawing process will be shortened, and the hand-following performance of the writing track display will be improved.

[0011] In addition, since the global layer created by the first electronic device is a system-level layer, each application developer does not need to perform additional development and upgrades for their own application programs, and the problem of poor chirality of the interface display can be solved through the system-level general solution.

[0012] Furthermore, no matter which application's content interface the user uses to input content, the processing steps and implementation details of the system-level general solution are consistent.

[0013] Also, in the above process, since the global layer is a transparent layer independent of the layer where the content interface is located and superimposed on the layer where the content interface is located, from the user's visual point of view, the writing track seen is like writing directly on the content interface, and the user's visual experience will be very good.

[0014] As a possible implementation, the first electronic device draws the corresponding writing track on the global layer according to the touch point reporting, including: the first electronic device determines multiple predicted reporting points according to the touch point reporting and the historical writing track; the first electronic device draws the corresponding writing track on the global layer according to the touch point reporting and the multiple predicted reporting points. Based on this, since the writing track drawn on the global layer will be drawn in advance with reference to the predicted reporting points, the hand-following performance of the writing track display can be further improved.

[0015] As a possible implementation, the method further includes: the first electronic device draws a corresponding writing track on the application layer according to the handwriting operation. Based on this, the conventional mechanism of drawing the handwriting track on the application layer can be retained without affecting the existing processing mechanism; in addition, since the writing track on the application layer is drawn according to the actual sliding path of the stylus or the user's finger on the screen, the accuracy of the handwriting track finally displayed can also be improved.

[0016] As a possible implementation method, the above content interface is generated based on the projection data sent by the second electronic device, and the above method also includes: the first electronic device sends operation information corresponding to the handwriting operation to the second electronic device, so that the second electronic device draws the corresponding application layer according to the operation information; the first electronic device receives interface data from the second electronic device, wherein the interface data includes information about the writing track corresponding to the above handwriting operation. Based on this, the application scenario of this solution can be extended to the projection scenario, solving the problem of poor hand tracking of the writing track display in the projection scenario.

[0017] As a possible implementation method, the above historical writing track is drawn on the application layer, and the above first electronic device draws the corresponding writing track on the global layer according to the touch point reporting and multiple predicted point reporting, including: the first electronic device uses a brush mode consistent with the historical writing track, and draws the corresponding writing track on the global layer according to the touch point reporting and multiple predicted point reporting. Based on this, since the historical writing track drawn on the application layer is drawn according to the actual sliding route of the stylus or the user's finger on the screen, the point reporting prediction based on the historical writing track can reduce the deviation between the predicted point reporting and the actual touch point reporting in the future, thereby improving the accuracy of the handwriting track finally displayed.

[0018] As a possible implementation, the method further includes: the first electronic device obtains brush mode parameters corresponding to the historical writing track, and the brush mode parameters include one or more of the following: brush type, line color, line width, and line texture. Based on this, the consistency of the writing track on the global layer and the writing track on the application layer can be guaranteed, so as to maintain the consistency of the display effect of the writing track drawn on the global layer and the writing track drawn on the application layer, thereby improving the user experience.

[0019] As a possible implementation, the first electronic device obtains the brush mode parameters corresponding to the latest writing track in the historical writing track, including: the first electronic device performs image analysis on the latest writing track in the historical writing track to obtain the brush mode parameters corresponding to the latest writing track. Based on this, the brush mode used by the historical writing track can be obtained smoothly and accurately.

[0020] Exemplarily, the first electronic device may first determine the latest writing trajectory in the historical writing trajectory, such as by comparing the current frame image with the previous frame image on the application layer to determine the latest writing trajectory on the application layer. Then, the first electronic device may perform image analysis on the identified latest writing trajectory to obtain one or more information corresponding to the latest writing trajectory, such as brush type, line color, line width, line texture, etc.

[0021] As a possible implementation, the method further includes: in the process of drawing the writing track on the global layer, the first electronic device erases the second handwriting track on the global layer corresponding to the first handwriting track drawn on the application layer. Based on this, the complexity of subsequent global layer drawing can be reduced without affecting the user's normal viewing of the writing tracks corresponding to all handwriting operations, further improving the drawing of the writing track on the global layer, and further improving the hand tracking of the writing track display.

[0022] As a possible implementation, the method further includes: in the process of drawing the writing track on the global layer, the first electronic device erases the handwriting track drawn on the global layer before the preset time period. Based on this, the complexity of subsequent global layer drawing can be reduced without affecting the user's normal viewing of the writing track corresponding to all handwriting operations, and the drawing of the writing track on the global layer can be further improved, thereby further improving the hand tracking of the writing track display.

[0023] In a second aspect, an electronic device is provided, which includes: a display unit for displaying a content interface; a global drawing unit for: when the display unit displays the content interface, in response to an operation of a user triggering a handwriting function, creating a global layer superimposed on an application layer corresponding to the content interface, wherein the global layer is a transparent layer; determining a corresponding touch point according to the user's handwriting operation on the content interface; and drawing a corresponding writing track on the global layer according to the touch point; a layer synthesis unit for synthesizing the above-mentioned application layer and the global layer and then displaying a second interface through the display unit, wherein the second interface includes the content interface and the writing track.

[0024] Exemplarily, the content interface may be an interface of any application or applet that supports user handwriting, such as a memo interface, a drawing interface, a handwriting interface, and the like.

[0025] Exemplarily, the first electronic device may draw the writing track in the global layer through the global drawing service.

[0026] Exemplarily, the handwriting operation may include but is not limited to the user's operation on the content interface through fingers or touch-assistive tools. The touch-assistive tools may include but are not limited to any of the following tools: a stylus, a touch pen, and a touch ball.

[0027] Exemplarily, the handwriting operation may include but is not limited to any of the following operations performed by the user on the content interface: drawing, writing, circling, and marking.

[0028] In the solution provided in the first aspect above, the first electronic device can create a system-level transparent layer, such as a global layer, which is independent of the layer where the content interface is located and superimposed on the layer where the content interface is located when the user performs handwriting operations such as drawing, writing, circling, and marking on the content interface, and draw the corresponding writing track on the global layer based on the touch report point corresponding to the user's handwriting operation. In the above process, since only the writing track is drawn on the global layer and the drawn content is simple, the time required for each link in the drawing process will be shortened, and the hand-following property of the writing track display will be improved. In addition, since the global layer created by the first electronic device is a system-level layer, there is no need for each application developer to perform additional development and upgrade for their respective applications, and the problem of poor hand-following property of the interface display can be solved through the system-level general solution. Moreover, no matter which application content interface the user inputs content on, the processing process points and implementation details of the system-level general solution are consistent. And, in the above process, since the global layer is a transparent layer independent of the layer where the content interface is located and superimposed on the layer where the content interface is located, from the user's visual effect, the writing track seen is like writing directly on the content interface, and the user's visual experience will be very good.

[0029] As a possible implementation, the global drawing unit draws the corresponding writing track on the global layer according to the touch point reporting, including: the global drawing unit determines multiple predicted reporting points according to the touch point reporting and the historical writing track; the global drawing unit draws the corresponding writing track on the global layer according to the touch point reporting and the multiple predicted reporting points. Based on this, since the writing track drawn on the global layer will be drawn in advance with reference to the predicted reporting points, the hand-following performance of the writing track display can be further improved.

[0030] As a possible implementation, the electronic device further includes: an application drawing unit, which is used to draw a corresponding writing track on the application layer according to the handwriting operation. Based on this, the conventional mechanism of drawing the handwriting track on the application layer can be retained without affecting the existing processing mechanism; in addition, since the writing track on the application layer is drawn according to the actual sliding path of the stylus or the user's finger on the screen, the accuracy of the handwriting track finally displayed can also be improved.

[0031] As a possible implementation method, the above content interface is generated based on the projection data sent by the second electronic device, and the above electronic device also includes: a communication unit, which is used to: send operation information corresponding to the handwriting operation to the second electronic device, so that the second electronic device can draw the corresponding application layer according to the operation information; and receive interface data from the second electronic device, wherein the interface data includes information about the writing track corresponding to the handwriting operation. Based on this, the application scenario of this solution can be extended to the projection scenario, solving the problem of poor hand tracking of the writing track display in the projection scenario.

[0032] As a possible implementation method, the above historical writing track is drawn on the application layer, and the above global drawing unit draws the corresponding writing track on the global layer according to the touch point reporting and multiple predicted point reporting, including: the global drawing unit adopts a brush mode consistent with the historical writing track, and draws the corresponding writing track on the global layer according to the touch point reporting and multiple predicted point reporting. Based on this, since the historical writing track drawn on the application layer is drawn according to the actual sliding route of the stylus or the user's finger on the screen, the point reporting prediction based on the historical writing track can reduce the deviation between the predicted point reporting and the actual touch point reporting in the future, thereby improving the accuracy of the handwriting track finally displayed.

[0033] As a possible implementation, the global drawing unit is further used to obtain brush mode parameters corresponding to the historical writing track, and the brush mode parameters include one or more of the following: brush type, line color, line width, and line texture. Based on this, the consistency of the writing track on the global layer and the writing track on the application layer can be guaranteed, so as to maintain the consistency of the display effect of the writing track drawn on the global layer and the writing track drawn on the application layer, thereby improving the user experience.

[0034] As a possible implementation method, the global drawing unit obtains the brush mode parameters corresponding to the latest writing track in the historical writing track, including: the global drawing unit performs image analysis on the latest writing track in the historical writing track to obtain the brush mode parameters corresponding to the latest writing track. Based on this, the brush mode used by the historical writing track can be obtained smoothly and accurately.

[0035] Exemplarily, the global drawing unit may first determine the latest writing trajectory in the historical writing trajectory, such as by comparing the current frame image with the previous frame image on the application layer to determine the latest writing trajectory on the application layer. Then, the global drawing unit may perform image analysis on the identified latest writing trajectory to obtain one or more information corresponding to the latest writing trajectory, such as the brush type, line color, line width, line texture, etc.

[0036] As a possible implementation, the global drawing unit is further used to: during the process of drawing the writing track on the global layer, erase the second handwriting track on the global layer corresponding to the first handwriting track drawn on the application layer. Based on this, the complexity of subsequent global layer drawing can be reduced without affecting the user's normal viewing of the writing tracks corresponding to all handwriting operations, further improving the drawing of the writing tracks on the global layer, and further improving the hand tracking of the writing track display.

[0037] As a possible implementation, the global drawing unit is further used to: in the process of drawing the writing track on the global layer, erase the handwriting track drawn on the global layer before the preset time period. Based on this, the complexity of subsequent global layer drawing can be reduced without affecting the user's normal viewing of the writing track corresponding to all handwriting operations, and the drawing of the writing track on the global layer can be further improved, thereby further improving the hand tracking of the writing track display.

[0038] In a third aspect, an electronic device is provided, comprising: a display screen for displaying content, such as a content interface and a handwriting track; a memory for storing computer program instructions; and a processor for executing the computer program instructions to support the electronic device to implement a method in any possible implementation of the first aspect.

[0039] In a fourth aspect, a communication system is provided, which includes a first electronic device and a second electronic device, wherein the second electronic device projects a content interface to the first electronic device, and the first electronic device is used to implement a method as in any possible implementation manner of the first aspect.

[0040] According to a fifth aspect, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method in any possible implementation of the first aspect is implemented.

[0041] According to a sixth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to implement a method as in any possible implementation of the first aspect.

[0042] In a seventh aspect, a chip system is provided, the chip system comprising a processing circuit and a storage medium, wherein the storage medium stores computer program instructions; when the computer program instructions are executed by the processor, the method in any possible implementation of the first aspect is implemented. The chip system may be composed of a chip, or may include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of a conventional interface display effect;

[0044] Figure 2 A schematic diagram of an interface display process provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of the effect of a time prediction mechanism provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0048] Figure 6 Schematic diagrams of two application scenarios provided for embodiments of the present application;

[0049] Figure 7 A flow chart of a method for displaying handwritten content provided in an embodiment of the present application;

[0050] Figure 8 Two operation diagrams for triggering handwriting functions provided in the embodiments of the present application;

[0051] Fig. 9 A schematic diagram of a scenario in which a user performs a handwriting operation provided in an embodiment of the present application;

[0052] Fig.10 A schematic diagram of a display process of handwritten content provided in an embodiment of the present application;

[0053] Fig.11 A schematic diagram of another process of displaying handwritten content provided in an embodiment of the present application;

[0054] Fig.12 A schematic diagram of a solution for erasing a writing track on a global layer provided in an embodiment of the present application;

[0055] Fig.13 A schematic diagram of another method for erasing writing tracks on a global layer provided in an embodiment of the present application;

[0056] Fig.14 A schematic diagram for comparing chirality effects provided in an embodiment of the present application;

[0057] Fig.15 A schematic diagram for comparing brush learning effects provided in an embodiment of the present application;

[0058] Fig.16 A schematic diagram of a process for determining a new writing track on an application layer provided in an embodiment of the present application;

[0059] Fig.17 A schematic diagram of a method for obtaining the line color corresponding to a newly added writing track provided in an embodiment of the present application;

[0060] Fig.18 A schematic diagram of a method for obtaining a line width corresponding to a newly added writing track provided in an embodiment of the present application;

[0061] Fig.19 A schematic diagram of another method for obtaining the line width corresponding to a newly added writing track provided in an embodiment of the present application;

[0062] Fig. 20 A schematic diagram of a method for obtaining a line texture corresponding to a newly added writing track provided in an embodiment of the present application;

[0063] Fig.21 A schematic diagram of a display process of handwritten content in a multi-device scenario provided by an embodiment of the present application;

[0064] Fig. 22 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0066] In the following, the terms "first", "second", etc. are only used to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. For example, if the described object is a "field", the ordinal number before the "field" in the "first field" and the "second field" does not limit the position or order between the "fields", and the "first" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of the "first field" and the "second field". For another example, if the described object is a "level", the ordinal number before the "level" in the "first level" and the "second level" does not limit the priority between the "levels". For another example, the number of described objects is not limited by the ordinal number, and can be one or more. Taking the "first device" as an example, the number of "devices" can be one or more. In addition, the objects modified by different prefixes can be the same or different. For example, if the described object is a "device", the "first device" and the "second device" can be the same type of device or different types of devices. For another example, if the described object is "information", the "first information" and the "second information" can be information of the same content or information of different contents. In summary, the use of prefixes such as ordinal numbers to distinguish the described objects in the embodiments of the present application does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and no unnecessary limitation should be constituted due to the use of such prefixes.

[0067] Furthermore, in the embodiments of the present application, "connection" may be a direct connection or an indirect connection; in addition, it may refer to an electrical connection or a communication connection; for example, the connection between two electrical components A and B may refer to a direct connection between A and B, or may refer to an indirect connection between A and B through other electrical components or connecting media, or may refer to an indirect connection between A and B through other communication devices or communication media, as long as communication between A and B can be achieved.

[0068] When a user draws, draws, marks, or inputs information by hand on the content interface of an electronic device, the electronic device will continuously perform handwriting input according to the user's operation information detected. Figure 2 After the touch bottom layer processing, input event processing, drawing, rendering, synthesis and other stages shown, the user's handwritten content is finally displayed at the corresponding position on the content interface.

[0069] The touch bottom processing stage is mainly used to obtain the user's operation information based on the original data collected by the electronic device, such as the original data collected by the electronic device through the touch sensor on the display screen, where the operation information may include but is not limited to the user's touch position on the touch screen and the touch pressure. As an example, Figure 2As shown, the electronic device can obtain the user's operation information by processing the original data collected by the electronic device based on a preset processing algorithm.

[0070] The input event processing phase is mainly used to convert the user's operation information into specific input events and distribute the input events. As an example, Figure 2 As shown, the electronic device can convert the user's operation information into a specific input event through InputReader, and then distribute the subsequent input event through InputDispatcher, such as distributing it to a corresponding application activity (Activity).

[0071] The drawing phase is mainly used to draw the interface layer to be displayed. As an example, Figure 2 As shown, the electronic device can draw the interface layer to be displayed by executing the doFrame command. The drawing stage may include the following processes: measure → layout → draw. Measurement is used to obtain and save the size of each View and ViewGrop corresponding to the content interface according to the settings of the control properties in the interface layout file and the code. Layout is used to determine the display position of each control on the content interface based on the information obtained by measurement. Drawing is used to draw the corresponding interface layer on the canvas (canvas) after determining the display position of each control, and construct a drawing instruction.

[0072] The rendering stage is mainly used to perform interface rendering such as brightness, contrast, saturation, etc. on the drawn interface layer. As an example, Figure 2 As shown, the electronic device can execute the DrawFrame command to perform interface rendering on the drawn interface layer and then cache it into the buffer area by calling the eglSwapBuffersWithDamageKHR function.

[0073] The synthesis stage is mainly used to combine multiple rendered layers, such as the rendered layers obtained from the cache, in an appropriate overlapping order to obtain a complete image frame. As an example, Figure 2 As shown, the electronic device can enter the multi-layer composition logic by calling the onMessageReceived function. For example, after entering the multi-layer composition logic, the electronic device can obtain the number of layers to be composed and the properties of each layer by calling the setUpHWComposer function and update the layer list and layer properties. Furthermore, the electronic device can compose multiple layers by calling the doComposition function.

[0074] After completing the layer synthesis, the electronic device can call the display driver (such as a liquid crystal display (LCD) driver, etc.) through the display stocker status (DSS) function to send the synthesized layer to the display screen for display according to the specified display area.

[0075] For example, please refer to Figure 2 , Figure 2 FIG. 1 shows a schematic diagram of an interface display process provided by an embodiment of the present application. Figure 2 As shown, assuming that the moment when the user touches the display screen of the electronic device is T1, the moment when the electronic device completes the extraction of operation information is T2, the moment when the input event is generated is T3, the moment when the input event distribution is completed is T4, the moment when the layer drawing is completed is T5, the moment when the layer rendering is completed is T6, the moment when the layer synthesis is completed is T7, plus the time consumed in the display process, the final moment displayed on the display screen is T8. The time consumed in each of the above stages will affect the chirality of the electronic device display. Based on this, based on the existing interface display mechanism, the content written by the user on the content interface at time T1 will not be displayed at the corresponding position of the content interface until after the duration of T8-T1, and the chirality of the display is very poor.

[0076] Among them, the chirality can be represented by the time delay from the user inputting on the touch screen to the touch screen displaying the corresponding writing trajectory; the smaller the time delay, the better the chirality; the larger the time delay, the worse the chirality.

[0077] In order to reduce the display delay of the electronic device, as a possible implementation method, the hand-tracking performance of the writing track display can be improved by increasing the display refresh rate or processing frequency of the electronic device. However, the above method of increasing the display refresh rate or processing frequency will increase the power consumption of the electronic device, and compared with the large time consumption of each link in the writing track drawing process, the space that this method can make up for the delay is limited, so it cannot well improve the hand-tracking performance of the writing track display.

[0078] In order to reduce the display latency of electronic devices, as another possible implementation method, the developer of an application that supports users to input information by touch can improve the application so that it has the function of drawing the user's touch trajectory in advance according to the point prediction mechanism. For example, based on the point prediction mechanism, the electronic device can predict the possible touch points in the future or a preset number of possible touch points in the future based on the real continuous historical touch trajectory input by the user, and draw, render and synthesize in advance based on the predicted points, so as to reduce the display latency of the electronic device and improve the hand tracking performance of the electronic device display. For example, if Figure 3As shown, compared with the interface display process without using the point prediction mechanism, the interface display process using the point prediction mechanism can greatly reduce the hand tracking delay of the interface display.

[0079] However, the above method requires each application developer to develop and upgrade their own application, which is a huge workload. In addition, since the application development and upgrade work of each application developer is carried out independently, the specific implementation details and algorithm accuracy of the reporting point prediction mechanism may vary, which will cause users to feel different display tracking effects when using different applications on electronic devices.

[0080] In order to solve the problems existing in the above-mentioned scheme, an embodiment of the present application provides a method for displaying handwritten content. Based on this method, the electronic device can provide a system-level transparent layer that is independent of the layer where the content interface is located and superimposed on the layer where the content interface is located. The transparent layer is used to draw the writing trajectory determined by the system based on the operation information of the user's operation and the predicted reporting point. As a general system-level solution, this solution can solve the problem of poor chirality of the interface display without the need for application developers to develop and upgrade their respective applications. In addition, since this solution is a general system-level solution, no matter which application's content interface the user inputs content on, the processing process and implementation details are consistent, so the display chirality effect felt by the user when using any application is the same.

[0081] Among them, the electronic device described in the embodiment of the present application can be an electronic device with a touch screen. For example, the electronic device can include but is not limited to a smart phone, a netbook, a tablet computer, a handwriting tablet, a smart watch, a smart bracelet, a phone watch, a smart camera, a PDA, a car computer, a personal computer (personal computer, PC), a personal digital assistant (personal digital assistant, PDA), a portable multimedia player (portable multimedia player, PMP), (augmented reality, AR) / virtual reality (virtual reality, VR) equipment, a smart TV, a projection device or a somatosensory game console in a human-computer interaction scene, etc. Alternatively, the electronic device can also be an electronic device with a touch screen of other types or structures, which is not limited in this application.

[0082] As an example, see Figure 4 , Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0083] like Figure 4As shown, the electronic device may include a processor 410, a memory (including an external memory interface 420 and an internal memory 421), a universal serial bus (USB) interface 430, a charging management module 440, a power management module 441, a battery 442, an antenna 1, an antenna 2, a mobile communication module 450, a wireless communication module 460, an audio module 470, a speaker 470A, a receiver 470B, a microphone 470C, an earphone interface 470D, a sensor module 480, a button 490, a motor 491, an indicator 492, a camera 493, a display screen 494, etc. Among them, the sensor module 480 may include a touch sensor 480A and a pressure sensor 480B. Optionally, the sensor module 480 may also include a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, an ambient light sensor, a bone conduction sensor, etc.

[0084] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0085] The processor 410 may include one or more processing units. For example, the processor 410 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a flight controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0086] The processor 410 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 410 is a cache memory. The memory may store instructions or data that the processor 410 has just used or cyclically used. If the processor 410 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 410, and thus improves the efficiency of the system.

[0087] In some embodiments, the processor 410 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0088] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 410 may include multiple groups of I2C buses. The processor 410 may be coupled to the touch sensor 480A, the microphone, the camera 493, etc. through different I2C bus interfaces. For example, the processor 410 may be coupled to the touch sensor 480A through the I2C interface, so that the processor 410 communicates with the touch sensor 480A through the I2C bus interface to realize the touch function of the electronic device.

[0089] In an embodiment of the present application, the processor 410 can obtain touch operations such as clicks (such as single clicks, double clicks, etc.), sliding operations, long press operations, or drag operations performed by the user on the interface detected by the touch sensor 480A through the I2C bus interface, thereby determining the specific intention corresponding to the touch operation, and then responding to the touch operation.

[0090] The charging management module 440 is used to receive charging input from the charger. The power management module 441 is used to connect the battery 442, the charging management module 440 and the processor 410. The power management module 441 receives input from the battery 442 and / or the charging management module 440 to power the processor 410, the internal memory 421, the display screen 494, the camera assembly 493 and the wireless communication module 460.

[0091] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 450, wireless communication module 460, modem processor and baseband processor.

[0092] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0093] The mobile communication module 450 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to electronic devices. The mobile communication module 450 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 450 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 450 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 450 may be arranged in the processor 410. In some embodiments, at least some of the functional modules of the mobile communication module 450 may be arranged in the same device as at least some of the modules of the processor 410.

[0094] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 470A, a receiver 470B, etc.), or displays an image or video through a display screen 494. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 410 and be set in the same device as the mobile communication module 450 or other functional modules.

[0095] The wireless communication module 460 can provide wireless communication solutions including wireless local area networks (WLAN) (such as WiFi networks), Bluetooth BT, global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to electronic devices. The wireless communication module 460 can be one or more devices integrating at least one communication processing module. The wireless communication module 460 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 410. The wireless communication module 460 can also receive the signal to be sent from the processor 410, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0096] In some embodiments, the antenna 1 of the electronic device is coupled to the mobile communication module 450, and the antenna 2 is coupled to the wireless communication module 460, so that the electronic device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).

[0097] In some embodiments of the present application, the electronic device can exchange information with other devices through the wireless communication module 460 based on wireless communication technology, such as receiving projection data, sending user input events received, receiving instructions in response to user operations, etc.

[0098] The electronic device implements the display function through a GPU, a display screen 494, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 494 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 410 may include one or more GPUs that execute program instructions to generate or change display information.

[0099] In the embodiment of the present application, the electronic device can use the GPU to draw and render the layers on the content interface of the application. Also, the electronic device can use the GPU to draw and render the content on the system-level transparent layer (such as lines or patterns drawn by the user, text written, etc.).

[0100] The display screen 494 is used to display images, videos, etc. The display screen 494 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 494, where N is a positive integer greater than 1.

[0101] In the embodiment of the present application, the electronic device can display the content interface and information such as lines or patterns drawn by the user, text written by the user, etc. through the display screen 494.

[0102] The electronic device can realize the shooting function through ISP, camera component 493, video codec, GPU, display screen 494 and application processor.

[0103] The external memory interface 420 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 410 through the external memory interface 420 to implement a data storage function.

[0104] Internal memory 421 may be used to store computer executable program code. For example, the computer program may include an operating system program and an application program. The operating system may include but is not limited to apple OS and other operating systems. Among them, the executable program code includes instructions. The processor 410 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 421. The internal memory 421 may include a program storage area and a data storage area. Among them, the program storage area may store the operating system, at least one application required for a function, etc. The data storage area may store data created during the use of the electronic device (such as user operation information, input events, drawn layers, etc.). In addition, the internal memory 421 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 410 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 421, and / or the instructions stored in the memory provided in the processor.

[0105] The electronic device can realize audio functions through the audio module 470, the speaker 470A, the receiver 470B, the microphone 470C and the application processor, such as music playing and recording. For the specific working principles and functions of the audio module 470, the speaker 470A, the receiver 470B and the microphone 470C, reference can be made to the introduction in the conventional technology.

[0106] Touch sensor 480A is also called a "touch panel". Touch sensor 480A can be set on display screen 494, and touch sensor 480A and display screen 494 form a touch screen, also called a "touch screen". Touch sensor 480A is used to detect touch operations acting on or near it. The touch sensor can pass the collected original information used to represent touch position, touch force, touch angle, touch area and touch duration to the processor for the processor to perform subsequent operation information acquisition, input event generation and subsequent processing. In addition, the processor can provide visual output related to the touch operation through display screen 494. In other embodiments, touch sensor 480A can also be set on the surface of the electronic device, which is different from the position of display screen 494.

[0107] The pressure sensor 480B is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 480B can be set on the display screen 494. There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor, the capacitance between the electrodes changes. The electronic device determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 494, the electronic device can detect the touch force of the touch operation through the pressure sensor. The electronic device can also calculate the touch position based on the detection signal of the pressure sensor 480B. In some embodiments, the pressure sensor 480B can be used to detect information such as the touch position, touch force, touch angle, touch area and touch duration of the stylus.

[0108] In the embodiment of the present application, the touch operation detected by the touch sensor 480A can be an operation performed by a user on the touch screen through a finger, or an operation performed by a user on the touch screen using a touch auxiliary tool such as a stylus, a touch pen, or a touch ball. This application does not limit this.

[0109] In addition, for the introduction of hardware such as buttons 490, motor 491, and indicator 492, reference may be made to conventional technologies and will not be repeated in the embodiments of the present application.

[0110] It is understandable that this application Figure 4 The illustrated structure does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0111] As a possible example, see Figure 5 , Figure 5 FIG. 1 shows a schematic diagram of the software structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the operating system of the electronic device may include an application layer, a framework layer (framework, FWK) and a kernel layer.

[0112] The application layer can include a range of applications, such as Figure 5The handwriting application, drawing application, memo, and screen projection application shown. Of course, the applications in the application layer may include but are not limited to word processing systems (wordprocessing system, WPS), mailboxes, cameras, galleries, calendars, calls, maps, navigation, Bluetooth, music, videos, short messages and other applications. For the convenience of description, the application will be referred to as the application below. The above applications can be native applications (such as applications integrated in the operating system) or functions, or they can be third-party applications (such as applications downloaded and installed by users through the application store), which are not limited in the embodiments of this application.

[0113] The framework layer is used to provide application programming interface (API) and programming framework for applications in the application layer. Figure 5 As shown, the framework layer can provide window management service (WindowManagerService, WMS), activity management service (activity management service, AMS), input synthesis (InputFlinger) service, global drawing service and surface synthesis (SurfaceFlinger) service.

[0114] Among them, the window management service is used to provide window management, such as window creation, window activation, window deletion, window movement, and management of window position / size / style, etc.

[0115] In an embodiment of the present application, the window management service can also be used to manage window information. The window information may include, but is not limited to, window identification, window level, identification of the interface displayed in the window, virtual screen object corresponding to the window, and other information. Exemplarily, the virtual screen object corresponding to the window is the displayId of the virtual screen where the window is located, for example, displayId can be a parameter of type int.

[0116] Among them, in a single device scenario (such as Figure 6 In the scenario shown in (a) above, the window management service of the electronic device can be used to manage the window information of all windows on the electronic device and the corresponding virtual screen objects.

[0117] In a multi-device scenario (such as Figure 6 In the scenario shown in (b) of FIG. 1 , the window management service of the electronic device is used to manage not only the window information and corresponding displayId of all windows on the electronic device, but also the window information and corresponding displayId of all windows on other devices cooperating with the electronic device.

[0118] The activity management service is responsible for managing activities, starting, switching, and scheduling various components in the system, as well as managing and scheduling applications. Specifically, AMS defines data classes for saving processes, activities, and tasks. Among them, the data class corresponding to the process can include process file information, process memory status information, and the activities and services contained in the process. Activity information can be saved in the task stack (ActivityStack). Among them, ActivityStack is used to uniformly schedule application activities. ActivityStack can specifically save information about all running activities. ActivityStack can also save information about historically run activities.

[0119] In the embodiment of the present application, the AMS of the electronic device can schedule the Activity corresponding to the target window to respond to the user's input event according to the instruction of the WMS.

[0120] The input synthesis service is used to collect, process user operation information and distribute input events. The input synthesis service can include the following two classes: InputReader and InputDispatcher. Among them, InputReader is used to convert user operation information into specific input events, and InputDispatcher is used to distribute subsequent input events. For example, InputDispatcher can send input events to WMS through a dedicated channel between WMS and it. After receiving the input event, WMS can distribute the input event to the activity corresponding to the target window in AMS. After that, the Activity can respond to the input event.

[0121] In some embodiments of the present application, after the input synthesis service converts the user's operation information into a specific input event, the input synthesis service can also provide the input event to the global drawing service while distributing the input event, so that the global drawing service can perform subsequent point prediction, writing trajectory drawing and rendering, global layer display, etc. The global drawing service can be used to provide functions such as touch point generation, point prediction, writing trajectory drawing and rendering. Exemplarily, the global drawing service can obtain the user's operation information from the sensor driver, or the global drawing service can obtain the input event from the input synthesis service, and obtain the touch point after relevant processing. And, the global drawing service can perform subsequent point prediction based on the actual continuous touch point based on the point prediction mechanism, such as predicting possible touch points within a period of time in the future (such as 2 seconds) or predicting a possible preset number of touch points in the future (such as 5), so as to support the electronic device to perform drawing and rendering in advance based on the predicted points. Among them, the global drawing service can be used to perform global drawing according to the predicted points and temporarily store them in the global drawing buffer so that the surface synthesis service can perform subsequent synthesis.

[0122] In some embodiments, Figure 5 As shown, the global drawing service can also be used to provide a brush learning function, and provide a writing track with a corresponding display style obtained by drawing according to the learned brush mode during the global drawing process. Exemplarily, the global drawing service can learn the brush mode of the application when drawing the writing track, such as learning the brush mode in the application layer obtained by the application drawing stored in the application drawing cache, such as the brush type (for example, pen, pencil, marker, watercolor pen, oil brush, etc.), line color (for example, black, red, green, blue, purple, etc.), line width, line texture (for example, no texture, water ripple, marble, wood grain, gravel, etc.), etc., and draw the corresponding writing track according to the learned brush mode. Based on this, drawing and displaying the writing track by using a brush mode consistent with the application can not only provide users with a consistent visual experience, but also users will not perceive that the writing track is drawn on the global layer.

[0123] It should be noted that Figure 5 The application layer stored in the application drawing cache is obtained by application drawing. Figure 2 The touch bottom layer processing, input event processing, drawing, rendering and synthesis shown in the figure are stored in the application drawing cache.

[0124] The surface synthesis service is used to synthesize graphic display data from multiple sources. For example, assuming that the application layer corresponding to the content interface includes four layers: status bar, navigation bar, wallpaper, and launcher icon layer, the drawing and rendering of the above status bar, navigation bar, wallpaper, and launcher icon layer are all performed separately. The surface synthesis service is mainly used to obtain the graphic display data corresponding to the status bar, navigation bar, wallpaper, and launcher icon layer from the cache area (such as the application drawing cache), synthesize it, and send it for display.

[0125] In the embodiment of the present application, when performing surface synthesis, the surface synthesis service will also obtain the graphic display data corresponding to the global layer with the writing track drawn from the cache area (such as the global drawing cache), synthesize it with the graphic display data corresponding to the application layer in the application drawing cache, and then send it for display. Finally, the user can not only see the content interface of the application, but also the lines, patterns, texts and other information written by the user.

[0126] Of course, the framework layer can also be used to provide other management services, such as package management services, content provision services, phone management services, resource management services, notification management services, location management services, and view system services. Among them, the package management service is responsible for program management within the system. The content provision service is used to support an application to access the data of another application (such as a contact database), or to share its own data with other applications, where the data may include video, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc. The phone management service is used to provide a method for accessing the status and information related to terminal device communication; and to provide communication functions of terminal devices, such as call status management (including connection, hang up, etc.). The resource management service is used to provide access to non-code resources, such as local strings, icons, pictures, graphics, video files or layout files. The notification management service is used to provide the function of displaying customized prompt information in the status bar. The prompt information can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. The location management service is used to provide services related to processing geographic location. The view system can be used to build the display interface of the application. The view system includes visual controls such as text box controls, list controls, grid controls, button controls, and embeddable web browser controls.

[0127] The kernel layer is the layer between hardware and software. The kernel layer can include camera driver, keyboard driver, display driver, microphone driver, headphone driver, sensor driver, Bluetooth driver, etc.

[0128] In an embodiment of the present application, when a user performs a handwriting operation on the touch screen area corresponding to the content interface, the sensor driver can obtain the user's operation information based on the user's touch operation detected by the sensor (such as a touch sensor and / or a pressure sensor) set on the touch screen, and provide it to the input synthesis service and the global drawing service.

[0129] For example, in some embodiments of the present application, the input operation received by the electronic device may include but is not limited to a user's click (such as single click, double click, etc.) operation / long press operation / slide operation / drag operation / floating gesture operation on the touch screen through a finger or a touch-assisted tool such as a stylus, a stylus, or a touch ball, etc. The present application does not make specific limitations on this.

[0130] It should be noted that Figure 5 It is only an example of a software structure diagram of an electronic device, in which only the levels and software modules related to the scheme of the present application are simply listed. In practical applications, the operating system of the electronic device may also include other levels and each level may also include other software modules for implementing one or more functions or services. In this regard, the embodiments of the present application do not make specific limitations. In addition, the embodiments of the present application do not limit the specific levels of each software module, for example, the global drawing service may also be located in the system library or other levels.

[0131] It should be noted that in the method for displaying handwritten content described in the embodiment of the present application, there is no specific limitation on the source of the content interface targeted by the user's handwriting. Figure 6 In the single device scenario shown in (a) of FIG. 1 , the interface of the application installed on the electronic device (such as Figure 6 Alternatively, the content interface targeted by the user's handwriting may also be Figure 6 In the multi-device scenario shown in (b) of FIG. 1 , the second electronic device projects the screen to the content interface of the first electronic device or the extended interface of the second electronic device (such as Figure 6 The first interface shown in (b)). Wherein, Figure 6 In the multi-device scenario shown in (b), the first electronic device can display the corresponding content interface (i.e., the first interface) according to the content interface data stream from the second electronic device, and send the operation information of the user's operation on the content interface to the second electronic device, so that the second electronic device can perform a corresponding interface response. In addition, the method for displaying handwritten content described in the embodiment of the present application does not specifically limit the specific type of content interface targeted by handwriting and the content displayed thereon.

[0132] The following will describe in detail the method for displaying handwritten content provided in the embodiments of the present application in conjunction with the accompanying drawings.

[0133] In the embodiments of the present application, the electronic device can receive handwritten content of the user. For example, the user can use a stylus to handwrite content on the touch screen of the electronic device; for another example, the user can use a finger to handwrite content on the touch screen of the electronic device. The following embodiments of the present application are introduced by taking the user using a stylus to input as an example.

[0134] Please refer to Figure 7 , Figure 7 FIG. 1 shows a flow chart of a method for displaying handwritten content provided by an embodiment of the present application. Figure 7 As shown, the method for displaying handwritten content provided in the embodiment of the present application may include S701-S706:

[0135] S701: When displaying a content interface, the electronic device receives an operation of triggering a handwriting function by a user.

[0136] The content interface displayed by the electronic device may be any interface of an application or applet that supports user handwriting, such as Figure 6 The application interface or applet interface installed on the electronic device and started on the electronic device as shown in (a) of FIG. Figure 6 As shown in (b), other electronic devices are projected onto the interface of the electronic device or other electronic devices are extended to the interface of the electronic device.

[0137] Exemplarily, the user triggers the handwriting function by connecting a touch-assistive tool such as a stylus, a touch pen, or a touch ball to an electronic device, or by bringing a touch-assistive tool such as a stylus, a touch pen, or a touch ball close to the touch screen of an electronic device.

[0138] Alternatively, the user's operation of triggering the handwriting function may include but is not limited to the user's operation of turning on the handwriting function (such as the user clicking on a related button or function option), the user's operation of selecting a virtual stylus pen type, the user's operation of selecting an annotation style, or the user's operation of starting handwriting on the content interface, etc. The present embodiment of the application does not specifically limit the user's operation of triggering the handwriting function. Among them, the virtual stylus pen type may include but is not limited to a virtual pencil, a virtual pen, a virtual ballpoint pen, or a virtual marker pen, etc. The annotation style may include but is not limited to highlighting, underlining, deleting, circling, graphic marking, or text annotation, etc.

[0139] Please refer to Figure 8 , Figure 8 The following is a schematic diagram showing two operation modes of triggering the handwriting function provided in the embodiment of the present application. The operation of the user triggering the handwriting function can be: Figure 8 As shown in (a) in the figure, the user clicks the "Annotation" function button on the content interface, or Figure 8As shown in (b), the user selects line width, line color, line texture and other options and then clicks the "OK" function option.

[0140] certainly, Figure 8 As just two examples of the operating interfaces for triggering the handwriting function, the interface for setting the brush mode may also include other brush mode related options, such as line type (such as solid line, dotted line, dash line, etc.), line style (such as straight line, curve, etc.), pen type (such as pencil, fountain pen, ballpoint pen, marker, etc.), annotation style (such as drawing, highlighting, underlining, deleting, circling, graphic marking, text annotation, etc.), and one or more other options.

[0141] in addition, Figure 8 As only two examples of operations for triggering the handwriting function, the embodiments of the present application do not limit the specific triggering operations. For example, the operation for triggering the handwriting function may also be an operation in which the user starts sliding on the touch screen area corresponding to the content interface. For another example, the operation for triggering the handwriting function may also be a preset sliding gesture of the user on the content interface (such as a "one"-shaped sliding gesture, a three-finger sliding gesture, a long press gesture in a blank space, a double-click gesture in a blank space, a gesture of sliding inward from the edge of the touch screen, etc.), a user's voice command, a user's facial expression command, etc.

[0142] S702: The electronic device creates a global layer, and the global layer is superimposed on the application layer corresponding to the content interface.

[0143] For example, after receiving an operation by a user to trigger a handwriting function, the electronic device can start a global drawing service, which will create a corresponding global layer after starting. The global drawing service is a system-level service of an electronic device, and the global drawing service can be used to provide system-level touch point generation, point prediction, writing trajectory drawing and rendering functions. The global layer is mainly used to draw the user's writing trajectory, and obtain a global layer including a real writing trajectory and a predicted writing trajectory; the content drawn on the global layer can be superimposed on the corresponding position of the content interface.

[0144] Among them, the global layer is usually a transparent layer. Since the global layer is only used to assist in drawing the user's writing trajectory, the electronic device superimposing the writing trajectory drawn on the global layer on the content interface does not affect the user's normal browsing. Of course, this application does not limit the transparency of the global layer. For example, the global layer can also be a partially transparent layer such as 80% transparent, 50% transparent, etc.

[0145] In addition, although the global layer is superimposed on the application layer corresponding to the content interface, the application response logic of the application or applet corresponding to the application layer at the lower layer to the user's operation on the content interface will not be changed. In other words, the operation of the global drawing service does not affect the reporting of the user's input events to the application layer, nor does it affect the normal response of the original application layer. Therefore, the writing traces superimposed on the global layer by the electronic device on the content interface will not affect the user's normal operation of the content interface.

[0146] For example, the global layer superimposed by the electronic device on the application layer may be as follows: Figure 8 (a) and Figure 8 As shown in (b) in . It should be noted that Figure 8 (a) and Figure 8 The misalignment of the positional relationship between the global layer and the application layer shown in (b) is to facilitate the distinction between the two. In fact, the edge of the global layer completely overlaps with the edge of the application layer and is superimposed on the application layer.

[0147] S703: The electronic device receives a handwriting operation of the user on the content interface.

[0148] The user's handwriting operation on the content interface may include but is not limited to the user's operation of drawing, writing, circling, marking, etc. on the touch screen area corresponding to the content interface using a finger or a touch-assisted tool such as a stylus, a touch pen, or a touch ball. Fig. 9 As shown in operation 901 in .

[0149] As an example, when a user performs a handwriting operation on the touch screen area corresponding to the content interface, the electronic device can detect the user's touch operation based on sensors set on the touch screen (such as touch sensors and / or pressure sensors), and obtain the user's operation information after processing the collected raw data corresponding to the touch operation based on a preset processing algorithm.

[0150] Among them, after obtaining the user's operation information, on the one hand, the electronic device can convert the operation information into specific input events according to the conventional processing logic, and then distribute the input events to the corresponding application activities; on the other hand, the electronic device can provide the operation information to the global drawing service, which is used by the global drawing service to determine the touch point and perform subsequent point prediction, writing trajectory drawing and rendering, global layer display, etc.

[0151] In some embodiments of the present application, after an application activity obtains an input event for itself, the application activity can draw, render, and synthesize corresponding graphics / text and other writing tracks according to the touch position, touch force, touch angle, touch area, and other information indicated by the input event in accordance with a conventional application response mechanism, and send the obtained writing track and content interface application layer for display. For example, the writing track drawn and rendered by the application according to the handwriting operation and finally displayed on the content interface may be as follows: Fig.10 As shown in the writing track 1.

[0152] Of course, the embodiments of the present application do not limit whether the electronic device distributes the input event corresponding to the user's handwriting operation to the corresponding application, and do not limit whether the application performs subsequent writing trajectory drawing. In some embodiments, the electronic device can also only provide the relevant information of the input event to the global drawing service, and the global drawing service is responsible for the subsequent point prediction, writing trajectory drawing rendering, global layer display and other processes.

[0153] After the global drawing service obtains the operation information or the related information of the input event, the global drawing service may continue to execute the following S704 based on the information.

[0154] S704: The electronic device determines the user's touch reporting point according to the handwriting operation, and determines multiple predicted reporting points according to the user's touch reporting point and the historical writing trajectory.

[0155] Exemplarily, the global drawing service of an electronic device can determine the user's touch reporting point based on the acquired information such as the touch position, touch strength, touch angle and touch area corresponding to the handwriting operation, and based on the reporting point prediction mechanism, predict the sliding pattern of the stylus according to the user's touch reporting point and historical writing trajectory, and then determine multiple predicted reporting points to support the electronic device to perform drawing and rendering in advance based on the predicted reporting points, thereby improving the display's hand tracking performance.

[0156] The user's touch report indicates one or more of the position of the drawing point corresponding to the user's handwriting operation, the depth of the drawing point, the width of the drawing line, etc. For example, the electronic device can determine the position of the drawing point according to the touch position corresponding to the handwriting operation, determine the depth of the drawing point according to the touch force corresponding to the handwriting operation, and determine the width of the drawing line according to the touch angle and / or touch area corresponding to the handwriting operation. The historical writing track is drawn by the global drawing service of the electronic device according to the user's historical handwriting operation within a preset historical time period; for example, the historical writing track may be as follows: Fig. 9 or Fig.10 The forecast report point indicates one or more of the position of the forecast drawing point, the depth of the forecast drawing point, the width of the forecast drawing line, etc.

[0157] As an example, when predicting the reporting points, the global drawing service of the electronic device can predict the possible touch points of the stylus within a preset time period (such as 2 seconds) in the future or the possible touch points of the stylus in the future (such as 5). Fig.10 For the specific process and algorithm of the prediction mechanism, reference may be made to the conventional prediction algorithm (such as Gaussian distribution algorithm) in conventional technology, and the present application embodiment does not limit this.

[0158] S705: The electronic device draws a corresponding writing track on the global layer according to the touch point reporting and the multiple predicted points reporting.

[0159] Exemplarily, the global drawing service of the electronic device can draw a writing track of corresponding depth and width at the corresponding position of the global layer based on information such as the positions corresponding to the touch points and multiple predicted points, the drawing line width, and the depth of the drawing points.

[0160] For example, upon receiving Fig. 9 In operation 901, the writing track drawn by the global drawing service on the global layer according to the touch point report and the multiple predicted points may be as follows: Fig.10 As shown in the writing track 2 and writing track 3, the writing track 2 is the writing track drawn by the global drawing service based on touch point reporting, and the writing track 3 is the writing track drawn by the global drawing service based on predicted point reporting.

[0161] S706: The electronic device displays a second interface, where the second interface includes a content interface and a writing track, wherein the content interface is drawn on the application layer, and the writing track is drawn on the global layer.

[0162] It can be understood that for the global drawing service to draw the writing track, the application does not draw the corresponding writing track, such as Fig.11 As shown, the global drawing service not only draws the writing trajectory based on the touch points corresponding to the handwriting operation (such as Fig.11 The writing trajectory is also drawn based on the predicted points (such as Fig.11 The writing trajectory shown in FIG3 ) is drawn based on the predicted points in advance, so the display delay of the global layer is very small (such as Fig.11 As shown in Figure 2, the hand tracking delay of the writing track drawn on the global layer after being sent to the display is very good. In addition, the global layer is a transparent layer with special properties, so the writing track actually seen by the user (such as Fig.11 The historical writing track, writing track 2, and writing track 3 are superimposed on the content interface. Therefore, from the user's visual point of view, it is like writing directly on the content interface, and the user's visual experience will be very good.

[0163] For the case where both the global drawing service and the application draw the writing track, such as Fig.10 As shown, when the application draws the writing trajectory, it only draws the writing trajectory corresponding to the actual sliding trajectory of the stylus on the application layer based on the information corresponding to the handwriting operation. Figure 2 The touch bottom layer processing, input event processing, drawing rendering and synthesis shown in the figure, and the above processes all take a long time. Therefore, when the application completes drawing the writing track for operation 901, the stylus has slid to a farther position. Fig.10 As shown, the display delay of the application layer drawn by the application (such as Fig.10 Compared with the application layer drawn by the application, such as Fig.10 As shown, the global drawing service not only draws the writing trajectory based on the touch points corresponding to the handwriting operation (such as Fig.10 The writing trajectory is also drawn based on the predicted points (such as Fig.10 The writing track drawing based on the predicted point is drawn in advance, so the writing track on the global layer is closer to the real-time position of the stylus than the writing track on the application layer, and the display delay is smaller (such as Fig.10 As shown in Figure 2, the hand tracking delay is 2). In addition, the global layer is a transparent layer with special properties. The writing track that the user actually sees is the writing track drawn on the global layer (such as Fig.10 Writing tracks 2 and 3 in the drawing) and writing tracks drawn on the application layer (such as Fig.10 Therefore, the writing track drawn on the global layer after being displayed can make up for the long hand-following delay of the writing track drawn on the application layer, and from the user's visual point of view, it is like writing directly on the content interface, and the user's visual experience will be very good.

[0164] It should be noted that the above example takes the case where the user is performing a handwriting operation and the historical writing track is always drawn on the global layer as an example, but the embodiment of the present application does not limit whether all the writing tracks are drawn on the global layer. For example, in the case where both the global drawing service and the application draw writing tracks, in other examples, the electronic device can also draw a new writing track while erasing the old writing track on the global layer. Exemplarily, the erasing method is to gradually erase in the order of time from the first to the last.

[0165] For example, Fig.12As shown, the electronic device can erase the historical writing track drawn by the application layer on the global layer and the writing track 2 corresponding to the writing track 1 according to the actual situation of the application drawing the writing track on the application layer.

[0166] Or, if Fig.13 As shown, the electronic device can erase the writing track drawn on the global layer before a preset time period (such as 2 seconds ago).

[0167] Alternatively, as time goes by, the side of the global layer close to the stylus continuously draws new writing tracks as the pen tip moves, and the historical writing tracks away from the pen tip disappear after being displayed for a period of time. After the user lifts the stylus for a period of time, the writing tracks on the global layer completely disappear.

[0168] In addition, in the case where both the global drawing service and the application draw writing tracks, in some examples, after the user completes the handwriting operation, such as when the user's finger or touch assistive tool leaves the touch screen, the electronic device can gradually erase the historical writing tracks drawn on the global layer until the application completes all the writing track drawing tasks of the handwriting operation and then erases them all. Alternatively, in other examples, after the application completes all the writing track drawing tasks of the handwriting operation, the electronic device can erase the historical writing tracks drawn on the global layer.

[0169] It should be noted that, in the case where an electronic device is drawing a new writing track while erasing the old writing track on the global layer, since the corresponding writing track has been drawn on the application layer, and the global layer is a transparent layer superimposed on the layer where the content interface is located, from the user's visual effect, the user can see the writing track corresponding to the erased writing track on the application layer through the global layer. Therefore, erasing the old writing track on the global layer does not affect the user's normal viewing of the writing track corresponding to all handwriting operations. In addition, the above-mentioned scheme of erasing the old writing track on the global layer can also reduce the complexity of subsequent global layer drawing, further improve the drawing of the writing track on the global layer, and further improve the hand tracking of the writing track display.

[0170] It can be understood that based on the method for displaying handwritten content provided in the embodiments of the present application, the electronic device can create a system-level transparent layer that is independent of the layer where the content interface is located and superimposed on the layer where the content interface is located, such as a global layer, when the user performs handwriting operations such as drawing, writing, circling, and marking on the content interface, and perform point prediction based on the user's handwriting operations and historical writing trajectories, and draw corresponding writing trajectories on the global layer based on the touch point reporting and predicted point reporting corresponding to the user's handwriting operations.

[0171] Among them, in the above process, since the writing trajectory drawn on the global layer includes the writing trajectory drawn in advance based on the predicted point, it can make up for the problem of high power consumption but limited effect of improving the chirality caused by the solution of increasing the display refresh rate and the solution of increasing the processing frequency in the conventional solution, or the problem of poor chirality of the writing trajectory display in the solution of drawing the writing trajectory only based on the actual sliding trajectory of the stylus.

[0172] In addition, in the above process, since only the writing track will be drawn on the global layer and the content of the drawing is simple, the time required for each link in the drawing process will be further shortened. In particular, compared with the drawing process based on the vertical synchronization (VSYNC) signal double buffering mechanism in the conventional solution, since the writing track on the global layer has a simple content and the drawing process takes a short time, it can be synthesized with subsequent layers in one VSYNC cycle. Among them, the VSYNC signal is a signal sent by the operating system (such as sent every 16ms) to trigger interface drawing, interface rendering, interface synthesis and display. The VSYNC signal double buffering mechanism means that a picture needs to go through 2 VSYNC cycles (used for drawing rendering and synthesis respectively) before it can be displayed to the user in the third VSYNC cycle. Therefore, the VSYNC signal double buffering mechanism will increase the display delay of the writing track. In the solution provided by the present application, there is no need to execute the VSYNC signal double buffering mechanism. Therefore, compared with the conventional solution, the display delay of the writing track can be further reduced, and the hand tracking of the writing track display can be further improved. For a detailed introduction to the double buffering mechanism of the VSYNC signal, please refer to the explanations and descriptions in conventional technologies, which will not be elaborated in this application.

[0173] In addition, in the above process, since the global layer created by the electronic device is a system-level layer, each application developer does not need to carry out additional development and upgrades for their respective applications, and the problem of poor hand tracking of the interface display can be solved through the system-level general solution. Moreover, no matter which application content interface the user enters content on, the processing process and implementation details of the system-level general solution are consistent.

[0174] For example, if each application developer independently develops and upgrades their own application based on the conventional solution, the implementation details or algorithm accuracy of the point prediction algorithm developed by each application developer may be different, so the chirality effect of the display felt by the end user may be quite different. Fig.14As shown in (a) of FIG. 1 , assuming that the writing trajectory is not drawn based on the dot prediction mechanism, the display delays of the writing trajectory corresponding to Application 1, Application 2, Application 3, and Application 4 are T1, T2, T3, and T4, respectively. If the dot prediction algorithms developed for Application 1, Application 2, and Application 3 (such as Fig.14 Prediction algorithm 1, prediction algorithm 2, and prediction algorithm 3) shown in (a) can compensate for the display delays of Δt1, Δt2, and Δt3 for application 1, application 2, and application 3, respectively. Finally, the display delays of the writing tracks corresponding to application 1, application 2, and application 3 are T1-Δt1, T2-Δt2, and T3-Δt3, respectively. If the system-level general solution provided in the embodiment of the present application is adopted, no matter which application is used to draw the writing track, the delay of the global layer drawing process is T', and the delay of the global prediction algorithm is Δt'. Therefore, the display delays of the writing tracks of application 1, application 2, application 3, and application 4 are all T'-Δt'. Therefore, based on the system-level general solution provided in the embodiment of the present application, the display tracking performance and effect felt by the user when using any application are the same.

[0175] Also, in the above process, since the global layer is a transparent layer independent of the layer where the content interface is located and superimposed on the layer where the content interface is located, from the user's visual point of view, the writing track seen is like writing directly on the content interface, and the user's visual experience will be very good.

[0176] In the case where both the global drawing service and the application draw writing tracks, in some embodiments, the brush mode used by the global drawing service when drawing writing tracks on the global layer is usually the default brush mode of the global drawing service. The brush mode used by the application when drawing writing tracks on the application layer is usually the default brush mode of the application or the brush mode setting option provided by the application that the user sets according to actual needs. However, in some cases, the default brush mode of the global drawing service is different from the default brush mode of the application. The inconsistency between the two may cause the user to visually see the writing track on the global layer when it is superimposed on the writing track on the application layer. Fig.15 The difference between the two shown in (a) results in a poor visual experience for users.

[0177] Alternatively, in the case where both the global drawing service and the application draw writing tracks, in some embodiments, the brush mode used by the global drawing service when drawing writing tracks on the global layer is usually set by the user in the brush mode setting options provided by the global drawing service according to actual needs. The brush mode used by the application when drawing writing tracks on the application layer is usually the default brush mode of the application or the user sets it according to actual needs in the brush mode setting options provided by the application. However, in some cases, the user hopes that the writing track can be displayed according to the brush mode he likes, so the user needs to set the same brush mode in the application and the global drawing service, which is cumbersome for the user to operate and has a poor user experience.

[0178] In order to solve the above problem and ensure the consistency of the writing track on the global layer and the writing track on the application layer, in an embodiment of the present application, the global drawing service can learn the brush mode of the application layer to maintain the consistency of the display effect of the writing track drawn on the global layer based on the learned brush mode and the writing track drawn on the application layer.

[0179] Among them, the brush mode represents any one or more of the following parameters: brush type (for example, pen, pencil, marker, watercolor pen, oil brush, etc.), line color (for example, black, red, green, blue, purple, etc.), line width, line texture (for example, no texture, water ripple, marble, wood grain, gravel, etc.).

[0180] As a possible implementation method, the global drawing service can obtain the brush mode of the application based on the pattern recognition method to achieve the learning of the brush mode of the application. Among them, obtaining the brush mode of the application based on the pattern recognition method mainly includes the following steps 1-2:

[0181] Step 1: Determine the newly added writing track (also called the latest writing track) on the application layer.

[0182] As an example, the global drawing service may determine a newly added writing track on the application layer by comparing a current frame image with a previous frame image on the application layer.

[0183] For example, Fig.16As shown, the global drawing service can first determine the changed layer area by comparing the current frame image on the application layer with the previous frame image, and then find the difference between the content in the layer area of ​​the current frame image and the content in the layer area of ​​the previous frame image to determine the newly added writing track on the application layer. Among them, the above-mentioned changed layer area can be a layer area of ​​a preset size including the changed content, or it can be a layer area with the smallest size including the changed content, which is not specifically limited in the embodiments of the present application. In some cases, in order to reduce the amount of calculation, the layer area with the smallest size including the changed content can be preferentially used as the changed layer area.

[0184] Step 2: Identify the brush mode of the newly added writing track.

[0185] For example, the global drawing service may perform image analysis on the identified new writing track to obtain one or more information such as brush type, line color, line width, line texture, etc. corresponding to the new writing track.

[0186] As an example, Fig.17 As shown, the global drawing service can separate the newly added writing track into three channels of R (red), G (green), and B (blue), and count the extreme points (R', G', B') other than 0 in the grayscale histograms of the three channels of R, G, and B respectively; the color corresponding to (R', G', B') is the line color.

[0187] As an example, the global drawing service can binarize the newly added writing track and perform a closing operation on the binarized writing track image to remove internal holes and irregular handwriting edges, and obtain the number of pixels A1 in the connected area. Fig.18 As shown in (a) of FIG. , the global rendering service can then perform a dilation operation on the original image to obtain a dilated image, and count the number of pixels A2 in the dilated image. For example, a 3×3 cross-shaped structure element is used to Fig.18 The original image shown in (a) is expanded to obtain Fig.18 The dilated image shown in (b) is obtained, and the number of pixels in the dilated image A2 is counted. Finally, the global drawing service can obtain the line width W according to the following formula ①.

[0188]

[0189] In one example, the global drawing service may use a binary image thinning algorithm (also known as an image skeleton extraction algorithm) to thin the connected area of ​​the newly added writing track into a width of one pixel. For example, the original image (the connected area of ​​the newly added writing track) is as follows: Fig.19As shown in (a), the image after the original image is refined using the binary image refinement algorithm is as follows Fig.19 The area of ​​the connected region of the newly added writing track divided by the area of ​​the connected region of the refined image is the line width.

[0190] As an example, the global drawing service can extract the local binary pattern (LBP) feature of the newly added writing track and count the LBP feature as the writing track texture feature, that is, the line texture. For example, the global drawing service can first obtain the gray value of each pixel of the newly added writing track, and then, Fig. 20 As shown in the figure, the global drawing service can define a 3×3 window, and use the grayscale value of the central pixel of the window as the threshold to compare the grayscale values ​​of the 8 adjacent pixels in the window. If the grayscale value of the surrounding pixels is greater than the grayscale value of the central pixel, the LBP mark of the surrounding pixels is recorded as 1; if the grayscale value of the surrounding pixels is less than or equal to the grayscale value of the central pixel, the LBP mark of the surrounding pixels is recorded as 0. Based on this, an 8-bit binary number corresponding to the window can be obtained, and the global drawing service can further convert it into a decimal number, that is, the LBP feature value corresponding to the window is obtained, so as to reflect the line texture of the track area corresponding to the 3×3 window.

[0191] We know that the difference between different brush types (such as fountain pen, pencil, marker, watercolor brush, oil brush, etc.) is mainly reflected in the texture of the main part of the writing track. Based on this, as an example, after obtaining the writing track texture features of the newly added writing track, the global drawing service can use a feature vector-based classification algorithm (such as histogram cross kernel, or principal component analysis (PCA) dimensionality reduction + support vector machine (SVM) machine learning method) to classify the writing track texture features and obtain the brush type.

[0192] Of course, the above-mentioned method and specific process of obtaining the corresponding brush type, line color, line width, line texture and other information of the newly added writing trajectory are only examples. In other embodiments, other methods in conventional technology can also be used to obtain brush mode parameters such as brush type, line color, line width, line texture, etc. according to the newly added writing trajectory. The embodiments of the present application are not limited to this.

[0193] Of course, the embodiments of the present application do not limit the specific method and process of the global drawing service obtaining the brush mode of the application. For example, as another possible implementation method, the global drawing service can obtain the brush mode of the application by obtaining the rendering command of the application to achieve the learning of the brush mode of the application. For example, the global drawing service can obtain the OpenGLES rendering pipeline command corresponding to the application, parse the brush attribute related setting field therefrom, to determine the brush mode of the application.

[0194] Alternatively, as another possible implementation, the global drawing service can learn the brush mode by obtaining the current brush mode parameters from the application. The current brush mode parameters from the application can be the parameters corresponding to the default brush mode of the application, or the parameters corresponding to the brush mode most recently set by the user, which is not specifically limited in the embodiments of the present application.

[0195] It should be noted that in the above embodiment of the present application, the content interface can be Figure 6 In the single device scenario shown in (a) of FIG. 1 , the interface of the application installed on the electronic device (such as Figure 6 The first interface shown in (a) in FIG. 1 may also be Figure 6 In the multi-device scenario shown in (b) of FIG. 1 , the second electronic device projects the screen to the content interface of the first electronic device or the extended interface of the second electronic device (such as Figure 6 The first interface shown in (b)).

[0196] Among them, in a single-device scenario, the above S701-S705 is the responsibility of the electronic device, and the application's response to the handwriting operation, such as drawing the writing track on the application layer, is also the responsibility of the electronic device. In a multi-device scenario, the above S701-S705 is the responsibility of the first electronic device, but the application's response to the handwriting operation, such as drawing the writing track on the application layer, is the responsibility of the device where the application is located, that is, the second electronic device.

[0197] For example, in a multi-device scenario, when receiving an operation by a user to trigger the handwriting function, the first electronic device may create a global layer superimposed on the application layer corresponding to the content interface. Fig.21As shown, when receiving the user's handwriting operation, the first electronic device can determine to obtain the user's operation information according to the handwriting operation. Afterwards, the first electronic device can send the operation information to the second electronic device, and the second electronic device can draw the writing track at the corresponding position of the content interface based on the operation information in the subsequent application layer drawing process, and send the code stream obtained by video encoding the drawn application layer to the first electronic device, wherein after receiving the code stream from the second electronic device, the first electronic device generates a projection layer (that is, the application layer) by decoding the code stream and temporarily stores it in the corresponding cache area; on the other hand, the first electronic device can draw the global layer according to the operation information, such as determining the user's touch report point according to the operation information, and determining multiple predicted reports according to the user's touch report point and the historical writing track, and then drawing the corresponding writing track on the global layer according to the touch report point and the multiple predicted reports, and then synthesizing the layer with the projection layer and sending it for display. Among them, in the above process, the first electronic device can also learn the brush mode of the writing track on the projection layer, and use the brush mode parameters consistent with it to draw the writing track on the global layer to ensure that the user has a consistent visual experience.

[0198] For other details of the handwritten content display process in the multi-device scenario, please refer to the relevant introduction in the above embodiment, which will not be repeated here.

[0199] It should be understood that the various schemes of the embodiments of the present application can be used in reasonable combination, and the explanations or descriptions of the various terms appearing in the embodiments can be mutually referenced or explained in the various embodiments, without limitation.

[0200] It should also be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0201] It is understandable that, in order to implement the functions of any of the above-mentioned embodiments, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0202] The embodiments of the present application can divide the electronic device into functional modules. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0203] As an example, Fig. 22 As shown, the electronic device may include a sensor unit 2210, an application drawing unit 2220, a global drawing unit 2230, a storage unit 2240 and a display unit 2250 according to the division of functional modules. Among them, the sensor unit 2210 is used to detect user operations and acquired operation information, and / or other processes related to the embodiments of the present application. The application drawing unit 2220 is used to draw application layers, and / or other processes related to the embodiments of the present application. The global drawing unit 2230 is used to draw global layers, and / or other processes related to the embodiments of the present application. The storage unit 2240 is used to store computer program instructions, data created during the use of the electronic device, such as application layers, global layers, historical writing tracks, etc. The display unit 2250 is used to display interfaces, such as displaying content interfaces and writing tracks. Of course, in some examples, the electronic device may also include other units, such as a communication unit, etc., where the communication unit can be used for the electronic device to communicate with other electronic devices, such as receiving a video stream (such as a bit stream) from a projection device in a multi-device scenario, sending user operation information on the content interface to the projection device, etc., which is not specifically limited in the embodiments of the present application.

[0204] It should also be understood that each module in the electronic device can be implemented in software and / or hardware form, without specific limitation. In other words, the electronic device is presented in the form of functional modules. The "module" here can refer to a specific application integrated circuit ASIC, a circuit, a processor and a memory that executes one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions.

[0205] In an optional manner, when data transmission is implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is implemented in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disk (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0206] The steps of the method or algorithm described in conjunction with the embodiments of the present application can be implemented in hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in an electronic device. Of course, the processor and the storage medium can also exist as discrete components.

[0207] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

Claims

1. A method for displaying handwritten content, characterized in that: The method is applied to an electronic device, and the method comprises: When displaying the content interface, in response to the user triggering the handwriting function, creating a global layer superimposed on the application layer corresponding to the content interface, wherein the global layer is a transparent layer; Determining a corresponding touch reporting point according to a handwriting operation of a user on the content interface; Draw a corresponding writing track on the global layer according to the touch reporting point; After the application layer and the global layer are synthesized, a second interface is displayed, where the second interface includes the content interface and the writing track.

2. The method according to claim 1, characterized in that: Drawing a corresponding writing track on the global layer according to the touch reporting point includes: Determine a plurality of predicted reporting points according to the touch reporting points and the historical writing tracks; Draw a corresponding writing track on the global layer according to the touch reporting point and the multiple predicted reporting points.

3. The method according to claim 2, characterized in that The method further comprises: A corresponding writing track is drawn on the application layer according to the handwriting operation.

4. The method according to claim 2, characterized in that: The content interface is generated according to the screen projection data sent by the second electronic device, and the method further includes: Sending operation information corresponding to the handwriting operation to the second electronic device, so that the second electronic device performs corresponding application layer drawing according to the operation information; Interface data is received from the second electronic device, where the interface data includes information about a writing track corresponding to the handwriting operation.

5. The method according to claim 3 or 4, characterized in that: The historical writing track is drawn on the application layer, and the corresponding writing track is drawn on the global layer according to the touch point reporting and the multiple predicted point reporting, including: A brush mode consistent with the historical writing trajectory is adopted, and a corresponding writing trajectory is drawn on the global layer according to the touch point reporting and the multiple predicted points reporting.

6. The method according to claim 5, characterized in that The method further comprises: The brush mode parameters corresponding to the historical writing track are obtained, where the brush mode parameters include one or more of the following: brush type, line color, line width, and line texture.

7. The method according to claim 6, characterized in that The step of obtaining the brush mode parameter corresponding to the latest writing track in the historical writing track includes: An image analysis is performed on the latest writing track in the historical writing tracks to obtain a brush mode parameter corresponding to the latest writing track.

8. The method according to claim 7, characterized in that The method further comprises: In the process of drawing the writing track on the global layer, a second handwriting track on the global layer corresponding to the first handwriting track drawn on the application layer is erased.

9. The method according to claim 7, characterized in that: The method further comprises: In the process of drawing the writing track on the global layer, the handwriting track drawn on the global layer before a preset time period is erased.

10. The method according to any one of claims 1 to 9, characterized in that The handwriting operation is an operation performed by a user on the content interface through a finger or a touch-assisting tool, and the touch-assisting tool includes any of the following tools: a stylus, a touch pen, and a touch ball.

11. The method according to any one of claims 1 to 10, characterized in that The handwriting operation includes any of the following operations performed by the user on the content interface: drawing, writing, circling, and marking.

12. An electronic device, characterized in that: The electronic device comprises: Display screen, used for interface display; a memory for storing computer program instructions; A processor is used to execute the computer program instructions to support the electronic terminal device to implement the method as described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processing circuit, the method according to any one of claims 1 to 11 is implemented.

14. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 11.

15. A chip system, characterized in that: The chip system includes a processing circuit and a storage medium, wherein the storage medium stores computer program instructions; when the computer program instructions are executed by the processing circuit, the method as described in any one of claims 1 to 11 is implemented.

Images