Note adding method and device, terminal and storage medium

By creating a note layer of the application interface on the user terminal, and adding notes directly on the interface and managing them, the problems of inefficient note addition and uneditable text in the existing technology are solved, and convenient note addition and management are achieved.

CN118939145APending Publication Date: 2024-11-12GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310541984.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-12
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In user terminals such as mobile phones, the prior art is used to add notes in the application interface inexperienced. The text loses editable features after screenshots, which has poor experience, and is not conducive to viewing too long.

Method used

By creating a note layer for the target application interface, users can write directly on the interface and store the note layer and interface links to facilitate the addition and management of notes.

Benefits of technology

It improves the efficiency of adding notes, provides an experience of writing directly on the application interface, and solves the problem that text cannot be edited after screenshots, simplifies the management and call of notes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939145A_ABST
    Figure CN118939145A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a note adding method and device, a terminal and a storage medium, and belongs to the field of human-computer interaction. The method comprises the steps of displaying a target application interface; in response to a note adding operation on the target application interface, creating a note layer corresponding to the target application interface, the note layer being used for bearing a note; and in response to the note saving operation, performing associated storage on the note layer and an interface link corresponding to the target application interface, the interface link being used for pointing to the target application interface. By adopting the scheme provided by the embodiment of the invention, a user can see the target application interface through the note layer and has the experience of directly writing on the target application interface; and meanwhile, the user can conveniently call the target application interface and the corresponding note layer, so that the note adding efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of human-computer interaction, and in particular to a note adding method, device, terminal and storage medium. Background Art

[0002] In the use scenarios of user terminals such as mobile phones, users often need to add notes to the content of a certain application interface. For example, they may annotate, draw, sketch, etc. while reading a browser interface.

[0003] In the related art, a screenshot of the application interface is usually taken, and the user draws notes on the screenshot to obtain a picture file containing the notes.

[0004] Adding notes to screenshots is not convenient and inefficient. For example, in order to include the entire content of the application interface, the screenshots are often too long and difficult to view; for another example, the text in the screenshots is rasterized (changed from text attributes to image attributes) and no longer has text features such as being selectable and editable (such as copying); for another example, the user experience of adding notes to screenshots is not strong. Summary of the invention

[0005] The present application provides a note adding method, device, terminal and storage medium, and the technical solution is as follows:

[0006] On the one hand, an embodiment of the present application provides a method for adding a note, the method comprising:

[0007] Display the target application interface;

[0008] In response to a note adding operation on the target application interface, creating a note layer corresponding to the target application interface, the note layer being used to carry the note;

[0009] In response to the note saving operation, the note layer and the interface link corresponding to the target application interface are stored in association, and the interface link is used to point to the target application interface.

[0010] On the other hand, an embodiment of the present application provides a method for adding a note, the method comprising:

[0011] Display the target application interface;

[0012] In response to a note adding operation on the target application interface, creating a note layer corresponding to the target application interface, the note layer being used to carry the note;

[0013] When receiving an input operation, determining an object of the input operation, wherein the object of the input operation includes at least one of the target application interface and the note layer;

[0014] The input operation is transparently transmitted to the action object, so that the action object responds to the input operation.

[0015] On the other hand, an embodiment of the present application provides a device for adding notes, the device comprising:

[0016] Display module, used to display the target application interface;

[0017] An adding module, configured to create a note layer corresponding to the target application interface in response to a note adding operation on the target application interface, wherein the note layer is used to carry notes;

[0018] The saving module is used to store the note layer and the interface link corresponding to the target application interface in association with each other in response to the note saving operation, wherein the interface link is used to point to the target application interface.

[0019] On the other hand, an embodiment of the present application provides a device for adding notes, the device comprising:

[0020] Display module, used to display the target application interface;

[0021] An adding module, configured to create a note layer corresponding to the target application interface in response to a note adding operation on the target application interface, wherein the note layer is used to carry notes;

[0022] An action object determination module, configured to determine an action object of the input operation when receiving the input operation, wherein the action object includes at least one of the target application interface and the note layer;

[0023] The transparent transmission module is used to transparently transmit the input operation to the action object so that the action object responds to the input operation.

[0024] On the other hand, an embodiment of the present application provides a terminal, which includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement the method described in the above aspect.

[0025] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one program is stored. The at least one program is loaded and executed by a processor to implement the method described in the above aspects.

[0026] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in the above aspect.

[0027] In some embodiments of the present application, by creating a note layer corresponding to the target application interface, the user can see the target application interface through the note layer and have the experience of writing directly on the target application interface; at the same time, by associating and storing the note layer and the interface link corresponding to the target application interface, the user can easily call the target application interface and its corresponding note layer, thereby improving the efficiency of note adding.

[0028] In some embodiments of the present application, by creating a note layer corresponding to the target application interface, when an input operation is received, the action object of the input operation is determined; and the input operation is transmitted to the action object so that the action object responds to the input operation. This allows the input operation to act on both the note layer and the target application interface at the same time, or only act on one of the note layer and the target application interface, so as to meet the different action requirements corresponding to different input operations of the user and improve the efficiency of note adding. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 is a flow chart of a note adding method shown in an exemplary embodiment of the present application;

[0031] Figure 2 is a flowchart of determining an object of an input operation shown in an exemplary embodiment of the present application;

[0032] Figure 3 is a schematic diagram of a multi-finger operation process shown in an exemplary embodiment of the present application;

[0033] Figure 4 is a schematic diagram of a long press operation process shown in an exemplary embodiment of the present application;

[0034] Figure 5 is a schematic diagram of an exemplary embodiment of the present application showing that the object of action is a note layer;

[0035] Figure 6 is a schematic diagram of an exemplary embodiment of the present application showing that the object of action is a target application interface;

[0036] Figure 7 is a schematic diagram showing an exemplary embodiment of the present application in which the input method is a stylus;

[0037] Figure 8 is a schematic diagram of an action object selection control shown in an exemplary embodiment of the present application;

[0038] Fig. 9 is a schematic diagram of a sliding operation process shown in an exemplary embodiment of the present application;

[0039] Fig.10 is a flow chart of an alignment process shown in an exemplary embodiment of the present application;

[0040] Fig.11 is a schematic diagram of an alignment process shown in an exemplary embodiment of the present application;

[0041] Fig.12 is a schematic diagram of a note display operation process shown in an exemplary embodiment of the present application;

[0042] Fig.13 is a schematic diagram showing an operation process of an exemplary embodiment of the present application;

[0043] Fig.14 is a structural block diagram of a note adding device provided by an exemplary embodiment of the present application;

[0044] Fig.15 is a flowchart of a note adding method shown in another exemplary embodiment of the present application;

[0045] Fig.16 is a structural block diagram of a note adding device provided by another exemplary embodiment of the present application;

[0046] Fig.17 It is a structural block diagram of a terminal device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0048] It should be understood that the "several" mentioned in this article refers to one or more, and "multiple" refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0049] Figure 1 FIG. 1 is a flowchart of a note adding method shown in an exemplary embodiment of the present application, and the method includes the following steps:

[0050] Step 101, display the target application interface.

[0051] The target application interface can be the interface corresponding to a third-party application (such as a browser interface, a reader interface, a chat application interface, a video application interface, etc.), or it can be the interface corresponding to an application that comes with the user terminal (such as a calendar application interface, a camera application interface, etc.).

[0052] The user terminal is a device with a display screen, such as a smart phone, a tablet computer, a laptop computer, etc., which is not limited in the embodiments of the present application.

[0053] Optionally, the user terminal has a capacitive touch screen and supports stylus input.

[0054] Step 102, in response to the note adding operation on the target application interface, create a note layer corresponding to the target application interface, and the note layer is used to carry the notes.

[0055] In some embodiments, the note layer can be located on the upper layer of the target application interface. In some embodiments, all or part of the area in the note layer has transparency, so that the user can see the content of the target application interface through the note layer. In some embodiments, the transparency of the note layer can be set. Exemplarily, when triggering the note adding operation, the user can choose to create a note layer with a transparency of 100% or 80%. On the note layer with full or partial transparency, there can also be some patterns, designs, borders, etc., and such deformations are within the scope of protection of this specification.

[0056] In some embodiments, the note layer may be located at the lower layer of the target application interface. In some embodiments, all or part of the area in the target application interface has transparency, so that the user can see the content of the note layer through the target application interface.

[0057] In some embodiments, a target application interface may create one or more corresponding note layers. For example, a target application interface may create three corresponding note layers located on the upper layer of the target application interface and two note layers located on the lower layer of the target application interface.

[0058] Optionally, the note adding operation is a function key operation, a shortcut key operation, a gesture operation, etc.

[0059] Optionally, the note adding operation is triggered by the stylus, for example, by a physical switch button of the stylus.

[0060] Optionally, the vertices of the note layer are aligned with the vertices of the target application interface, for example, the upper left vertex of the note layer is aligned with the upper left vertex of the target application interface, and the upper right vertex of the note layer is aligned with the upper right vertex of the target application interface.

[0061] Optionally, the width of the note layer is consistent with the physical screen width of the user terminal. For example, if the physical screen width of the user terminal is 1080px, the width of the note layer is also 1080px.

[0062] Optionally, the length of the note layer is extended along with the length of the target application interface. For example, when the user does not perform a sliding operation, the length of the target application interface is 1920px, and the length of the note layer is consistent with the length of the target application interface, also 1920px; when the user performs a sliding operation, the length of the target application interface is extended to 2680px along with the sliding, and the length of the note layer is also extended to 2680px.

[0063] Optionally, the length of the note layer is greater than, equal to, or less than the length of the target application interface.

[0064] Optionally, the width of the note layer is greater than, equal to, or less than the width of the target application interface.

[0065] The note layer is used to carry notes. Notes can include text, lines, doodles, pictures, audio and video, and other content. In some embodiments, notes can be input by handwriting (such as inputting handwritten text and doodles through a stylus) or by keyboard input (such as inputting a specific font through a keyboard based on an input method). In some embodiments, notes can also be input in other ways, such as inputting pictures through local files, or uploading audio and video content through links. The embodiment of this application does not limit the input method of notes.

[0066] In some embodiments, the note layer may include one or more controls for controlling the note layer. Figure 3As shown, the note layer 320 may include a minimize control 322 for minimizing the note layer; the note layer 320 may include a close control 323 for closing the note layer. In some embodiments, the note layer may also include a zoom control, a position adjustment control, etc.

[0067] In some embodiments, in response to different operations, different controls may be displayed or hidden in the note layer. Figure 5 As shown, when the user clicks on the doodle 521 on the note layer 520 , a doodle toolbar 522 is displayed on the note layer 520 for performing operations such as deleting, modifying, and saving the doodle 521 .

[0068] In some embodiments, the size, position, etc. of the note layer can be adjusted through the freeform mode in the multi-window mode of the Android system. For example, the target application interface and the note layer have corresponding window configurations respectively. When the user changes the size of the note layer through the size adjustment control, the system updates the window configuration corresponding to the note layer and notifies the activity of the configuration change to achieve free adjustment of the note layer size. For another example, when the user adjusts the size of the target application interface or restores the target application interface to full-screen mode, the terminal simultaneously updates the window configurations corresponding to the target application interface and the note layer to achieve synchronous update of the target application interface and the note layer.

[0069] Step 103, in response to the note saving operation, the note layer and the interface link corresponding to the target application interface are associated and stored, and the interface link is used to point to the target application interface.

[0070] Optionally, the note saving operation is a function key operation, a shortcut key operation, a gesture operation, etc.

[0071] Optionally, the interface link is a Uniform Resource Locator (URL) or a deep link.

[0072] The interface link is used to point to the target application interface. In some embodiments, the note layer can be opened based on a specific application (such as a note application, a sticky note application). When the note layer is displayed, the target application interface corresponding to the note layer can be called based on the interface link. In some embodiments, when the target application interface is displayed, the interface link is also used to call the note layer corresponding to the target application interface.

[0073] Optionally, the database associates and stores the interface link corresponding to the target application interface according to the note layer ID corresponding to the note layer.

[0074] To summarize, by creating a note layer corresponding to the target application interface, the user can see the target application interface through the note layer and have the experience of writing directly on the target application interface; at the same time, by associating and storing the note layer and the interface link corresponding to the target application interface, the user can easily call the target application interface and its corresponding note layer, thereby improving the efficiency of note adding.

[0075] Some embodiments in this specification are described by taking the creation of a note layer on the upper layer of the target application interface as an example, which does not constitute a limitation on whether the note layer is located on the upper layer or lower layer of the target application interface.

[0076] In the case where a note layer is created on the upper layer of the target application interface, when the user performs an input operation, the input operation may only target the note layer (for example, the user wants to add handwritten text on the note layer, but has no operation on the target application interface), or the input operation may only target the target application interface (for example, the user wants to select some text in the target application interface, or click a function button in the target application interface, but has no operation on the note layer), or the input operation may also target both the target application interface and the note layer (for example, the user wants to zoom, slide, etc. on both simultaneously). Therefore, when receiving the user's input operation, it is first necessary to determine the input operation's target, and then respond to the input operation on the target.

[0077] Figure 2 The flowchart of determining the object of the input operation is shown in an exemplary embodiment of the present application.

[0078] When an input operation is received, an object of the input operation is determined, where the object includes at least one of a target application interface and a note layer.

[0079] Input actions are actions triggered by the user on the note layer.

[0080] Optionally, different input operations may include different features, such as different operation types, indicated operation positions, and input methods.

[0081] Exemplarily, the operation type of the input operation may be an action type of the input operation, such as a multi-finger operation, a single-finger operation, a sliding operation, a zooming operation, and the like.

[0082] Exemplarily, the operation position indicated by the input operation may be a position on the note layer, or a position on the target application interface corresponding to the position on the note layer.

[0083] Exemplarily, the input operation may be performed by a touch pen or by a touch finger.

[0084] Optionally, the input operation targets only the note layer; or, the input operation targets only the target application interface; or, the input operation targets both the note layer and the target application interface.

[0085] The method for determining the object of the input operation may include at least one of the following steps or a combination thereof.

[0086] 1. Step 211, based on the operation type of the input operation, determine the object of the input operation.

[0087] In some embodiments, the correspondence between the operation type of the input operation and the action object may be determined according to a preset rule.

[0088] In a possible implementation, when the operation type of the input operation is a multi-finger operation, the objects of action are determined to be the target application interface and the note layer.

[0089] Among them, multi-finger operation refers to an operation performed simultaneously with two or more fingers, such as a two-finger reverse swipe operation, a three-finger upward swipe operation, etc.

[0090] In a possible usage scenario, when a user performs a multi-finger operation, the user intends to operate both the note layer and the target application interface at the same time, such as zooming in and out, swiping up, etc. on both the note layer and the target application interface at the same time. Therefore, the objects corresponding to the multi-finger operation can be the target application interface and the note layer.

[0091] For example, Figure 3 As shown, a transparent note layer 320 is created on the upper layer of the browser interface 310, and the user's hand-drawn graffiti 321 is located on the note layer 320. When the user performs a multi-finger operation (i.e., a zoom operation) of sliding two fingers in opposite directions, the zoom operation is applied to both the browser interface 310 and the note layer 320. That is, after the zoom operation, the content on the browser interface 310 is enlarged by a certain proportion, the note layer 320 is also enlarged by a certain proportion, and the graffiti 321 located on the note layer 320 is also enlarged by a certain proportion.

[0092] It is understandable that due to the need for drawing the accompanying drawings, the vertices of the browser interface 310 and the note layer 320 are not aligned in the drawing to highlight the difference between the two. In actual applications, the vertices of the target application interface and the note layer are usually aligned. Other embodiments of the present application and the corresponding drawings also follow this principle and do not constitute a limitation on the position of the target application interface and the note layer.

[0093] In a possible implementation, when the operation type of the input operation is a long press operation, the action object is determined to be a target application interface.

[0094] Among them, a long press operation refers to an input operation in which the contact time with the screen exceeds a threshold (such as 1 second).

[0095] In a possible usage scenario, when the user performs a long press operation, the user intends to operate the target application interface, such as selecting and copying text, pictures, etc. in the target application interface. Therefore, the object of the long press operation may be the target application interface.

[0096] For example, Figure 4 As shown, in the browser interface 410, there is a text 411 with the words "XX Review Blogger". On the upper layer of the browser interface 410, a transparent note layer 420 is created. When the user performs a long press operation on the note layer 420, the object of the long press operation is the browser interface 410. That is, the effect of the long press operation is to select the text 411 located in the browser interface 410, forming a selection box 412. As shown in the figure, the user's long press operation on the note layer can be transmitted to the target application interface to achieve the target application interface's response to the operation.

[0097] 2. Step 212: determining an object of the input operation based on the operation position indicated by the input operation.

[0098] In a possible usage scenario, when a user performs an input operation on a note layer, it can be determined whether there is an operation object at the operation position indicated by the input operation in the note layer or the target application interface, and whether the operation object supports the input operation.

[0099] Optionally, the operation object may be text, picture, audio or video, function button, control, note, doodle or other object.

[0100] Step 212a: If an operation object exists at the operation position indicated by the input operation in the note layer and the operation object supports the input operation, it is determined that the operation object is the note layer.

[0101] For example, Figure 5As shown, a transparent note layer 520 is created on the upper layer of the browser interface 510. On the note layer 520, there is a doodle 521 hand-drawn by the user, and the doodle 521 circles the date "2022-10-05" on the browser interface 510. When the user performs a click operation on the note layer 520, there is an operation object (doodle 521) at the position corresponding to the click operation in the note layer 520, and the doodle 521 supports the click operation. Therefore, it is determined that the object of the click operation is the note layer 520. The click operation acts on the doodle 521 on the note layer 520, and the doodle toolbar 522 is displayed. The doodle toolbar 522 is used to delete, modify, save, and other operations on the doodle 521.

[0102] Step 212b: If an operation object exists at the operation position indicated by the input operation in the target application interface and the operation object supports the input operation, the operation object is determined to be the target application layer.

[0103] For example, Figure 6 As shown, in the browser interface 610, there is a thumbs up icon 611 for giving a thumbs up. A transparent note layer 620 is created on the upper layer of the browser interface 610. When the user performs a click operation on the note layer 620, in the browser interface 610, there is an operation object (thumbs up icon 611) at the position corresponding to the click operation, and the thumbs up icon 611 supports the click operation; whereas in the note layer 620, there is no operation object at the position corresponding to the click operation. Therefore, it is determined that the object of the click operation is the browser interface 610, not the note layer 620. The click operation is transmitted to the browser interface 610, and acts on the thumbs up icon 611, forming a thumbs up icon 612.

[0104] 3. Step 213: Determine the object of the input operation based on the input mode corresponding to the input operation.

[0105] In a possible usage scenario, the user may use a stylus pen or a finger to touch the screen to implement an input operation. Optionally, the input operation corresponds to an input method of a stylus pen or a touching finger.

[0106] Exemplarily, when the terminal screen is a capacitive touch screen, the terminal can identify whether the input method is a stylus or a touching finger according to the position, size and touch sensitivity of the touch point.

[0107] In a possible implementation, when the input operation is performed using a stylus pen, the action object is determined to be a note layer.

[0108] See also Figure 7 , Figure 7It is a schematic diagram showing an exemplary embodiment of the present application in which the input method is a stylus.

[0109] like Figure 7 As shown, a transparent note layer 720 is created on the upper layer of the browser interface 710. The user performs a drawing operation on the note layer 720 based on the stylus 730. When the user terminal recognizes that the input method of the drawing operation is the stylus, it determines that the object of the drawing operation is the note layer 720. Therefore, the drawing content 721 of the drawing operation is displayed on the note layer 720.

[0110] In a possible implementation, when the input operation is performed by touching a finger, the object of action is determined to be a target application interface.

[0111] For example, Figure 4 As shown, the user clicks the text 411 on the browser interface 410 with a finger through the note layer 420. When the user terminal recognizes that the input method of the click operation is a touch finger, it determines that the object of the click operation is the browser interface 410, and forms a selection box 412 for the text 411 on the browser interface 410.

[0112] In a possible scenario, to determine the object of the input operation, using only one of the above methods may not be accurate enough. Therefore, the above methods may be combined, or other methods may be used to determine the object of the input operation.

[0113] For example, Figure 5 As shown, when the user performs a click operation on the note layer 520, there is an operation object (graffiti 521) at the position corresponding to the click operation in the note layer 520, and the graffiti 521 supports the click operation; however, in the browser interface 510, there is a date "2022-10-05" near the position corresponding to the click operation, and the date also supports the click operation. Therefore, the user terminal may mistakenly interpret the user's intention of clicking the date on the browser interface 510 as the intention of clicking the graffiti 521 on the note layer 520, resulting in incorrect determination of the object of action.

[0114] To resolve the situation where the object of action is incorrectly determined, in some embodiments, the operation type of the input operation, the operation position indicated by the input operation, and the input method of the input operation can be combined to jointly determine that the object of the input operation is the note layer and / or the target application interface.

[0115] Exemplarily, if the input method of the click operation is a stylus pen, and there is graffiti on the note layer and the word "date" on the browser interface at the position corresponding to the click operation, and both the graffiti and the word "date" support the click operation, then the action object is determined to be the note layer; if the input method of the click operation is a touching finger, and there is graffiti on the note layer and the word "date" on the browser interface at the position corresponding to the click operation, and both the graffiti and the word "date" support the click operation, then the action object is determined to be the browser interface; if the input method of the click operation is a stylus pen, and there is a date on the browser interface at the position corresponding to the click operation, the word "date" supports the click operation, and there is no operation object on the note layer, then the action object is determined to be the browser interface.

[0116] As such, methods of jointly determining the object of the input operation by combining at least two of the operation type of the input operation, the operation position indicated by the input operation, and the input method of the input operation are all within the protection scope of this specification.

[0117] 4. Step 214, based on the trigger state of the object selection control in the note layer, determine the object of the input operation.

[0118] To solve the problem of incorrectly determining the action object, in some embodiments, at least one action object selection control is provided in the note layer for determining the action object.

[0119] In a possible implementation, an action object selection control is provided in the note layer, and different action objects are determined based on different trigger operations on the action object selection control. For example, when the user clicks the action object selection control, the action object is determined to be the note layer; when the user double-clicks the action object selection control, the action object is determined to be the target application interface; when the user long presses the action object selection control, the action object is determined to be the note layer and the target application interface.

[0120] In another possible implementation, two action object selection controls are provided in the note layer, namely, the note layer selection control and the target application interface selection control. In response to the triggering operation of the note layer selection control, the action object is determined to be the note layer; in response to the triggering operation of the target application interface selection control, the action object is determined to be the target application interface; in response to the triggering operation of the note layer selection control and the target application interface selection control, the action object is determined to be the note layer and the target application interface. Figure 8 , Figure 8 It is a schematic diagram of an action object selection control shown in an exemplary embodiment of the present application.

[0121] like Figure 8As shown, a note layer 820 is created on the upper layer of the browser interface 810 , and two kinds of action object selection controls may be provided on the note layer 820 , namely a note layer selection control 821 and a browser interface selection control 822 .

[0122] In one possible implementation, when the user turns on or off the above-mentioned two types of action object selection controls at the same time, the action objects of subsequent input operations are determined to be the note layer 820 and the browser interface 810; when the user only turns on the note layer selection control 821, the action object of subsequent input operations is determined to be the note layer 820; when the user only turns on the browser interface selection control 822, the action object of subsequent input operations is determined to be the browser interface 810.

[0123] Step 220: transparently transmit the input operation to the action object so that the action object responds to the input operation.

[0124] Transparent transmission refers to the technical means of passing input operations to the note layer and / or the target application interface. By transparently transmitting the input operation to the object, the object can respond to the input operation, so that the user can control the object.

[0125] In a possible manner, the transparent transmission can be implemented based on the touch event processing mechanism of the Android system. Specifically, the Android system can distribute the input operation through InputDispatcher to distribute the input operation to the action object.

[0126] In some embodiments, transparent transmission may be achieved through other methods, which are not limited in this specification.

[0127] For example, Figure 6 As shown, after determining that the object of the click operation is the browser interface 610, the user terminal transmits the click operation to the browser interface 610, so that the browser interface 610 responds to the click operation and clicks on the like icon 611 on the browser interface 610 to form a like icon 612.

[0128] For example, Figure 3 As shown, after determining that the objects of the zoom operation are the browser interface 310 and the note layer 320, the user terminal transmits the zoom operation to the browser interface 310 and the note layer 320, so that the browser interface 310 and the note layer 320 respond to the zoom operation at the same time, thereby achieving synchronous zooming of the browser interface and the note layer.

[0129] In this embodiment, the object of the input operation is determined by at least one of the operation type of the input operation, the operation position indicated by the input operation, the input method of the input operation, and the trigger state of the object selection control, and the input operation is transmitted to the object. This can meet the user's needs for operating on the target application interface and / or the note layer. The user can operate on the target application interface and the note layer at the same time (such as sliding, zooming), or operate on the target application interface alone (such as selecting), or operate on the note layer alone (such as graffiti), thereby adding to the richness of the note layer function and improving the efficiency of note adding.

[0130] Regarding the method of creating a note layer on the upper layer of the target application interface, in a possible implementation, the note layer can be mounted on a child node of the target application interface. In a possible implementation, the ability of mounting parent-child nodes between layers of the Android system can be used to mount (reparent) the view corresponding to the note layer to the child node of the target application interface, so as to realize the synchronous display of the note layer on the upper layer of the target application interface.

[0131] In some embodiments, the running state of the note layer and the target application interface are consistent. For example, when the target application interface switches from foreground operation to background operation, the control note layer also switches from foreground operation to background operation; when the target application interface switches from background operation to foreground operation, the control note layer also switches from background operation to foreground operation.

[0132] In one possible scenario, a user can perform zooming and / or sliding operations on the note layer, and the zooming and / or sliding operations can be transmitted to the note layer and the target application interface at the same time. The note layer and the target application interface perform synchronous zooming / synchronous sliding based on the received zooming and / or sliding operations.

[0133] However, in the actual synchronous sliding / synchronous zooming process, the zoom ratio / sliding step size of the note layer and the target application interface may be different. For example, the default settings of the target application interfaces corresponding to different applications may be different, and their responses to the same zoom / sliding operation may also be different. For example, for a reader interface containing more text, a sliding operation defaults to sliding the interface down 270px, while for a social application interface containing more large pictures, the same sliding operation defaults to sliding the interface down 540px. Based on the above differences, the note layer usually slides according to a sliding step size that is common to most applications. However, the common sliding step size may not be suitable for the current target application interface, so the position of the note on the target application interface will drift.

[0134] Fig. 9It is a schematic diagram of a sliding operation process shown in an exemplary embodiment of the present application.

[0135] like Fig. 9 As shown, a transparent note layer 920 is created on the upper layer of the browser interface 910, and a graffiti 921 drawn by the user is on the note layer 920, and the position of the graffiti 921 is circled in the words "the following contents" located on the browser interface 910. When the user performs a sliding operation, the objects of the sliding operation include both the browser interface 910 and the note layer 920. Among them, the browser interface 910 slides up 300px according to the default settings of the browser, and the words "the following contents" located thereon also slide up 300px; the note layer 920 slides up 275px according to the general settings, and the graffiti 921 located thereon also slides up 275px. Therefore, after the sliding operation, the graffiti 921 no longer circles the words "the following contents", that is, the position of the graffiti 921 on the browser interface 910 drifts.

[0136] In order to solve the problem of note position drift on the target application interface after zooming and / or sliding operations, in one possible implementation method, when the objects of action are the target application interface and the note layer, and the input operation is a zooming and / or sliding operation, the target application interface and the note layer after the operation response can be aligned.

[0137] Fig.10 Detailed description of the invention is given in detail below.

[0138] Step 1001, display the target application interface.

[0139] Step 1002, creating a note layer on the upper layer of the target application interface.

[0140] Step 1003, determining a first reference point in the note layer and a second reference point in the target application interface, wherein the initial coordinates of the first reference point and the second reference point are consistent.

[0141] The first reference point is at least one marking point in the note layer, and the second reference point is at least one marking point in the target application interface. In some embodiments, there are multiple first reference points and second reference points, and each of the first reference points corresponds to each of the second reference points.

[0142] The initial coordinates of the first / second reference point refer to the coordinates of the first / second reference point before the zoom operation and / or the slide operation.

[0143] Optionally, the initial coordinates of the first / second reference point are the coordinates of the first / second reference point relative to the upper left corner vertex of the terminal screen before the zoom operation and / or the sliding operation.

[0144] As Fig.11 shown, before the sliding operation, a note layer 1120a is created on the browser interface 1110a.

[0145] In the note layer 1120a, three first reference points are determined, namely the first reference points 1121a, 1122a, and 1123a. Among them, the initial coordinates of the three first reference points are the coordinates relative to the top left vertex of the terminal screen before the sliding operation, which are (200, -300), (50, -350), and (280, -350) respectively.

[0146] In the target application interface 1110a, three second reference points are determined, namely the second reference points 1111a, 1112a, and 1113a. Among them, the initial coordinates of the three second reference points are the coordinates relative to the top left vertex of the terminal screen before the sliding operation, which are the same as the initial coordinates of the first reference points, and are also (200, -300), (50, -350), and (280, -350) respectively. The three first reference points and the three second reference points correspond one by one.

[0147] Step 1004, determine the reference point features corresponding to the second reference point, where the reference point features are the features of the second reference point and the pixel points on the periphery of the second reference point.

[0148] Optionally, the reference point features of the second reference point include the RGB values of the pixel points at the initial coordinates of the second reference point. Optionally, the reference point features of the second reference point include the RGB values of the pixel points on the periphery of the initial coordinates of the second reference point. For example, the RGB values of all pixel points within a circular area with a radius of 5px near the initial coordinates of the second reference point.

[0149] In one possible implementation, the reference point features of the second reference point can be obtained through image recognition technology. In one possible implementation, the source code of the target application corresponding to the target application interface can also be called to obtain the reference point features of the second reference point. The embodiments of the present application do not limit the manner of obtaining the reference point features of the second reference point.

[0150] Exemplarily, Fig.11 the reference point features of the three second reference points in can be the features of the second reference point and the pixel points on the periphery of the second reference point. For example, the features of the second reference point 1111a with coordinates (200, -300) and the pixel points on its periphery are the RGB values of the image near ":"; the features of the second reference point 1112a with coordinates (50, -350) and the pixel points on its periphery are the RGB values of the image near the character "今"; the features of the second reference point 1113a with coordinates (280, -350) and the pixel points on its periphery are the RGB values of the image near the character "记".

[0151] Step 1005, when the input operation is a zoom operation and / or a sliding operation, the input operation is transparently transmitted to the target application interface and the note layer.

[0152] When the input operation is a zoom operation and / or a sliding operation, the input operation is transparently transmitted to the target application interface and the note layer, so as to zoom / slide the target application interface and the note layer at the same time.

[0153] Step 1006, obtain the first real-time coordinates of the first reference point after the operation response, and based on the reference point feature corresponding to the second reference point, identify the second reference point from the target application interface after the operation response, and obtain the second real-time coordinates of the second reference point.

[0154] The first / second real-time coordinates refer to the coordinates of the first / second reference point after the zoom operation and / or the slide operation.

[0155] Optionally, the first / second real-time is the coordinates of the first / second reference point relative to the top left corner of the terminal screen after the zoom operation and / or the sliding operation.

[0156] In some embodiments, the second reference point can be identified from the target application interface after the operation response based on the reference point features corresponding to the second reference point through image recognition technology. In some embodiments, the second real-time coordinates of the second reference point can also be determined based on the reference point features of the second reference point by calling the target application source code corresponding to the target application interface. The method for determining the second real-time coordinates of the second reference point is not limited in the embodiments of the present application.

[0157] The zoom ratio / slide step size of the target application interface and the note layer may be inconsistent. Fig.11 As shown, the browser interface 1110a slides up 250px according to the default settings of the browser to form the browser interface 1110b; the note layer 1120a slides up 200px according to the general settings to form the note layer 1120b.

[0158] In the note layer 1120b after the sliding up operation, the positions of the three first reference points slide up 200px as the note layer 1120b slides up and becomes the first reference points 1121b, 1122b and 1123b; the first real-time coordinates of the first reference points 1121b, 1122b and 1123b are: (200, -100), (50, -150) and (280, -150), respectively.

[0159] In the browser interface 1110b after the upward sliding operation, the positions of the three second reference points slide upward by 250 px along with the browser interface 1110b, becoming the second reference points 1111b, 1112b, and 1113b; the second real-time coordinates of the second reference points 1111b, 1112b, and 1113b are respectively: (200, -50), (50, -100), and (280, -100). Optionally, the second real-time coordinates of the second reference points 1111b, 1112b, and 1113b are respectively identified based on the RGB values of the images near the character ":" which is the feature of the second reference point 1111a and its surrounding pixel points, the RGB values of the images near the character "今" which is the feature of the second reference point 1112a and its surrounding pixel points, and the RGB values of the images near the character "记" which is the feature of the second reference point 1113a and its surrounding pixel points.

[0160] Step 1007, in the case where the first real-time coordinates are different from the second real-time coordinates, perform an alignment process on the note layer based on the coordinate differences between the first real-time coordinates and the second real-time coordinates.

[0161] In some embodiments, based on the coordinate differences between the first real-time coordinates and the second real-time coordinates, adjust the position of the note layer so that the first real-time coordinates of the first reference point in the note layer are consistent with the second real-time coordinates of the second reference point in the target application interface.

[0162] As Fig.11 shown, the upward sliding step of the note layer 1120b is inconsistent with that of the browser interface 1110b, resulting in a 50-px difference in the vertical coordinates between the first real-time coordinates of the first reference points 1121b, 1122b, and 1123b and the second real-time coordinates of the second reference points 1111b, 1112b, and 1113b. Therefore, based on this difference, the note layer 1120b can be slid upward by 50 px again so that the note layer 1120b is aligned with the browser interface 1110b, thereby achieving the synchronous upward sliding of the note layer and the browser interface.

[0163] In this embodiment, by performing an alignment process on the note layer through the differences between the first real-time coordinates of the first reference point and the second real-time coordinates of the second reference point, the situation of the note drifting on the target application interface after the zoom operation and / or the sliding operation can be solved, so that the note layer and the target application interface achieve synchronous sliding / synchronous zooming.

[0164] In order to solve the problem of note position drift on the target application interface after zooming and / or sliding operations, in another possible implementation method, the default settings of the target application (such as the default sliding step for a sliding operation, the default zoom ratio for a zoom operation, etc.) can be obtained in advance, the default settings of the target application can be stored, and the response of the note layer to the input operation can be controlled based on the default settings of the target application, so that the zoom ratio / sliding step of the note layer is adapted to the corresponding target application interface.

[0165] In some embodiments, when an operation mapping relationship corresponding to the target application interface is stored, the note layer is controlled to respond to the input operation based on the operation mapping relationship.

[0166] The operation mapping relationship includes at least one of a zoom ratio mapping relationship and a sliding distance mapping relationship. For example, the operation mapping relationship may include a default sliding step length of a sliding operation, a default zoom ratio of a zoom operation, and the like.

[0167] In some embodiments, when the operation mapping relationship corresponding to the target application interface is not stored, the note layer is controlled to respond to the input operation based on the preset operation mapping relationship. For example, the note layer is controlled to respond to the sliding and / or zooming operation based on the universal sliding step size and / or universal zoom ratio of common applications.

[0168] In some embodiments, when the operation mapping relationship corresponding to the target application interface is not stored, the preset operation mapping relationship can also be updated based on the difference between the first real-time coordinates of the first reference point and the second real-time coordinates of the second reference point obtained in the alignment process, so as to obtain the operation mapping relationship corresponding to the target application interface, and the obtained result can be saved for application in subsequent zoom operations / sliding operations.

[0169] like Fig.11 As shown, in a sliding operation, the note layer slides up 200px according to the preset operation mapping relationship. In the alignment process, the difference between the first real-time coordinate of the first reference point and the second real-time coordinate of the second reference point is 50px. Therefore, 250px can be stored as the operation mapping relationship corresponding to the browser interface. The next time the user slides the browser interface, it can slide directly according to the sliding step of 250px without the need for alignment processing, thereby improving the efficiency of the note layer sliding synchronously with the target application interface.

[0170] In this embodiment, when the operation mapping relationship corresponding to the target application interface is stored, the note layer is controlled to respond to the input operation through the operation mapping relationship. The zoom ratio / sliding step size adapted to the target application interface can be used to directly realize synchronous sliding / synchronous zooming without the need for alignment processing, thereby improving the utilization efficiency of the note layer; when the operation mapping relationship corresponding to the target application interface is not stored, the preset operation mapping relationship can be adjusted based on the result of the alignment processing for application in subsequent zoom operations / sliding operations, thereby improving the utilization efficiency of subsequent note layers.

[0171] In one possible scenario, when the user reopens the target application interface, the user can open the notes added in the past, and then continue to view, edit, modify, delete, and save the notes added in the past.

[0172] In some embodiments, when the user terminal displays the target application interface, in response to the note display operation, a note layer corresponding to the target application interface is determined.

[0173] Optionally, the note display operation is a click operation, a function key operation, a shortcut key operation, a gesture operation, etc.

[0174] Optionally, the note display operation is triggered by a stylus, for example, by a note display button of the stylus.

[0175] In some embodiments, when the user reopens the target application interface, the terminal can display the note layer corresponding to the target application interface based on the interface link corresponding to the target application interface.

[0176] In some embodiments, a note bar containing multiple note layer thumbnails may be displayed on the target application interface, and the user may click and select any note layer thumbnail in the note bar to display the note layer corresponding to the note layer thumbnail.

[0177] like Fig.12 As shown, the user terminal displays a browser application interface 1210, and a note bar 1220 containing multiple note layer thumbnails is displayed on the browser application interface 1210, for example, note layer thumbnails 1221, 1222, and 1223. When the user clicks on the note layer thumbnail 1222, the note display operation is triggered, and the user terminal determines the note layer 1230 corresponding to the note layer thumbnail 1222 based on the interface link corresponding to the target application interface, and displays the note layer 1230. Among them, the notes on the note layer 1230 are consistent with the notes displayed by the note layer thumbnail 1222.

[0178] In some embodiments, the interface link corresponding to the target application interface is used to point to the historical note content. Based on the interface link corresponding to the target application interface, the user terminal displays a note layer on the upper layer of the target application interface, and the note layer includes the historical note content on the target application interface.

[0179] In this embodiment, by calling the historical note content through the interface link corresponding to the target application interface, the historical note content can be displayed on the upper layer of the target application interface, so that the user can continue to view, edit, modify, delete and save the notes added in the past.

[0180] In one possible scenario, when the user views the note layer alone (for example, when operating alone through a note application or a sticky note application), the user can open the corresponding target application interface or a screenshot of the target application interface based on the interface link corresponding to the note layer, and then compare the content of the note layer with that of the corresponding target application interface.

[0181] In some embodiments, the user terminal obtains an interface link corresponding to the note layer in response to a display operation on the note layer.

[0182] Optionally, the display operation of the note layer is function key operation, shortcut key operation, gesture operation, etc.

[0183] Optionally, the display operation of the note layer is triggered by the stylus, for example, by an application display button of the stylus.

[0184] In some embodiments, the interface link corresponding to the note layer is used to point to the target application interface corresponding to the note layer. In some embodiments, the interface link corresponding to the note layer is used to point to a screenshot of the target application interface corresponding to the note layer. The user terminal displays the target application interface or a screenshot corresponding to the target application interface based on the interface link, and displays the note layer on the upper layer of the target application interface, and the note layer contains the historical note content on the target application interface.

[0185] like Fig.13 As shown, the user terminal displays a note layer 1320. When the user performs a display operation on the note layer, the user terminal displays the corresponding target application interface 1310 based on the interface link, and displays the note layer 1320 on the upper layer of the target application interface 1310, and the note layer 1320 contains the historical note content on the target application interface.

[0186] In this embodiment, by calling the target application interface through the interface link corresponding to the note layer, the target application interface can be displayed simultaneously when viewing historical notes, and the content in the note layer and its corresponding target application interface can be compared and viewed, thereby improving the efficiency of viewing notes.

[0187] In one possible scenario, the interface link corresponding to the note layer may be invalid, resulting in the inability to call the target application interface and display the note layer on top of the target application interface. For example, when there is a server failure, gateway blocking, or the URL of the target application is updated, the interface link corresponding to the note layer may be invalid.

[0188] To solve the problem that the interface link corresponding to the note layer may become invalid, in some embodiments, in response to a note save operation, the note layer and the snapshot link corresponding to the target application interface may be stored in association.

[0189] The snapshot link is used to point to the web snapshot of the target application interface. When the interface link corresponding to the note layer is invalid, the snapshot link can be called.

[0190] In some embodiments, in response to a display operation on the note layer, a snapshot link corresponding to the note layer is obtained; a web page snapshot of the target application interface is displayed based on the snapshot link; and the note layer is displayed on an upper layer of the web page snapshot, wherein the note layer includes historical note content on the target application interface.

[0191] In this embodiment, by associating and storing the note layer and the snapshot link corresponding to the target application interface, when the interface link of the target application interface becomes invalid, a web page snapshot of the target application interface can still be obtained, and the historical note content can be displayed on the upper layer of the web page snapshot, thereby compensating for the inconvenience of note display caused by the invalid web page.

[0192] In one possible scenario, the note content is displayed dynamically over time. For example, the note content is the derivation process of a formula, and the note content at different time points is different formula steps; for another example, the note content is an annotation for a video played on the target application interface, and different annotation contents correspond to different time points.

[0193] To achieve dynamic display of note content, in some embodiments, in response to a note saving operation, the user terminal uses the note layer and the display time corresponding to the note in the note layer as the note dynamic display information.

[0194] The dynamic display information of the notes represents the display duration, display order, and other information of different notes in the note layer. In some embodiments, the dynamic display information of the notes can be determined based on the input operation. For example, the input time of formula 1 in the note is recorded as t_1, and the input time of formula 2 is recorded as t_2, then the dynamic display information of the notes can be recorded as (formula 1: t_1; formula 2: t_2). Optionally, in addition to the input time of the formula, the dynamic display information of the notes can also include information such as the writing process of the formula and the order of strokes.

[0195] In some embodiments, the database stores the dynamic display information of the notes and the interface link corresponding to the target application interface in association; in response to the note display operation, based on the interface link, the note layer is displayed on the upper layer of the target application interface. The historical note content in the note layer is dynamically displayed based on the dynamic display information of the notes.

[0196] Exemplarily, a note layer is displayed on the upper layer of the target application interface. When the display time of the target application interface reaches t_1, formula 1 is displayed on the note layer; when the display time of the target application interface reaches t_2, formula 2 is displayed on the note layer.

[0197] When the target application interface is a video playback interface, the dynamically displayed note content may also have a temporal correlation with the video playback content.

[0198] Optionally, the video playback interface is a video website interface, or a camera application interface, etc.

[0199] In some embodiments, the user terminal can obtain video dynamic display information of the video playback interface.

[0200] The video dynamic display information includes at least one frame in the video and the display time corresponding to the at least one frame. Exemplarily, the display time of the first key frame of the video is T_1, and the display time of the second key frame is T_2, then the video dynamic display information can be recorded as (first key frame: T_1; second key frame: T_2). Optionally, in addition to the display time of the key frame, the video dynamic display information can also include information such as the display time of other frames and the display content of the video frame.

[0201] In some embodiments, in response to a note saving operation, the user terminal obtains time association information corresponding to the video dynamic display information and the note dynamic display information.

[0202] The time association information represents the time association between the dynamic display of the video and the dynamic display of the notes. In some embodiments, the time association information can be determined based on the input operation. For example, when a user derives a formula on the note layer while playing a video, the input time of the formula can be associated with the display time of the video frame displayed at the same time. Exemplarily, if the time when the user inputs Formula 1 is t_1, and the video frame displayed at the same time is the second key frame, the time association information can be recorded as (t_1: T_1).

[0203] In some embodiments, the database associates and stores video dynamic display information, note dynamic display information, time association information, and an interface link corresponding to the target application interface.

[0204] In some embodiments, in response to the note display operation, the user terminal displays a note layer on the upper layer of the target application interface based on the interface link, wherein the historical note content in the note layer is dynamically displayed along with at least one frame in the video based on the time association information.

[0205] Exemplarily, a note layer is displayed on the upper layer of the target application interface, and when the playing time of the video played in the target application interface reaches T_1, Formula 1 is displayed on the note layer.

[0206] In this embodiment, by associating and storing the note dynamic display information and the interface link corresponding to the target application interface, it is possible to dynamically display the note content on the note layer; by associating and storing the video dynamic display information, the note dynamic display information, the time association information and the interface link corresponding to the target application interface, it is possible to synchronously display the corresponding dynamic notes while the video is playing, enriching the scenarios for adding notes and improving the efficiency of adding notes to video content.

[0207] Fig.14 : is a structural block diagram of a note adding device provided by an exemplary embodiment of the present application, the device comprising:

[0208] The display module 1401 is used to display the target application interface.

[0209] The adding module 1402 is used to create a note layer corresponding to the target application interface in response to a note adding operation on the target application interface, and the note layer is used to carry the notes.

[0210] The saving module 1403 is used to store the note layer and the interface link corresponding to the target application interface in association with each other in response to the note saving operation, wherein the interface link is used to point to the target application interface.

[0211] Optionally, a module 1402 is added for:

[0212] When receiving an input operation, determining an object of the input operation, wherein the object of the input operation includes at least one of the target application interface and the note layer;

[0213] The input operation is transparently transmitted to the action object, so that the action object responds to the input operation.

[0214] Optionally, a module 1402 is added for:

[0215] Determining the object of the input operation based on the operation type of the input operation; or,

[0216] Determining the object of the input operation based on the operation position indicated by the input operation; or,

[0217] Determine the object of the input operation based on the input mode corresponding to the input operation; or,

[0218] The action object of the input operation is determined based on a trigger state of an action object selection control in the note layer.

[0219] Optionally, a module 1402 is added for:

[0220] In a case where the operation type of the input operation is a multi-finger operation, determining that the action objects are the target application interface and the note layer;

[0221] When the operation type of the input operation is a long press operation, the action object is determined to be the target application interface.

[0222] Optionally, a module 1402 is added for:

[0223] In the case where the input operation is performed using a stylus pen, determining that the action object is the note layer;

[0224] In the case where the input operation is performed in the form of a touch finger, the action object is determined to be the target application interface.

[0225] Optionally, a module 1402 is added for:

[0226] There is an operation object at the operation position indicated by the input operation in the note layer, and the operation object supports the input operation, and the operation object is determined to be the note layer;

[0227] An operation object exists at the operation position indicated by the input operation in the target application interface, and the operation object supports the input operation, and the operation object is determined to be the target application layer.

[0228] Optionally, when the action object is the target application interface and the note layer, and the input operation is a zoom operation and / or a sliding operation, a module 1402 is added for:

[0229] The target application interface and the note layer after the operation response are aligned.

[0230] Optionally, a module 1402 is added for:

[0231] Determine a first reference point in the note layer, and determine a second reference point in the target application interface, wherein initial coordinates of the first reference point and the second reference point are consistent;

[0232] The aligning of the target application interface and the note layer after the operation response includes:

[0233] Acquire a first real-time coordinate of the first reference point after the operation response, and a second real-time coordinate of the second reference point;

[0234] When the first real-time coordinates are different from the second real-time coordinates, the note layer is aligned based on the coordinate difference between the first real-time coordinates and the second real-time coordinates.

[0235] Optionally, a module 1402 is added for:

[0236] After determining the second reference point in the target application interface, determining a reference point feature corresponding to the second reference point, the reference point feature being a feature of the second reference point and pixels around the second reference point;

[0237] Based on the reference point feature, the second reference point is identified from the target application interface after the operation response, and the second real-time coordinates of the second reference point are acquired.

[0238] Optionally, a module 1402 is added for:

[0239] In the case where the operation mapping relationship corresponding to the target application interface is not stored, controlling the note layer to respond to the input operation based on a preset operation mapping relationship, wherein the operation mapping relationship includes at least one of a zoom ratio mapping relationship and a sliding distance mapping relationship;

[0240] In a case where an operation mapping relationship corresponding to the target application interface is stored, the note layer is controlled to respond to the input operation based on the operation mapping relationship.

[0241] Optionally, a module 1402 is added for:

[0242] Based on the coordinate difference and the preset operation mapping relationship, determining the operation mapping relationship corresponding to the target application interface;

[0243] The operation mapping relationship corresponding to the target application interface is stored.

[0244] Optionally, a module 1402 is added for:

[0245] The note layer is mounted on a subnode of the target application interface, and in response to the target application interface switching from foreground operation to background operation, the note layer is controlled to switch from foreground operation to background operation;

[0246] In response to the target application interface switching from background operation to foreground operation, the note layer is controlled to switch from background operation to foreground operation.

[0247] Optionally, the device further comprises a note display module, which is used to:

[0248] In a case where the target application interface is displayed, in response to a note display operation, determining the note layer corresponding to the target application interface;

[0249] The note layer corresponding to the target application interface is displayed, and the note layer includes historical note content on the target application interface.

[0250] Optionally, the device further includes a note display module, which is used to:

[0251] In response to a display operation on the note layer, obtaining the interface link corresponding to the note layer;

[0252] The target application interface and the note layer corresponding to the target application interface are displayed based on the interface link, and the note layer includes historical note content on the target application interface.

[0253] Fig.15 It is a flowchart of a note adding method shown in another exemplary embodiment of the present application, and the method includes the following steps.

[0254] Step 1501, display the target application interface.

[0255] The target application interface can be the interface corresponding to a third-party application (such as a browser interface, a reader interface, a chat application interface, a video application interface, etc.), or it can be the interface corresponding to an application that comes with the user terminal (such as a calendar application interface, a camera application interface, etc.).

[0256] The user terminal is a device with a display screen, such as a smart phone, a tablet computer, a laptop computer, etc., which is not limited in the embodiments of the present application.

[0257] Optionally, the user terminal has a capacitive touch screen and supports stylus input.

[0258] Step 1502, in response to the note adding operation on the target application interface, create a note layer corresponding to the target application interface, and the note layer is used to carry the notes.

[0259] In some embodiments, the note layer can be located on the upper layer of the target application interface. In some embodiments, all or part of the area in the note layer has transparency, so that the user can see the content of the target application interface through the note layer. In some embodiments, the transparency of the note layer can be set. Exemplarily, when triggering the note adding operation, the user can choose to create a note layer with a transparency of 100% or 80%. On the note layer with full or partial transparency, there can also be some patterns, designs, borders, etc., and such deformations are within the scope of protection of this specification.

[0260] In some embodiments, the note layer may be located at the lower layer of the target application interface. In some embodiments, all or part of the area in the target application interface has transparency, so that the user can see the content of the note layer through the target application interface.

[0261] In some embodiments, a target application interface may create one or more corresponding note layers. For example, a target application interface may create three corresponding note layers located on the upper layer of the target application interface and two note layers located on the lower layer of the target application interface.

[0262] Optionally, the note adding operation is a function key operation, a shortcut key operation, a gesture operation, etc.

[0263] Optionally, the note adding operation is triggered by the stylus, for example, by a physical switch button of the stylus.

[0264] Optionally, the vertices of the note layer are aligned with the vertices of the target application interface, for example, the upper left vertex of the note layer is aligned with the upper left vertex of the target application interface, and the upper right vertex of the note layer is aligned with the upper right vertex of the target application interface.

[0265] Optionally, the width of the note layer is consistent with the physical screen width of the user terminal. For example, if the physical screen width of the user terminal is 1080px, the width of the note layer is also 1080px.

[0266] Optionally, the length of the note layer is extended along with the length of the target application interface. For example, when the user does not perform a sliding operation, the length of the target application interface is 1920px, and the length of the note layer is consistent with the length of the target application interface, also 1920px; when the user performs a sliding operation, the length of the target application interface is extended to 2680px along with the sliding, and the length of the note layer is also extended to 2680px.

[0267] Optionally, the length of the note layer is greater than, equal to, or less than the length of the target application interface.

[0268] Optionally, the width of the note layer is greater than, equal to, or less than the width of the target application interface.

[0269] The note layer is used to carry notes. Notes can include text, lines, doodles, pictures, audio and video, and other content. In some embodiments, notes can be input by handwriting (such as inputting handwritten text and doodles through a stylus) or by keyboard input (such as inputting a specific font through a keyboard based on an input method). In some embodiments, notes can also be input in other ways, such as inputting pictures through local files, or uploading audio and video content through links. The embodiment of this application does not limit the input method of notes.

[0270] In some embodiments, the note layer may include one or more controls for controlling the note layer. Figure 3 As shown, the note layer 320 may include a minimize control 322 for minimizing the note layer; the note layer 320 may include a close control 323 for closing the note layer. In some embodiments, the note layer may also include a zoom control, a position adjustment control, etc.

[0271] In some embodiments, in response to different operations, different controls may be displayed or hidden in the note layer. Figure 5 As shown, when the user clicks on the doodle 521 on the note layer 520 , a doodle toolbar 522 is displayed on the note layer 520 for performing operations such as deleting, modifying, and saving the doodle 521 .

[0272] In some embodiments, the target application interface and the note layer can be displayed simultaneously through the freeform mode in the multi-window mode of the Android system, and the user can adjust the size and position of the note layer.

[0273] Step 1503, when an input operation is received, determining an object of the input operation, the object of the input operation including at least one of a target application interface and a note layer.

[0274] In the case where a note layer is created on the upper layer of the target application interface, when the user performs an input operation, the input operation may only target the note layer (for example, the user wants to add handwritten text on the note layer, but has no operation on the target application interface), or only target the target application interface (for example, the user wants to select some text in the target application interface, or click a function button in the target application interface, but has no operation on the note layer), or the input operation may also target both the target application interface and the note layer (for example, the user wants to zoom, slide, etc. on both simultaneously). Therefore, when receiving the user's input operation, it is first necessary to determine the input operation's target, and then respond to the input operation on the target.

[0275] Figure 2 The flowchart of determining the object of the input operation is shown in an exemplary embodiment of the present application.

[0276] When an input operation is received, an object of the input operation is determined, where the object includes at least one of a target application interface and a note layer.

[0277] Input actions are actions triggered by the user on the note layer.

[0278] Optionally, different input operations may include different features, such as different operation types, indicated operation positions, and input methods.

[0279] Exemplarily, the operation type of the input operation may be an action type of the input operation, such as a multi-finger operation, a single-finger operation, a sliding operation, a zooming operation, and the like.

[0280] Exemplarily, the operation position indicated by the input operation may be a position on the note layer, or a position on the target application interface corresponding to the position on the note layer.

[0281] Exemplarily, the input operation may be performed by a touch pen or by a touch finger.

[0282] Optionally, the input operation targets only the note layer; or, the input operation targets only the target application interface; or, the input operation targets both the note layer and the target application interface.

[0283] The method for determining the object of the input operation may include at least one of the following steps or a combination thereof.

[0284] 1. Step 211, based on the operation type of the input operation, determine the object of the input operation.

[0285] In some embodiments, the correspondence between the operation type of the input operation and the action object may be determined according to a preset rule.

[0286] In a possible implementation, when the operation type of the input operation is a multi-finger operation, the objects of action are determined to be the target application interface and the note layer.

[0287] Among them, multi-finger operation refers to an operation performed simultaneously with two or more fingers, such as a two-finger reverse swipe operation, a three-finger upward swipe operation, etc.

[0288] In a possible usage scenario, when a user performs a multi-finger operation, the user intends to operate both the note layer and the target application interface at the same time, such as zooming in and out, swiping up, etc. on both the note layer and the target application interface at the same time. Therefore, the objects corresponding to the multi-finger operation can be the target application interface and the note layer.

[0289] For example, Figure 3 As shown, a transparent note layer 320 is created on the upper layer of the browser interface 310, and the user's hand-drawn graffiti 321 is located on the note layer 320. When the user performs a multi-finger operation (i.e., a zoom operation) of sliding two fingers in opposite directions, the zoom operation is applied to both the browser interface 310 and the note layer 320. That is, after the zoom operation, the content on the browser interface 310 is enlarged by a certain proportion, the note layer 320 is also enlarged by a certain proportion, and the graffiti 321 located on the note layer 320 is also enlarged by a certain proportion.

[0290] It is understandable that due to the need for drawing the accompanying drawings, the vertices of the browser interface 310 and the note layer 320 are not aligned in the drawing to highlight the difference between the two. In actual applications, the vertices of the target application interface and the note layer are usually aligned. Other embodiments of the present application and the corresponding drawings also follow this principle and do not constitute a limitation on the position of the target application interface and the note layer.

[0291] In a possible implementation, when the operation type of the input operation is a long press operation, the action object is determined to be a target application interface.

[0292] Among them, a long press operation refers to an input operation in which the contact time with the screen exceeds a threshold (such as 1 second).

[0293] In a possible usage scenario, when the user performs a long press operation, the user intends to operate the target application interface, such as selecting and copying text, pictures, etc. in the target application interface. Therefore, the object of the long press operation may be the target application interface.

[0294] For example, Figure 4As shown, in the browser interface 410, there is a text 411 with the words "XX Review Blogger". On the upper layer of the browser interface 410, a transparent note layer 420 is created. When the user performs a long press operation on the note layer 420, the object of the long press operation is the browser interface 410. That is, the effect of the long press operation is to select the text 411 located in the browser interface 410, forming a selection box 412. As shown in the figure, the user's long press operation on the note layer can be transmitted to the target application interface to achieve the target application interface's response to the operation.

[0295] 2. Step 212: determining an object of the input operation based on the operation position indicated by the input operation.

[0296] In a possible usage scenario, when a user performs an input operation on a note layer, it can be determined whether there is an operation object at the operation position indicated by the input operation in the note layer or the target application interface, and whether the operation object supports the input operation.

[0297] Optionally, the operation object may be text, picture, audio or video, function button, control, note, doodle or other object.

[0298] In a possible implementation, there is an operation object at the operation position indicated by the input operation in the note layer, and the operation object supports the input operation, and the operation object is determined to be the note layer.

[0299] For example, Figure 5 As shown, a transparent note layer 520 is created on the upper layer of the browser interface 510. On the note layer 520, there is a doodle 521 hand-drawn by the user, and the doodle 521 circles the date "2022-10-05" on the browser interface 510. When the user performs a click operation on the note layer 520, there is an operation object (doodle 521) at the position corresponding to the click operation in the note layer 520, and the doodle 521 supports the click operation. Therefore, it is determined that the object of the click operation is the note layer 520. The click operation acts on the doodle 521 on the note layer 520, and the doodle toolbar 522 is displayed. The doodle toolbar 522 is used to delete, modify, save, and other operations on the doodle 521.

[0300] In a possible implementation, there is an operation object at the operation position indicated by the input operation in the target application interface, and the operation object supports the input operation, and the operation object is determined to be the target application layer.

[0301] For example, Figure 6As shown, in the browser interface 610, there is a thumbs up icon 611 for giving a thumbs up. A transparent note layer 620 is created on the upper layer of the browser interface 610. When the user performs a click operation on the note layer 620, in the browser interface 610, there is an operation object (thumbs up icon 611) at the position corresponding to the click operation, and the thumbs up icon 611 supports the click operation; whereas in the note layer 620, there is no operation object at the position corresponding to the click operation. Therefore, it is determined that the object of the click operation is the browser interface 610, not the note layer 620. The click operation is transmitted to the browser interface 610, and acts on the thumbs up icon 611, forming a thumbs up icon 612.

[0302] 3. Step 213: Determine the object of the input operation based on the input mode corresponding to the input operation.

[0303] In a possible usage scenario, the user may use a stylus pen or a finger to touch the screen to implement an input operation. Optionally, the input operation corresponds to an input method of a stylus pen or a touching finger.

[0304] Exemplarily, when the terminal screen is a capacitive touch screen, the terminal can identify whether the input method is a stylus or a touching finger according to the position, size and touch sensitivity of the touch point.

[0305] In a possible implementation, when the input operation is performed using a stylus pen, the action object is determined to be a note layer.

[0306] For example, Figure 7 As shown, a transparent note layer 720 is created on the upper layer of the browser interface 710. The user performs a drawing operation on the note layer 720 based on the stylus 730. When the user terminal recognizes that the input method of the drawing operation is the stylus, it determines that the object of the drawing operation is the note layer 720. Therefore, the drawing content 721 of the drawing operation is displayed on the note layer 720.

[0307] In a possible implementation, when the input operation is performed by touching a finger, the object of action is determined to be a target application interface.

[0308] For example, Figure 4 As shown, the user clicks the text 411 on the browser interface 410 with a finger through the note layer 420. When the user terminal recognizes that the input method of the click operation is a touch finger, it determines that the object of the click operation is the browser interface 410, and forms a selection box 412 for the text 411 on the browser interface 410.

[0309] In a possible scenario, to determine the object of the input operation, using only one of the above methods may not be accurate enough. Therefore, the above methods may be combined, or other methods may be used to determine the object of the input operation.

[0310] For example, Figure 5 As shown, when the user performs a click operation on the note layer 520, there is an operation object (graffiti 521) at the position corresponding to the click operation in the note layer 520, and the graffiti 521 supports the click operation; however, in the browser interface 510, there is a date "2022-10-05" near the position corresponding to the click operation, and the date also supports the click operation. Therefore, the user terminal may mistakenly interpret the user's intention of clicking the date on the browser interface 510 as the intention of clicking the graffiti 521 on the note layer 520, resulting in incorrect determination of the object of action.

[0311] To resolve the situation where the object of action is incorrectly determined, in some embodiments, the operation type of the input operation, the operation position indicated by the input operation, and the input method of the input operation can be combined to jointly determine that the object of the input operation is the note layer and / or the target application interface.

[0312] Exemplarily, if the input method of the click operation is a stylus pen, and there is graffiti on the note layer and the word "date" on the browser interface at the position corresponding to the click operation, and both the graffiti and the word "date" support the click operation, then the action object is determined to be the note layer; if the input method of the click operation is a touching finger, and there is graffiti on the note layer and the word "date" on the browser interface at the position corresponding to the click operation, and both the graffiti and the word "date" support the click operation, then the action object is determined to be the browser interface; if the input method of the click operation is a stylus pen, and there is a date on the browser interface at the position corresponding to the click operation, the word "date" supports the click operation, and there is no operation object on the note layer, then the action object is determined to be the browser interface.

[0313] As such, methods of jointly determining the object of the input operation by combining at least two of the operation type of the input operation, the operation position indicated by the input operation, and the input method of the input operation are all within the protection scope of this specification.

[0314] 4. Step 214, based on the trigger state of the object selection control in the note layer, determine the object of the input operation.

[0315] To solve the problem of incorrectly determining the action object, in some embodiments, at least one action object selection control is provided in the note layer for determining the action object.

[0316] In a possible implementation, an action object selection control is provided in the note layer, and different action objects are determined based on different trigger operations on the action object selection control. For example, when the user clicks the action object selection control, the action object is determined to be the note layer; when the user double-clicks the action object selection control, the action object is determined to be the target application interface; when the user long presses the action object selection control, the action object is determined to be the note layer and the target application interface.

[0317] In another possible implementation, two action object selection controls are provided in the note layer, namely, a note layer selection control and a target application interface selection control. In response to a triggering operation on the note layer selection control, the action object is determined to be the note layer; in response to a triggering operation on the target application interface selection control, the action object is determined to be the target application interface; in response to triggering operations on the note layer selection control and the target application interface selection control, the action objects are determined to be the note layer and the target application interface.

[0318] See also Figure 8 , Figure 8 It is a schematic diagram of an action object selection control shown in an exemplary embodiment of the present application.

[0319] like Figure 8 As shown, a note layer 820 is created on the upper layer of the browser interface 810 , and two kinds of action object selection controls may be provided on the note layer 820 , namely a note layer selection control 821 and a browser interface selection control 822 .

[0320] In one possible implementation, when the user turns on or off the above-mentioned two types of action object selection controls at the same time, the action objects of subsequent input operations are determined to be the note layer 820 and the browser interface 810; when the user only turns on the note layer selection control 821, the action object of subsequent input operations is determined to be the note layer 820; when the user only turns on the browser interface selection control 822, the action object of subsequent input operations is determined to be the browser interface 810.

[0321] Step 1504: transparently transmit the input operation to the action object so that the action object responds to the input operation.

[0322] Transparent transmission refers to the technical means of passing input operations to the note layer and / or the target application interface. By transparently transmitting the input operation to the object, the object can respond to the input operation, so that the user can control the object.

[0323] In a possible manner, the transparent transmission can be implemented based on the touch event processing mechanism of the Android system. Specifically, the Android system can distribute the input operation through InputDispatcher to distribute the input operation to the action object.

[0324] In some embodiments, transparent transmission may be achieved through other methods, which are not limited in this specification.

[0325] For example, Figure 6 As shown, after determining that the object of the click operation is the browser interface 610, the user terminal transmits the click operation to the browser interface 610, so that the browser interface 610 responds to the click operation and clicks on the like icon 611 on the browser interface 610 to form a like icon 612.

[0326] For example, Figure 3 As shown, after determining that the objects of the zoom operation are the browser interface 310 and the note layer 320, the user terminal transmits the zoom operation to the browser interface 310 and the note layer 320, so that the browser interface 310 and the note layer 320 respond to the zoom operation at the same time, thereby achieving synchronous zooming of the browser interface and the note layer.

[0327] To summarize, the object of the input operation is determined through at least one of the operation type of the input operation, the operation position indicated by the input operation, the input method of the input operation, and the trigger state of the object selection control, and the input operation is transmitted to the object. This can meet the user's needs for operating on the target application interface and / or the note layer. The user can operate on the target application interface and the note layer at the same time (such as sliding, zooming), or operate on the target application interface alone (such as selecting), or operate on the note layer alone (such as graffiti), thereby adding to the richness of the note layer function and improving the efficiency of note adding.

[0328] Regarding the method of creating a note layer on the upper layer of the target application interface, in a possible implementation, the note layer can be mounted on a child node of the target application interface. In a possible implementation, the ability of mounting parent-child nodes between layers of the Android system can be used to mount (reparent) the view corresponding to the note layer to the child node of the target application interface, so as to realize the synchronous display of the note layer on the upper layer of the target application interface.

[0329] In some embodiments, the running state of the note layer and the target application interface are consistent. For example, when the target application interface switches from foreground operation to background operation, the control note layer also switches from foreground operation to background operation; when the target application interface switches from background operation to foreground operation, the control note layer also switches from background operation to foreground operation.

[0330] In one possible scenario, a user can perform zooming and / or sliding operations on the note layer, and the zooming and / or sliding operations can be transmitted to the note layer and the target application interface at the same time. The note layer and the target application interface perform synchronous zooming / synchronous sliding based on the received zooming and / or sliding operations.

[0331] However, in the actual synchronous sliding / synchronous zooming process, the zoom ratio / sliding step size of the note layer and the target application interface may be different. For example, the default settings of the target application interfaces corresponding to different applications may be different, and their responses to the same zoom / sliding operation may also be different. For example, for a reader interface containing more text, a sliding operation defaults to sliding the interface down 270px, while for a social application interface containing more large pictures, the same sliding operation defaults to sliding the interface down 540px. Based on the above differences, the note layer usually slides according to a sliding step size that is common to most applications. However, the common sliding step size may not be suitable for the current target application interface, so the position of the note on the target application interface will drift.

[0332] Fig. 9 It is a schematic diagram of a sliding operation process shown in an exemplary embodiment of the present application.

[0333] like Figure 8 As shown, a transparent note layer 920 is created on the upper layer of the browser interface 910, and a graffiti 921 drawn by the user is on the note layer 920, and the position of the graffiti 921 is circled in the words "the following contents" located on the browser interface 910. When the user performs a sliding operation, the objects of the sliding operation include both the browser interface 910 and the note layer 920. Among them, the browser interface 910 slides up 300px according to the default settings of the browser, and the words "the following contents" located thereon also slide up 300px; the note layer 920 slides up 275px according to the general settings, and the graffiti 921 located thereon also slides up 275px. Therefore, after the sliding operation, the graffiti 921 no longer circles the words "the following contents", that is, the position of the graffiti 921 on the browser interface 910 drifts.

[0334] In order to solve the problem of note position drift on the target application interface after zooming and / or sliding operations, in one possible implementation method, when the objects of action are the target application interface and the note layer, and the input operation is a zooming and / or sliding operation, the target application interface and the note layer after the operation response can be aligned.

[0335] Fig.10 Detailed description of the invention is given in detail below.

[0336] Step 1001, display the target application interface.

[0337] Step 1002, creating a note layer on the upper layer of the target application interface.

[0338] Step 1003, determining a first reference point in the note layer and a second reference point in the target application interface, wherein the initial coordinates of the first reference point and the second reference point are consistent.

[0339] The first reference point is at least one marking point in the note layer, and the second reference point is at least one marking point in the target application interface. In some embodiments, there are multiple first reference points and second reference points, and each of the first reference points corresponds to each of the second reference points.

[0340] The initial coordinates of the first / second reference point refer to the coordinates of the first / second reference point before the zoom operation and / or the slide operation.

[0341] Optionally, the initial coordinates of the first / second reference point are the coordinates of the first / second reference point relative to the upper left corner vertex of the terminal screen before the zoom operation and / or the sliding operation.

[0342] like Fig.11 As shown, before the sliding operation, a note layer 1120a is created on the browser interface 1110a.

[0343] In the note layer 1120a, three first reference points are determined, namely, first reference points 1121a, 1122a and 1123a. The initial coordinates of the three first reference points are the coordinates of the top left corner of the terminal screen before the sliding operation, namely, (200, -300), (50, -350) and (280, -350).

[0344] In the target application interface 1110a, three second reference points are determined, namely, second reference points 1111a, 1112a and 1113a. The initial coordinates of the three second reference points are the coordinates of the vertex of the upper left corner of the terminal screen before the sliding operation, which are consistent with the initial coordinates of the first reference point, namely, (200, -300), (50, -350) and (280, -350). The three first reference points correspond one to one with the three second reference points.

[0345] Step 1004: determine a reference point feature corresponding to the second reference point, where the reference point feature is a feature of the second reference point and pixels around the second reference point.

[0346] Optionally, the reference point feature of the second reference point includes the RGB values of the pixel points at the initial coordinates of the second reference point. Optionally, the reference point feature of the second reference point includes the RGB values of the pixel points on the periphery of the initial coordinates of the second reference point. For example, the RGB values of all the pixel points within a circular area with a radius of 5px near the initial coordinates of the second reference point.

[0347] In a possible implementation, the reference point feature of the second reference point can be obtained through image recognition technology. In a possible implementation, the target application source code corresponding to the target application interface can also be called to obtain the reference point feature of the second reference point. The embodiments of the present application do not limit the manner of obtaining the reference point feature of the second reference point.

[0348] Exemplarily, Fig.11 The reference point features of the 3 second reference points in [example] can be the features of the second reference point and the pixel points on its periphery. For example, the feature of the second reference point 1111a with coordinates (200, -300) and the pixel points on its periphery is the RGB value of the image near ": "; the feature of the second reference point 1112a with coordinates (50, -350) and the pixel points on its periphery is the RGB value of the image near the character "今"; the feature of the second reference point 1113a with coordinates (280, -350) and the pixel points on its periphery is the RGB value of the image near the character "记".

[0349] Step 1005, in the case where the input operation is a zoom operation and / or a sliding operation, pass the input operation transparently to the target application interface and the note layer.

[0350] In the case where the input operation is a zoom operation and / or a sliding operation, pass the input operation transparently to the target application interface and the note layer to simultaneously zoom / slide the target application interface and the note layer.

[0351] Step 1006, obtain the first real-time coordinates of the first reference point after obtaining the operation response, and based on the reference point feature corresponding to the second reference point, identify the second reference point from the target application interface after the operation response, and obtain the second real-time coordinates of the second reference point.

[0352] The first / second real-time coordinates refer to the coordinates of the first / second reference point after the zoom operation and / or the sliding operation.

[0353] Optionally, the first / second real-time is the coordinates of the first / second reference point relative to the top left vertex of the terminal screen after the zoom operation and / or the sliding operation.

[0354] In some embodiments, the second reference point can be identified from the target application interface after the operation response through image recognition technology based on the reference point features corresponding to the second reference point. In some embodiments, the source code of the target application corresponding to the target application interface can also be called to determine the second real-time coordinates of the second reference point based on the reference point features of the second reference point. The embodiments of the present application do not limit the method for determining the second real-time coordinates of the second reference point.

[0355] The scaling ratio / sliding step of the target application interface and the note layer may be inconsistent. As Fig.10 shown, the browser interface 1110a slides up 250px according to the browser default settings to form the browser interface 1110b; the note layer 1120a slides up 200px according to the general settings to form the note layer 1120b.

[0356] In the note layer 1120b after the upward sliding operation, the positions of the three first reference points slide up 200px with the note layer 1120b and become the first reference points 1121b, 1122b, and 1123b; the first real-time coordinates of the first reference points 1121b, 1122b, and 1123b are respectively: (200, -100), (50, -150), and (280, -150).

[0357] In the browser interface 1110b after the upward sliding operation, the positions of the three second reference points slide up 250px with the browser interface 1110b and become the second reference points 1111b, 1112b, and 1113b; the second real-time coordinates of the second reference points 1111b, 1112b, and 1113b are respectively: (200, -50), (50, -100), and (280, -100). Optionally, the second real-time coordinates of the second reference points 1111b, 1112b, and 1113b are respectively identified based on the features of the second reference point 1111a and its surrounding pixel points as the RGB values of the image near ":", the features of the second reference point 1112a and its surrounding pixel points as the RGB values of the image near the character "今", and the features of the second reference point 1113a and its surrounding pixel points as the RGB values of the image near the character "记".

[0358] Step 1007, in the case where the first real-time coordinates are different from the second real-time coordinates, perform alignment processing on the note layer based on the coordinate difference between the first real-time coordinates and the second real-time coordinates.

[0359] In some embodiments, based on the coordinate difference between the first real-time coordinates and the second real-time coordinates, adjust the position of the note layer so that the first real-time coordinates of the first reference points in the note layer are consistent with the second real-time coordinates of the second reference points in the target application interface.

[0360] As Fig.11 As shown, the sliding step lengths of the note layer 1120b and the browser interface 1110b are inconsistent, resulting in a 50px difference in the vertical coordinates between the first real-time coordinates of the first reference points 1121b, 1122b and 1123b and the second real-time coordinates of the second reference points 1111b, 1112b and 1113b. Therefore, based on this difference, the note layer 1120b can be slid up 50px again to align the note layer 1120b with the browser interface 1110b, thereby achieving synchronous sliding up of the note layer and the browser interface.

[0361] In this embodiment, the note layer is processed by the difference between the first real-time coordinates of the first reference point and the second real-time coordinates of the second reference point. This can solve the problem of position drift of the notes on the target application interface after the zoom operation and / or the sliding operation, so that the note layer and the target application interface can achieve synchronous sliding / synchronous zooming.

[0362] In order to solve the problem of note position drift on the target application interface after zooming and / or sliding operations, in another possible implementation method, the default settings of the target application (such as the default sliding step for a sliding operation, the default zoom ratio for a zoom operation, etc.) can be obtained in advance, the default settings of the target application can be stored, and the response of the note layer to the input operation can be controlled based on the default settings of the target application, so that the zoom ratio / sliding step of the note layer is adapted to the corresponding target application interface.

[0363] In some embodiments, when an operation mapping relationship corresponding to the target application interface is stored, the note layer is controlled to respond to the input operation based on the operation mapping relationship.

[0364] The operation mapping relationship includes at least one of a zoom ratio mapping relationship and a sliding distance mapping relationship. For example, the operation mapping relationship may include a default sliding step length of a sliding operation, a default zoom ratio of a zoom operation, and the like.

[0365] In some embodiments, when the operation mapping relationship corresponding to the target application interface is not stored, the note layer is controlled to respond to the input operation based on the preset operation mapping relationship. For example, the note layer is controlled to respond to the sliding and / or zooming operation based on the universal sliding step size and / or universal zoom ratio of common applications.

[0366] In some embodiments, when the operation mapping relationship corresponding to the target application interface is not stored, the preset operation mapping relationship can also be updated based on the difference between the first real-time coordinates of the first reference point and the second real-time coordinates of the second reference point obtained in the alignment process, so as to obtain the operation mapping relationship corresponding to the target application interface, and the obtained result can be saved for application in subsequent zoom operations / sliding operations.

[0367] like Fig.11 As shown, in a sliding operation, the note layer slides up 200px according to the preset operation mapping relationship. In the alignment process, the difference between the first real-time coordinate of the first reference point and the second real-time coordinate of the second reference point is 50px. Therefore, 250px can be stored as the operation mapping relationship corresponding to the browser interface. The next time the user slides the browser interface, it can slide directly according to the sliding step of 250px without the need for alignment processing, thereby improving the efficiency of the note layer sliding synchronously with the target application interface.

[0368] In this embodiment, when the operation mapping relationship corresponding to the target application interface is stored, the note layer is controlled to respond to the input operation through the operation mapping relationship. The zoom ratio / sliding step size adapted to the target application interface can be used to directly realize synchronous sliding / synchronous zooming without the need for alignment processing, thereby improving the utilization efficiency of the note layer; when the operation mapping relationship corresponding to the target application interface is not stored, the preset operation mapping relationship can be adjusted based on the result of the alignment processing for application in subsequent zoom operations / sliding operations, thereby improving the utilization efficiency of subsequent note layers.

[0369] In some embodiments, in response to a note saving operation, the note layer and the interface link corresponding to the target application interface are stored in association, and the interface link is used to point to the target application interface.

[0370] Optionally, the note saving operation is a function key operation, a shortcut key operation, a gesture operation, etc.

[0371] Optionally, the interface link is a Uniform Resource Locator (URL) or a deep link.

[0372] The interface link is used to point to the target application interface. In some embodiments, the note layer can be opened based on a specific application (such as a note application, a sticky note application). When the note layer is displayed, the target application interface corresponding to the note layer can be called based on the interface link. In some embodiments, when the target application interface is displayed, the interface link is also used to call the note layer corresponding to the target application interface.

[0373] Optionally, the database associates and stores the interface link corresponding to the target application interface according to the note layer ID corresponding to the note layer.

[0374] In one possible scenario, when the user reopens the target application interface, the user can open the notes added in the past, and then continue to view, edit, modify, delete, and save the notes added in the past.

[0375] In some embodiments, when the user terminal displays the target application interface, in response to the note display operation, the note layer corresponding to the target application interface is determined based on the interface link corresponding to the target application interface.

[0376] Optionally, the note display operation is a click operation, a function key operation, a shortcut key operation, a gesture operation, etc.

[0377] Optionally, the note display operation is triggered by a stylus, for example, by a note display button of the stylus.

[0378] In some embodiments, when the user reopens the target application interface, the terminal can display the note layer corresponding to the target application interface based on the interface link corresponding to the target application interface.

[0379] In some embodiments, a note bar containing multiple note layer thumbnails may be displayed on the target application interface, and the user may click and select any note layer thumbnail in the note bar to display the note layer corresponding to the note layer thumbnail.

[0380] like Fig.12 As shown, the user terminal displays a browser application interface 1210, and a note bar 1220 containing multiple note layer thumbnails is displayed on the browser application interface 1210, for example, note layer thumbnails 1221, 1222, and 1223. When the user clicks on the note layer thumbnail 1222, the note display operation is triggered, and the user terminal determines the note layer 1230 corresponding to the note layer thumbnail 1222 based on the interface link corresponding to the target application interface, and displays the note layer 1230. Among them, the notes on the note layer 1230 are consistent with the notes displayed by the note layer thumbnail 1222.

[0381] In some embodiments, the interface link corresponding to the target application interface is used to point to the historical note content. Based on the interface link corresponding to the target application interface, the user terminal displays a note layer on the upper layer of the target application interface, and the note layer includes the historical note content on the target application interface.

[0382] In this embodiment, by calling the historical note content through the interface link corresponding to the target application interface, the historical note content can be displayed on the upper layer of the target application interface, so that the user can continue to view, edit, modify, delete and save the notes added in the past.

[0383] In one possible scenario, when the user views the note layer alone (for example, when operating alone through a note application or a sticky note application), the user can open the corresponding target application interface or a screenshot of the target application interface based on the interface link corresponding to the note layer, and then compare the content of the note layer with that of the corresponding target application interface.

[0384] In some embodiments, the user terminal obtains an interface link corresponding to the note layer in response to a display operation on the note layer.

[0385] Optionally, the display operation of the note layer is function key operation, shortcut key operation, gesture operation, etc.

[0386] Optionally, the display operation of the note layer is triggered by the stylus, for example, by an application display button of the stylus.

[0387] In some embodiments, the interface link corresponding to the note layer is used to point to the target application interface corresponding to the note layer. In some embodiments, the interface link corresponding to the note layer is used to point to a screenshot of the target application interface corresponding to the note layer. The user terminal displays the target application interface or a screenshot corresponding to the target application interface based on the interface link, and displays the note layer on the upper layer of the target application interface, and the note layer contains the historical note content on the target application interface.

[0388] like Fig.13 As shown, the user terminal displays a note layer 1320. When the user performs a display operation on the note layer, the user terminal displays the corresponding target application interface 1310 based on the interface link, and displays the note layer 1320 on the upper layer of the target application interface 1310, and the note layer 1320 contains the historical note content on the target application interface.

[0389] In this embodiment, by calling the target application interface through the interface link corresponding to the note layer, the target application interface can be displayed simultaneously when viewing historical notes, and the content in the note layer and its corresponding target application interface can be compared and viewed, thereby improving the efficiency of viewing notes.

[0390] In one possible scenario, the interface link corresponding to the note layer may be invalid, resulting in the inability to call the target application interface and display the note layer on top of the target application interface. For example, when there is a server failure, gateway blocking, or the URL of the target application is updated, the interface link corresponding to the note layer may be invalid.

[0391] To solve the problem that the interface link corresponding to the note layer may become invalid, in some embodiments, in response to a note save operation, the note layer and the snapshot link corresponding to the target application interface may be stored in association.

[0392] The snapshot link is used to point to the web snapshot of the target application interface. When the interface link corresponding to the note layer is invalid, the snapshot link can be called.

[0393] In some embodiments, in response to a display operation on the note layer, a snapshot link corresponding to the note layer is obtained; a web page snapshot of the target application interface is displayed based on the snapshot link; and the note layer is displayed on an upper layer of the web page snapshot, wherein the note layer includes historical note content on the target application interface.

[0394] In this embodiment, by associating and storing the note layer and the snapshot link corresponding to the target application interface, when the interface link of the target application interface becomes invalid, a web page snapshot of the target application interface can still be obtained, and the historical note content can be displayed on the upper layer of the web page snapshot, thereby compensating for the inconvenience of note display caused by the invalid web page.

[0395] In one possible scenario, the content of the note is displayed dynamically over time. For example, the content of the note is the derivation process of a formula, and the content of the note at different time points is different steps of the formula; for another example, the content of the note is an annotation of a video played on the target application interface, and different annotation contents correspond to different time points.

[0396] To achieve dynamic display of note content, in some embodiments, in response to a note saving operation, the user terminal uses the note layer and the display time corresponding to the note in the note layer as the note dynamic display information.

[0397] The dynamic display information of the notes represents the display duration, display order, and other information of different notes in the note layer. In some embodiments, the dynamic display information of the notes can be determined based on the input operation. For example, the input time of formula 1 in the note is recorded as t_1, and the input time of formula 2 is recorded as t_2, then the dynamic display information of the notes can be recorded as (formula 1: t_1; formula 2: t_2). Optionally, in addition to the input time of the formula, the dynamic display information of the notes can also include information such as the writing process of the formula and the order of strokes.

[0398] In some embodiments, the database stores the dynamic display information of the notes and the interface link corresponding to the target application interface in association; in response to the note display operation, based on the interface link, the note layer is displayed on the upper layer of the target application interface. The historical note content in the note layer is dynamically displayed based on the dynamic display information of the notes.

[0399] Exemplarily, a note layer is displayed on the upper layer of the target application interface. When the display time of the target application interface reaches t_1, formula 1 is displayed on the note layer; when the display time of the target application interface reaches t_2, formula 2 is displayed on the note layer.

[0400] When the target application interface is a video playback interface, the dynamically displayed note content may also have a temporal correlation with the video playback content.

[0401] Optionally, the video playback interface is a video website interface, or a camera application interface, etc.

[0402] In some embodiments, the user terminal can obtain video dynamic display information of the video playback interface.

[0403] The video dynamic display information includes at least one frame in the video and the display time corresponding to the at least one frame. Exemplarily, the display time of the first key frame of the video is T_1, and the display time of the second key frame is T_2, then the video dynamic display information can be recorded as (first key frame: T_1; second key frame: T_2). Optionally, in addition to the display time of the key frame, the video dynamic display information can also include information such as the display time of other frames and the display content of the video frame.

[0404] In some embodiments, in response to a note saving operation, the user terminal obtains time association information corresponding to the video dynamic display information and the note dynamic display information.

[0405] The time association information represents the time association between the dynamic display of the video and the dynamic display of the notes. In some embodiments, the time association information can be determined based on the input operation. For example, when a user derives a formula on the note layer while playing a video, the input time of the formula can be associated with the display time of the video frame displayed at the same time. Exemplarily, if the time when the user inputs Formula 1 is t_1, and the video frame displayed at the same time is the second key frame, the time association information can be recorded as (t_1: T_1).

[0406] In some embodiments, the database associates and stores video dynamic display information, note dynamic display information, time association information, and an interface link corresponding to the target application interface.

[0407] In some embodiments, in response to the note display operation, the user terminal displays a note layer on the upper layer of the target application interface based on the interface link, wherein the historical note content in the note layer is dynamically displayed along with at least one frame in the video based on the time association information.

[0408] Exemplarily, a note layer is displayed on the upper layer of the target application interface, and when the playing time of the video played in the target application interface reaches T_1, Formula 1 is displayed on the note layer.

[0409] In this embodiment, by associating and storing the note dynamic display information and the interface link corresponding to the target application interface, it is possible to dynamically display the note content on the note layer; by associating and storing the video dynamic display information, the note dynamic display information, the time association information and the interface link corresponding to the target application interface, it is possible to synchronously display the corresponding dynamic notes while the video is playing, enriching the scenarios for adding notes and improving the efficiency of adding notes to video content.

[0410] Fig.16 : is a structural block diagram of a note adding device provided by another exemplary embodiment of the present application, the device comprising:

[0411] Display module 1601, used to display the target application interface;

[0412] An adding module 1602, configured to create a note layer corresponding to the target application interface in response to a note adding operation on the target application interface, wherein the note layer is used to carry notes;

[0413] The action object determination module 1603 is used to determine the action object of the input operation when receiving the input operation, and the action object includes at least one of the target application interface and the note layer;

[0414] The transparent transmission module 1604 is used to transparently transmit the input operation to the action object so that the action object responds to the input operation.

[0415] Optionally, the action object determination module 1603 is used to:

[0416] Determining the object of the input operation based on the operation type of the input operation; or,

[0417] Determining the object of the input operation based on the operation position indicated by the input operation; or,

[0418] Determine the object of the input operation based on the input mode corresponding to the input operation; or,

[0419] The action object of the input operation is determined based on the triggering state of the action object selection control in the note layer, and at least one action object selection control is set in the note layer, and different action object selection controls correspond to different action objects.

[0420] Optionally, the action object determination module 1603 is used to:

[0421] In a case where the operation type of the input operation is a multi-finger operation, determining that the action objects are the target application interface and the note layer;

[0422] When the operation type of the input operation is a long press operation, the action object is determined to be the target application interface.

[0423] Optionally, the action object determination module 1603 is used to:

[0424] In the case where the input operation is performed using a stylus pen, determining that the action object is the note layer;

[0425] In the case where the input operation is performed by touching a finger, the action object is determined to be the target application interface, or the target application interface and the note layer.

[0426] Optionally, the action object determination module 1603 is used to:

[0427] There is an operation object at the operation position indicated by the input operation in the note layer, and the operation object supports the input operation, and the operation object is determined to be the note layer;

[0428] An operation object exists at the operation position indicated by the input operation in the target application interface, and the operation object supports the input operation, and the operation object is determined to be the target application layer.

[0429] Optionally, when the action object is the target application interface and the note layer, and the input operation is a zoom operation and / or a sliding operation, the transparent transmission module 1604 is used to:

[0430] The target application interface and the note layer after the operation response are aligned.

[0431] Optionally, after creating the note layer corresponding to the target application interface, the transparent transmission module 1604 is used to:

[0432] Determine a first reference point in the note layer, and determine a second reference point in the target application interface, wherein initial coordinates of the first reference point and the second reference point are consistent;

[0433] The aligning of the target application interface and the note layer after the operation response includes:

[0434] Acquire a first real-time coordinate of the first reference point after the operation response, and a second real-time coordinate of the second reference point;

[0435] When the first real-time coordinates are different from the second real-time coordinates, the note layer is aligned based on the coordinate difference between the first real-time coordinates and the second real-time coordinates.

[0436] Optionally, after determining the second reference point in the target application interface, the transparent transmission module 1604 is used to:

[0437] Determine a reference point feature corresponding to the second reference point, where the reference point feature is a feature of the second reference point and pixels around the second reference point;

[0438] The step of acquiring the second real-time coordinates of the second reference point includes:

[0439] Based on the reference point feature, the second reference point is identified from the target application interface after the operation response, and the second real-time coordinates of the second reference point are acquired.

[0440] Optionally, before aligning the target application interface and the note layer after the operation response, the transparent transmission module 1604 is used to:

[0441] In the case where the operation mapping relationship corresponding to the target application interface is not stored, controlling the note layer to respond to the input operation based on a preset operation mapping relationship, wherein the operation mapping relationship includes at least one of a zoom ratio mapping relationship and a sliding distance mapping relationship;

[0442] The method further comprises:

[0443] In a case where an operation mapping relationship corresponding to the target application interface is stored, the note layer is controlled to respond to the input operation based on the operation mapping relationship.

[0444] Optionally, after aligning the note layer based on the coordinate difference between the first real-time coordinate and the second real-time coordinate, the transparent transmission module 1604 is used to:

[0445] Based on the coordinate difference and the preset operation mapping relationship, determining the operation mapping relationship corresponding to the target application interface;

[0446] The operation mapping relationship corresponding to the target application interface is stored.

[0447] Optionally, the note layer is mounted on a subnode of the target application interface, and in response to the target application interface switching from foreground operation to background operation, the note layer is controlled to switch from foreground operation to background operation;

[0448] In response to the target application interface switching from background operation to foreground operation, the note layer is controlled to switch from background operation to foreground operation.

[0449] Optionally, the device further includes a note saving module, which is used to:

[0450] In response to the note saving operation, the note layer and the interface link corresponding to the target application interface are stored in association, and the interface link is used to point to the target application interface.

[0451] Optionally, the note saving module is used to:

[0452] In a case where the target application interface is displayed, in response to a note display operation, determining the note layer corresponding to the target application interface;

[0453] The note layer corresponding to the target application interface is displayed, and the note layer includes historical note content on the target application interface.

[0454] Optionally, the note saving module is used to:

[0455] In response to a display operation on the note layer, obtaining the interface link corresponding to the note layer;

[0456] The target application interface and the note layer corresponding to the target application interface are displayed based on the interface link, and the note layer includes historical note content on the target application interface.

[0457] Fig.17 It is a structural block diagram of a terminal device provided by an exemplary embodiment of the present application.

[0458] The terminal can execute the note adding method of the above embodiment. The terminal can be a terminal device with a display screen, such as a smart phone, a laptop, a tablet computer, a navigation device, a smart TV, etc. The terminal may also be called a user device, a portable terminal, etc. The terminal may include one or more of the following components: a processor 1710 and a memory 1720.

[0459] Optionally, the processor 1710 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 1720, and calling data stored in the memory 1720. Optionally, the processor 1710 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1710 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a baseband chip. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the touch display; the NPU is used to implement artificial intelligence (AI) functions; and the baseband chip is used to process wireless communications. It is understandable that the above-mentioned baseband chip may not be integrated into the processor 1710, but may be implemented by a separate chip.

[0460] The memory 1720 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 1720 includes a non-transitory computer-readable storage medium. The memory 1720 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1720 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc.; the data storage area may store data created according to the use of the electronic device (such as audio data, a phone book), etc.

[0461] In addition, those skilled in the art will appreciate that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figures, or a combination of certain components, or a different arrangement of components.

[0462] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations of the above aspects.

[0463] The embodiment of the present application also provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the audio transmission method described in the above embodiment. Optionally, the computer-readable storage medium may include: ROM, RAM, solid state drive (SSD, Solid State Drives) or optical disk, etc. Among them, RAM may include resistance random access memory (ReRAM, Resistance Random Access Memory) and dynamic random access memory (DRAM, Dynamic Random Access Memory).

[0464] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for adding notes, characterized in that: The method comprises: Display the target application interface; In response to a note adding operation on the target application interface, creating a note layer corresponding to the target application interface, the note layer being used to carry the note; In response to the note saving operation, the note layer and the interface link corresponding to the target application interface are stored in association, and the interface link is used to point to the target application interface.

2. The method according to claim 1, characterized in that After creating the note layer corresponding to the target application interface, the method further includes: When receiving an input operation, determining an object of the input operation, wherein the object of the input operation includes at least one of the target application interface and the note layer; The input operation is transparently transmitted to the action object, so that the action object responds to the input operation.

3. The method according to claim 2, characterized in that The determining of the object of the input operation includes at least one of the following methods: Determining the object of the input operation based on the operation type of the input operation; or, Determining the object of the input operation based on the operation position indicated by the input operation; or, Determining the object of the input operation based on the input mode corresponding to the input operation; or, The action object of the input operation is determined based on a trigger state of an action object selection control in the note layer.

4. The method according to claim 3, characterized in that The determining the object of the input operation based on the operation type of the input operation includes: In a case where the operation type of the input operation is a multi-finger operation, determining that the action objects are the target application interface and the note layer; When the operation type of the input operation is a long press operation, the action object is determined to be the target application interface.

5. The method according to claim 3, characterized in that: The determining the object of the input operation based on the input mode corresponding to the input operation includes: In the case where the input operation is performed using a stylus pen, determining that the action object is the note layer; In the case where the input operation is performed by touching a finger, the action object is determined to be the target application interface, or the target application interface and the note layer.

6. The method according to claim 3, characterized in that: The determining the action object of the input operation based on the operation position indicated by the input operation includes: There is an operation object at the operation position indicated by the input operation in the note layer, and the operation object supports the input operation, and the operation object is determined to be the note layer; An operation object exists at the operation position indicated by the input operation in the target application interface, and the operation object supports the input operation, and the operation object is determined to be the target application layer.

7. The method according to claim 2, characterized in that When the action objects are the target application interface and the note layer, and the input operation is a zoom operation and / or a sliding operation; After transparently transmitting the input operation to the action object, the method further includes: The target application interface and the note layer after the operation response are aligned.

8. The method according to claim 7, characterized in that After creating the note layer corresponding to the target application interface, the method further includes: Determine a first reference point in the note layer, and determine a second reference point in the target application interface, wherein initial coordinates of the first reference point and the second reference point are consistent; The aligning of the target application interface and the note layer after the operation response includes: Acquire a first real-time coordinate of the first reference point after the operation response, and a second real-time coordinate of the second reference point; When the first real-time coordinates are different from the second real-time coordinates, the note layer is aligned based on the coordinate difference between the first real-time coordinates and the second real-time coordinates.

9. The method according to claim 8, characterized in that After determining the second reference point in the target application interface, the method further includes: Determine a reference point feature corresponding to the second reference point, where the reference point feature is a feature of the second reference point and pixels around the second reference point; The step of acquiring the second real-time coordinates of the second reference point includes: Based on the reference point feature, the second reference point is identified from the target application interface after the operation response, and the second real-time coordinates of the second reference point are acquired.

10. The method according to claim 7, characterized in that Before aligning the target application interface after the operation response with the note layer, the method includes: In the case where the operation mapping relationship corresponding to the target application interface is not stored, controlling the note layer to respond to the input operation based on a preset operation mapping relationship, wherein the operation mapping relationship includes at least one of a zoom ratio mapping relationship and a sliding distance mapping relationship; The method further comprises: In a case where an operation mapping relationship corresponding to the target application interface is stored, the note layer is controlled to respond to the input operation based on the operation mapping relationship.

11. The method according to claim 10, characterized in that After aligning the note layer based on the coordinate difference between the first real-time coordinate and the second real-time coordinate, the method further includes: Based on the coordinate difference and the preset operation mapping relationship, determining the operation mapping relationship corresponding to the target application interface; The operation mapping relationship corresponding to the target application interface is stored.

12. The method according to claim 1, characterized in that The note layer is mounted on a subnode of the target application interface; The method further comprises: In response to the target application interface switching from foreground operation to background operation, controlling the note layer to switch from foreground operation to background operation; In response to the target application interface switching from background operation to foreground operation, the note layer is controlled to switch from background operation to foreground operation.

13. The method according to claim 1, characterized in that The method further comprises: In a case where the target application interface is displayed, in response to a note display operation, determining the note layer corresponding to the target application interface; The note layer corresponding to the target application interface is displayed, and the note layer includes historical note content on the target application interface.

14. The method according to claim 1, characterized in that The method further comprises: In response to a display operation on the note layer, obtaining the interface link corresponding to the note layer; The target application interface and the note layer corresponding to the target application interface are displayed based on the interface link, and the note layer includes historical note content on the target application interface.

15. A method for adding notes, characterized in that: The method comprises: Display the target application interface; In response to a note adding operation on the target application interface, creating a note layer corresponding to the target application interface, the note layer being used to carry the note; When receiving an input operation, determining an object of the input operation, wherein the object of the input operation includes at least one of the target application interface and the note layer; The input operation is transparently transmitted to the action object, so that the action object responds to the input operation.

16. The method according to claim 15, characterized in that The determining of the object of the input operation includes at least one of the following methods: Determining the object of the input operation based on the operation type of the input operation; or, Determining the object of the input operation based on the operation position indicated by the input operation; or, Determining the object of the input operation based on the input mode corresponding to the input operation; or, The action object of the input operation is determined based on a trigger state of an action object selection control in the note layer.

17. The method according to claim 16, characterized in that The determining the object of the input operation based on the operation type of the input operation includes: In a case where the operation type of the input operation is a multi-finger operation, determining that the action objects are the target application interface and the note layer; When the operation type of the input operation is a long press operation, the action object is determined to be the target application interface.

18. The method according to claim 16, characterized in that The determining the object of the input operation based on the input mode corresponding to the input operation includes: In the case where the input operation is performed using a stylus pen, determining that the action object is the note layer; In the case where the input operation is performed by touching a finger, the action object is determined to be the target application interface, or the target application interface and the note layer.

19. The method according to claim 16, characterized in that The determining the action object of the input operation based on the operation position indicated by the input operation includes: There is an operation object at the operation position indicated by the input operation in the note layer, and the operation object supports the input operation, and the operation object is determined to be the note layer; An operation object exists at the operation position indicated by the input operation in the target application interface, and the operation object supports the input operation, and the operation object is determined to be the target application layer.

20. The method according to claim 15, characterized in that When the action objects are the target application interface and the note layer, and the input operation is a zoom operation and / or a sliding operation; After transparently transmitting the input operation to the action object, the method further includes: The target application interface and the note layer after the operation response are aligned.

21. The method according to claim 20, characterized in that After creating the note layer corresponding to the target application interface, the method further includes: Determine a first reference point in the note layer, and determine a second reference point in the target application interface, wherein initial coordinates of the first reference point and the second reference point are consistent; The aligning of the target application interface and the note layer after the operation response includes: Acquire a first real-time coordinate of the first reference point after the operation response, and a second real-time coordinate of the second reference point; When the first real-time coordinates are different from the second real-time coordinates, the note layer is aligned based on the coordinate difference between the first real-time coordinates and the second real-time coordinates.

22. The method according to claim 21, characterized in that After determining the second reference point in the target application interface, the method further includes: Determine a reference point feature corresponding to the second reference point, where the reference point feature is a feature of the second reference point and pixels around the second reference point; The step of acquiring the second real-time coordinates of the second reference point includes: Based on the reference point feature, the second reference point is identified from the target application interface after the operation response, and the second real-time coordinates of the second reference point are acquired.

23. The method according to claim 7, characterized in that Before aligning the target application interface after the operation response with the note layer, the method includes: In the case where the operation mapping relationship corresponding to the target application interface is not stored, controlling the note layer to respond to the input operation based on a preset operation mapping relationship, wherein the operation mapping relationship includes at least one of a zoom ratio mapping relationship and a sliding distance mapping relationship; The method further comprises: In a case where an operation mapping relationship corresponding to the target application interface is stored, the note layer is controlled to respond to the input operation based on the operation mapping relationship.

24. The method according to claim 23, characterized in that After aligning the note layer based on the coordinate difference between the first real-time coordinate and the second real-time coordinate, the method further includes: Based on the coordinate difference and the preset operation mapping relationship, determining the operation mapping relationship corresponding to the target application interface; The operation mapping relationship corresponding to the target application interface is stored.

25. The method according to claim 15, characterized in that The note layer is mounted on a subnode of the target application interface, and in response to the target application interface switching from foreground operation to background operation, the note layer is controlled to switch from foreground operation to background operation; In response to the target application interface switching from background operation to foreground operation, the note layer is controlled to switch from background operation to foreground operation.

26. The method according to claim 15, characterized in that The method comprises: In response to the note saving operation, the note layer and the interface link corresponding to the target application interface are stored in association, and the interface link is used to point to the target application interface.

27. The method according to claim 15, characterized in that The method further comprises: In a case where the target application interface is displayed, in response to a note display operation, determining the note layer corresponding to the target application interface; The note layer corresponding to the target application interface is displayed, and the note layer includes historical note content on the target application interface.

28. The method according to claim 15, characterized in that The method further comprises: In response to a display operation on the note layer, obtaining the interface link corresponding to the note layer; The target application interface and the note layer corresponding to the target application interface are displayed based on the interface link, and the note layer includes historical note content on the target application interface.

29. A note adding device, characterized in that: The device comprises: Display module, used to display the target application interface; An adding module, configured to create a note layer corresponding to the target application interface in response to a note adding operation on the target application interface, wherein the note layer is used to carry notes; The saving module is used to store the note layer and the interface link corresponding to the target application interface in association with each other in response to the note saving operation, wherein the interface link is used to point to the target application interface.

30. A note adding device, characterized in that: The device comprises: Display module, used to display the target application interface; An adding module, configured to create a note layer corresponding to the target application interface in response to a note adding operation on the target application interface, wherein the note layer is used to carry notes; An action object determination module, configured to determine an action object of the input operation when receiving the input operation, wherein the action object includes at least one of the target application interface and the note layer; The transparent transmission module is used to transparently transmit the input operation to the action object so that the action object responds to the input operation.

31. A terminal, characterized in that: The terminal includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement the note adding method as described in any one of claims 1 to 28.

32. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is loaded and executed by the processor to implement the note adding method as described in any one of claims 1 to 28.

33. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the processor of the terminal reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the terminal executes the note adding method as described in any one of claims 1 to 28.