Note recording with contextual information

Abstract

The present disclosure presents a method, an apparatus, and a computer program product for note recording. A set of text segments in a target window and a set of locations corresponding to the set of text segments may be obtained. A word graph corresponding to the target window can be constructed according to the set of text fragments and the set of positions, and the word graph comprises a plurality of words in the set of text fragments and a plurality of edges connecting the plurality of words. A current word that is currently input into the note may be detected. A word suggestion for the current word can be generated according to the word graph, and the word suggestion comprises words located behind the current word in the target window.

Description

Background Technology

[0001] Note-taking applications such as Microsoft's OneNote offer a convenient way to create digital notes. Users can use these applications to create digital notes containing text, tables, images, videos, and more to record ideas or manage tasks. These digital notes can be shared among multiple users for collaboration or communication. Summary of the Invention

[0002] This invention is provided to introduce a set of concepts, which will be further described in the following detailed description. This invention is not intended to identify key or essential features of the protected subject matter, nor is it intended to limit the scope of the protected subject matter.

[0003] Embodiments of this disclosure provide methods, apparatus, and computer program products for note-taking. A set of text fragments in a target window and a set of positions corresponding to the set of text fragments can be obtained. A word graph corresponding to the target window can be constructed based on the set of text fragments and the set of positions, the word graph containing multiple words in the set of text fragments and multiple edges connecting the multiple words. The current word currently input into the note can be detected. Word suggestions for the current word can be generated based on the word graph, the word suggestions containing words located after the current word in the target window.

[0004] It should be noted that one or more of the above aspects include the features described in detail below and specifically pointed out in the claims. The following description and drawings set forth certain illustrative features of the one or more aspects in detail. These features merely indicate various ways in which the principles of each aspect can be employed, and this disclosure is intended to include all such aspects and their equivalents. Attached Figure Description

[0005] The following description will take into account several aspects disclosed, which are provided to illustrate rather than limit the aspects disclosed.

[0006] Figure 1 An exemplary process for note-taking according to an embodiment of this disclosure is shown.

[0007] Figure 2 An exemplary target window according to an embodiment of this disclosure is shown.

[0008] Figure 3 Examples of generating word suggestions and text fragment suggestions for characters according to embodiments of this disclosure are shown.

[0009] Figure 4An exemplary process for constructing a word graph corresponding to a target window according to an embodiment of this disclosure is shown.

[0010] Figure 5 An example of assigning an index to a text fragment according to an embodiment of this disclosure is shown.

[0011] Figure 6 An exemplary word map is shown according to an embodiment of this disclosure.

[0012] Figure 7 An exemplary process for generating word suggestions for the current word, according to an embodiment of this disclosure, is shown.

[0013] Figure 8 An example of generating word suggestions for the current word according to an embodiment of this disclosure is shown.

[0014] Figure 9 An exemplary process for generating annotation suggestions for text fragments in notes, according to embodiments of the present disclosure, is shown.

[0015] Figure 10 An example of generating annotation suggestions for text fragments in notes according to embodiments of this disclosure is shown.

[0016] Figure 11 An exemplary process for searching for text fragments in notes according to embodiments of the present disclosure is shown.

[0017] Figure 12 An example of searching for text fragments in notes according to an embodiment of this disclosure is shown.

[0018] Figure 13 This is a flowchart of an exemplary method for note-taking according to embodiments of the present disclosure.

[0019] Figure 14 An exemplary apparatus for note-taking according to an embodiment of the present disclosure is shown.

[0020] Figure 15 Another exemplary apparatus for note-taking according to an embodiment of this disclosure is shown. Detailed Implementation

[0021] This disclosure will now be discussed with reference to several exemplary embodiments. It should be understood that this discussion of embodiments is merely intended to enable those skilled in the art to better understand and thus implement the embodiments of this disclosure, and is not intended to teach any limitation on the scope of this disclosure.

[0022] Users can create digital notes while browsing the internet or participating in video conferences using their computing devices. For simplicity, this article will refer to digital notes simply as notes. The content of notes may come from information on the screen of the computing device, such as web pages in a browser or presentations presented in a meeting application. Existing note-taking applications may not be aware of information on the screen, so users need to manually input the content on the screen into their notes. This method is inefficient and reduces the user's focus when browsing the web or participating in video conferences.

[0023] Embodiments of this disclosure propose note-taking using contextual information. Contextual information can be information from the screen or target window of a computing device when the note-taking application is enabled on that device. Here, the target window refers to the current window of the target application whose content is to be recorded. The target application can be an application independent of the note-taking application or an application integrated with a note-taking application. As examples, the target application can be a browser application, a video conferencing application, a document authoring application, etc. A set of text fragments in the target window and a set of positions corresponding to those text fragments can be obtained using techniques such as Optical Character Recognition (OCR) and User Interface (UI) automation. Here, a text fragment refers to a group of texts that are physically close to each other. A line of text, a paragraph of text, etc., can be referred to as a text fragment. A word graph corresponding to the target window can be constructed based on the set of text fragments and the set of positions. The word graph can contain multiple words in the set of text fragments and multiple edges connecting the multiple words. When the current word currently input into the note is detected, word suggestions for the current word can be generated based on the word graph. Word suggestions can include words located after the current word in the target window. The words included in the word suggestions can be referred to as candidate words. Users can select candidate words from the word suggestions by clicking predefined keyboard shortcuts and / or mouse buttons. When a selection of a candidate word is detected, the selected candidate word can be automatically entered into the notes. Additionally, text snippet suggestions for the current word can be generated. These text snippet suggestions can contain a text snippet of the current word in the target window. Users can select text snippets from the text snippet suggestions by clicking keyboard shortcuts and / or mouse buttons that are different from those used to select candidate words. Accordingly, the selected text snippets can be automatically entered into the notes. The technical effect of the above method is to improve the speed of note-taking and enhance the user's focus when interacting with the target application.

[0024] When a set of words or a text fragment is detected in the notes, a Large Language Model (LLM) can generate annotation suggestions for the text fragments in the notes based on a set of text fragments in the target window. The annotation suggestions can include comments on the text fragment. In this paper, a Large Language Model refers to a deep learning model capable of understanding the meaning of natural language, generating natural language text, or performing other natural language tasks. It should be understood that a Large Language Model includes multimodal models capable of performing natural language and other modal processing tasks. The Large Language Model can generate annotation suggestions for the text fragments in the notes based on its understanding of the content in the target window and the content in the notes, utilizing its knowledge base and logical reasoning capabilities. Users can select the annotation suggestion by clicking predefined keyboard shortcuts and / or mouse buttons. When a selection of an annotation suggestion is detected, the comments contained in the annotation suggestion can be appended to the text fragment. The content in the notes may be fragmented and semantically incoherent. The technical effect of annotating notes is to enable users to recall the meaning of the recorded content when reading the notes later, and to facilitate subsequent searches within the notes. The advantage of using large language models to annotate notes is that it improves the efficiency and accuracy of note annotation.

[0025] The annotations generated in the above manner can be stored as key-value pairs along with their corresponding text fragments in the notes. The annotation can be stored as the "key," and the text fragment as the "value" corresponding to that key. When a search query for a note is received, the annotation set of the note can be searched for to match the search query using methods such as vector similarity matching and large language models. Subsequently, the text fragment corresponding to that annotation in the note can be extracted as the search result for the search query. The technical advantage of this method is that it enables accurate and rapid retrieval of text fragments in notes even when there is no precise search query matching the text fragment.

[0026] The various embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0027] Figure 1 An exemplary process 100 for note-taking according to an embodiment of the present disclosure is shown. Process 100 can be performed by a note-taking application according to an embodiment of the present disclosure.

[0028] At point 102, a set of text fragments in the target window and a set of positions corresponding to those text fragments can be obtained. This step can be performed when the note-taking application is enabled or when the content in the target window changes. The target window can be the current window of the target application whose content is to be recorded. The target application can be an application independent of the note-taking application or an application integrated with a note-taking application. As examples, the target application can be a browser application, a video conferencing application, a document authoring application, etc. A set of text fragments in the target window and a set of positions corresponding to those text fragments can be obtained using techniques such as OCR and UI automation. Figure 2 An exemplary target window 200 according to an embodiment of the present disclosure is shown. In the target window 200, text fragments 202 to 210 are shown. The position of each text fragment can be defined by the coordinates of the top-left vertex of the bounding box that defines the text fragment. For example, the position of text fragment 202 can be defined by the coordinates of vertex 212.

[0029] Return to reference Figure 1 At position 104, a word graph corresponding to the target window can be constructed based on the set of text fragments and their positions. The word graph can contain multiple words from the set of text fragments and multiple edges connecting these words. This will be discussed later in conjunction with... Figure 4 This illustrates an exemplary process for constructing a word graph.

[0030] At position 106, characters entered into the notes can be detected.

[0031] At point 108, word suggestions and / or text fragment suggestions for a character can be generated through string comparison. Preferably, this step can be performed when the character is the first or a predetermined number of characters preceding the current text fragment, i.e., when a text fragment begins to be entered in the notes. Words matching the character can be identified from multiple words in the target window through string comparison. Word suggestions for the character can be generated based on the identified words. Alternatively or additionally, text fragments matching the character can be identified from a set of text fragments in the target window through string comparison. Text fragment suggestions for the character can be generated based on the identified text fragments. Word suggestions and text fragment suggestions can be presented in a pop-up window. Preferably, text fragment suggestions can be generated when a predetermined number of characters are detected in the current text fragment. This predetermined number is, for example, "2". That is, text fragment suggestions are only presented when the current text fragment contains two or more characters. The technical effect of this approach is to improve the accuracy of text fragment suggestions. Multiple text fragments matching the character may exist in the target window. Preferably, text fragment suggestions can contain only a predetermined number of text fragments. This predetermined number can be "1". The text snippets included in the text snippet suggestions can be those that users are most likely to pay attention to, such as the text snippet closest to the top left corner of the target window. The technical advantage of this approach is that it reduces the number of text snippets presented to the user, thereby reducing the complexity of user interaction.

[0032] Figure 3 An example 300 of generating word suggestions and text fragment suggestions for characters according to an embodiment of the present disclosure is shown. In example 300, a note-taking application 302 is enabled, and a note 304 is created. The characters 306 "g" and 308 "i" are detected when input into the note 304. A word suggestion 310 for the characters 306 "g" and 308 "i" is generated, which includes the candidate word "git". The user can select the candidate word in the word suggestion 310 by clicking a predefined keyboard shortcut and / or mouse button. The selected candidate word can be automatically entered into the note 304. It should be understood that when multiple words matching the characters 306 "g" and 308 "i" exist in the target window, the word suggestion 310 can correspondingly include multiple words. In addition, the word suggestion 310 can be generated when only the character 306 "g" is detected and the character 308 "i" is not detected.

[0033] Additionally, text snippet suggestion 312 is generated for the characters 306 "g" and 308 "i", which contains text snippet 202 "git add text.text". Text snippets 204 and 206 also match the characters 306 "g" and 308 "i", but text snippet suggestion 312 only contains text snippet 202 because it is the text snippet closest to the top left corner of the target window and is most likely the text snippet the user is currently focusing on. It should be understood that it is also possible to include multiple text snippets in a text snippet suggestion. The user can select a text snippet from text snippet suggestion 312 by clicking a keyboard shortcut and / or mouse button, which is different from selecting candidate words. Accordingly, the selected text snippet can be automatically entered into note 304.

[0034] Return to reference Figure 1 At point 110, the current word entered into the note can be detected. The current word can be entered by the user by pressing letters, numbers, or other keys on the keyboard, or by selecting candidate words from the word suggestions.

[0035] In step 112, word suggestions can be generated for the current word based on the word graph. These suggestions can include words located after the current word in the target window. This will be discussed later. Figure 7 This section describes an exemplary process for generating word suggestions for the current word. Alternatively, text snippet suggestions for the current word can be generated, for example, through string comparison. These text snippet suggestions can be contained within a text snippet of the current word in a target window. The word suggestions and / or text snippet suggestions can be presented in a pop-up window.

[0036] At point 114, the selection of candidate words from the word suggestions can be detected. Users can select words from the word suggestions by clicking predefined keyboard shortcuts and / or mouse buttons.

[0037] At point 116, in response to detecting the selection of a candidate word in the word suggestions, the selected candidate word can be automatically entered into the note. The text fragment in the note can be updated accordingly.

[0038] Steps 110 through 116 can be performed iteratively until the input of the text fragment is complete.

[0039] In process 100, contextual information from the target window is used to generate word suggestions and text snippet suggestions for the characters or words currently entered into the note. Users can select words or text snippets from the word suggestions and text snippet suggestions to take notes. The technical effect of this method is to improve the speed of note-taking and enhance the user's focus on interacting with the target application.

[0040] It should be understood that the above text, in combination with... Figures 1 to 3 The described note-taking process is merely exemplary. Depending on the specific application requirements, the steps in the note-taking process can be replaced or modified in any way, and the process may include more or fewer steps. Furthermore, the specific order or hierarchy of the steps in process 100 is merely exemplary, and the note-taking process can be performed in an order different from that described. For example, in process 100, step 106 is described after step 104, but steps 104 and 106 can be performed simultaneously, or step 104 can be performed after step 106.

[0041] Figure 4 An exemplary process 400 for constructing a word map corresponding to a target window according to an embodiment of this disclosure is shown. Process 400 may correspond to Figure 1 Step 104 in the process.

[0042] At position 402, the text fragments can be sorted according to a set of positions corresponding to a set of text fragments in the target window. As mentioned earlier, the position of each text fragment can be defined by the coordinates of the top-left vertex of the bounding box that defines the text fragment. Text fragments whose positions are closer to the top-left corner of the target window can be sorted higher.

[0043] At position 404, an index can be assigned to the group of text fragments based on their sort order. Text fragments that are sorted earlier can be assigned smaller indices and have a higher priority accordingly. Figure 5 An example 500 for assigning indexes to text segments according to an embodiment of the present disclosure is shown. Text segments 202 to 210 are respectively assigned indexes “0” to “4”.

[0044] At position 406, the text fragment can be split into multiple words. This can be done by first detecting the language of the text fragment and then using a word segmentation technique corresponding to that language to split it into multiple words. Figure 5 Taking text fragments 202 to 210 as an example, this group of text fragments can be split into the following words: "git", "add", "text.text", "merge", "master", "readme.md", "yarn" and "@reduxjs / toolkit".

[0045] At point 408, for each of the plurality of words, a word index list for that word can be generated based on one or more indices of one or more text segments in which the word is located. In this document, a word index list refers to a list of indices corresponding to a single word. One or more indices of one or more text segments in which a word is located can be combined to form the word index list for that word. The word index list can indicate one or more text segments in which the word is located. The indices in the word index list can be arranged in ascending order. Continuing... Figure 5 Taking text fragments 202 to 210 as examples, the word index list for the word "git" can be [0,1,2], the word index list for the word "add" can be [0,2,3,4], and so on.

[0046] At point 410, multiple edges between multiple words can be created based on multiple word index lists corresponding to multiple words. For example, for every two words in the multiple words, it can be determined whether the word index lists corresponding to the two words contain a common index. If the word index lists corresponding to the two words contain a common index, it indicates that the two words are located in the same text segment. In this case, it can be determined whether the two words are adjacent in the text segment corresponding to the common index. If it is determined that the two words are adjacent in the text segment corresponding to the common index, i.e., there are no other words between the two words, an edge can be created between the two words. The direction of the edge can be from the earlier word to the later word.

[0047] At position 412, multiple words and edges can be combined to form a word graph. Since each edge in this word graph has a direction, it can be called a directed word graph.

[0048] Figure 6 An exemplary word map 600 according to an embodiment of this disclosure is shown. Word map 600 can be... Figure 5The word graph corresponding to the target window is shown in the figure. Word graph 600 includes the following words: “git”, “add”, “text.text”, “merge”, “master”, “readme.md”, “yarn”, and “@reduxjs / toolkit”. Below each word is a list of its word indices. The word index list can indicate one or more text segments in which the word is located. When two word index lists corresponding to two words contain a common index and the two words are adjacent, an edge can be created between the two words. The direction of the edge can be from the earlier word to the later word. For example, the word index list for the word “git” is [0,1,2], and the word index list for the word “add” is [0,2,3,4]. The word index lists for these two words contain a common index, namely “0” and “2”. This indicates that the words “git” and “add” are both located in the text segment with index “0” and the text segment with index “2”, respectively. It can be further determined whether there are other words between the word "git" and the word "add" in the text fragment with index "0" and / or index "2". The text fragment with index "0" is "git add text.text". The text fragment with index "2" is "git addreadme.md". In these two text fragments, the words "git" and "add" are adjacent. Therefore, an edge can be created between the words "git" and "add". The direction of the edge is from the word "git" to the word "add". Edges between other words can be created in the same way. It should be understood that... Figure 6 The word graph 600 shown is merely an example. Depending on the specific application requirements, the word graph can have any other structure and can include more or fewer elements.

[0049] The technical advantage of process 400 lies in its ability to construct a word map that corresponds to the relative position of the text in the target window. Using this word map, word suggestions for the words in the notes can be generated quickly and accurately, thereby improving the speed of note-taking.

[0050] It should be understood that the above text, in combination with... Figures 4 to 6 The described process for constructing a word map corresponding to a target window is merely exemplary. Depending on the specific application requirements, the steps in the process of constructing the word map can be replaced or modified in any way, and the process may include more or fewer steps. Furthermore, the specific order or hierarchy of the steps in process 400 is merely exemplary, and the process for constructing a word map corresponding to a target window can be performed in an order different from the described order.

[0051] Figure 7An exemplary process 700 for generating word suggestions for a current word according to an embodiment of this disclosure is shown. Process 700 may correspond to Figure 1 Step 112 in the process.

[0052] At position 702, the fragment index list of the text segment containing the current word can be determined. In this document, the fragment index list refers to the list of indices corresponding to a text segment containing one or more words. The fragment index list of a text segment can include the common index of a set of word index lists corresponding to a set of words in that text segment. For example, suppose the current word is "add", and the text segment in the note containing this current word is "git add", meaning the text segment contains the words "git" and "add". The word index list for the word "git" is [0,1,2], and the word index list for the word "add" is [0,2,3,4]. The common indexes of these two word index lists are "0" and "2". Therefore, the fragment index list of the text segment "git add" is [0,2]. The fragment index list can be updated each time a word is entered into the text segment.

[0053] At position 704, a group of words that have an edge with the current word can be determined based on the word graph. Preferably, the determined words should be the words pointed to by the edge, that is, the words located after the current word in the target window.

[0054] At point 706, candidate words whose word index list and fragment index list contain a common index can be identified within the word group. The technical effect of steps 704 and 706 is that word suggestions are only included in the target window following the current word, while ensuring that the text fragment formed by the current text fragment and candidate words in the notes is consistent with the text fragment in the target window, thereby improving the accuracy of word suggestions. This will be discussed later in conjunction with... Figure 8 The example in the text illustrates this point.

[0055] At point 708, word suggestions for the current word can be generated based on the identified candidate words. Preferably, when multiple candidate words exist, these candidate words can be sorted according to the first index in their word index list. The smaller the first index in the word index list, the higher the ranking. The smaller the word index, the closer the word is to the upper left corner of the screen in the target window, and the more likely it is to be the word the user is currently focusing on. The technical effect of this approach is that words more likely to be of interest to the user appear above the word suggestions, making them easier for the user to see and select, thereby further improving the speed of note-taking.

[0056] Figure 8 An example of generating word suggestions for the current word according to an embodiment of this disclosure is shown.

[0057] In Figure 800a, the current word 804 "git" input into note 802 is detected. The text fragment only includes this current word 804 "git". Accordingly, the fragment index list of the text fragment is consistent with the word index list of the current word 804 "git", i.e., [0,1,2]. Based on the word graph, a group of words that have edges with the current word 804 "git" can be determined. For example, refer to... Figure 6 In the word graph 600, a pair of words with an edge to the current word 804 "git" are the words "add" and "merge". Subsequently, candidate words whose word index lists and fragment index lists share a common index can be identified for "add" and "merge". The word index list for "add" is [0,2,3,4], which shares a common index with the fragment index list [0,1,2], i.e., index "0" and index "2". "Add" can be considered a candidate word. The word index list for "merge" is [1], which shares a common index with the fragment index list [0,1,2], i.e., index "1". "Merge" can also be considered a candidate word. Therefore, the generated word suggestion 806 includes two candidate words, namely "add" and "merge". It should be understood that in Figure 800a, the candidate words in word suggestion 806 are arranged vertically, but other arrangements, such as horizontal arrangements, are also possible.

[0058] Figure 800a also shows a text fragment suggestion 808 for the current word 804. Text fragments matching the current word can be identified from a set of text fragments in the target window through string comparison. Subsequently, text fragment suggestions for the current word can be generated based on the identified text fragments. For example, text fragments matching "git" from text fragments 202 to 210 could include text fragment 202 "git add text.text", text fragment 204 "git merge master", and text fragment 206 "git add readme.md". However, text fragment suggestion 808 only contains text fragment 202 "git add text.text". Text fragment 202 "git add text.text" is the text fragment closest to the top left corner of the target window and is likely the text fragment with the highest probability of user attention. The technical effect of including only one or a few text fragments in text fragment suggestion 808 is to reduce the number of text fragments presented to the user, thereby reducing the complexity of user interaction.

[0059] Suppose the user selects the candidate word "add" from word suggestion 806. The current word 810 input into note 802 is detected as "add". At this point, the text fragment is updated to "git add". Correspondingly, the fragment index list of the text fragment is updated to [0,2]. Based on the word graph, a group of words with edges existing between them and the current word 810 "add" can be determined. For example, see [reference]. Figure 6 In word graph 600, a group of words with edges to the current word 810 "add" are "text.text", "master", "readme.md", and "@reduxjs / toolkit". Subsequently, candidate words whose word index lists and fragment index lists share a common index can be identified among "text.text", "master", "readme.md", and "@reduxjs / toolkit". The word index list of "text.text" is [0], which shares a common index with the fragment index list [0,2], i.e., index "0". "text.text" can be considered a candidate word. The word index list of "master" is [1,3], which does not share a common index with the fragment index list [0,2]. "master" cannot be considered a candidate word. The word index list of "readme.md" is [2], which shares a common index with the fragment index list [0,2], i.e., index "2". "readme.md" can be considered a candidate word. The word index list for the word “@reduxjs / toolkit” is [4], which does not share an index with the fragment index list [0,2]. The word “@reduxjs / toolkit” cannot be a candidate word. Therefore, the generated word suggestion 812 includes two candidate words, namely the word “text.text” and the word “readme.md”, as shown in Figure 800b. In addition, the text fragment suggestion 814 for the current word 810 “add” still includes the text fragment 202 “git add text.text”.

[0060] As shown in Figure 800b, although the word graph identifies a group of words with edges to the current word 810 "add," including the words "master" and "@reduxjs / toolkit," these words are not considered as candidate words for generating word suggestions because their word index lists do not share a common index with the text fragment index list of the note "git add." This is consistent with the content in the target window, which does not contain the text fragments "git add master" and "git add@reduxjs / toolkit."

[0061] In the above process, a group of words that have edges with the current word are first identified based on the word graph. Then, candidate words whose word index list and fragment index list contain common indices are identified within this group. Next, word suggestions are generated based on the identified candidate words. The technical advantage of the word suggestion generation method proposed in this disclosure is that it ensures that word suggestions only include words located after the current word in the target window, while also ensuring that the text fragment formed by the current text fragment and candidate words in the notes is consistent with the text fragment in the target window, thereby improving the accuracy of word suggestions.

[0062] It should be understood that the above text, in combination with... Figures 7 to 8 The described process for generating word suggestions for the current word is merely exemplary. Depending on the specific application requirements, the steps in the process for generating word suggestions for the current word can be replaced or modified in any way, and the process may include more or fewer steps. Furthermore, the specific order or hierarchy of the steps in process 700 is merely exemplary, and the process for generating word suggestions for the current word may be performed in an order different from the described order.

[0063] Figure 9 An exemplary process 900 for generating annotation suggestions for text fragments in notes, according to an embodiment of the present disclosure, is illustrated. In process 900, annotation suggestions 922 for text fragments 904 in notes can be generated based on a set of text fragments 902 in a target window using a large language model 920. Annotation suggestions 922 may include annotations 924 for text fragments 904 generated by the large language model 920. The large language model 920 is, for example, a Generative Pre-trained Transformer (GPT) model.

[0064] A cue word 912 to be provided to the large language model 920 can be created using a cue word creator 910. The cue word creator 910 can create the cue word 912 based on a set of text fragments 902 in the target window and text fragments 904 in the notes. Preferably, a cue word template 906 can be pre-designed for the cue word creator 910. The cue word template 906 may include multiple variable sections for loading the set of text fragments 902 in the target window and the text fragments 904 in the notes, respectively. Additionally, the cue word template 906 may include response instructions to guide the large language model 920 on how to respond, as well as several examples that can be learned.

[0065] The prompt word 912 can be provided to the large language model 920. Based on its understanding of a set of text fragments 902 in the target window and text fragments 904 in the notes, the large language model 920 can generate annotation suggestions 922 for the text fragments 904 in the notes, utilizing its knowledge base and logical reasoning capabilities. The annotation suggestions 922 contain annotations 924 for the text fragments 904 generated by the large language model 920.

[0066] Figure 10 An example of generating annotation suggestions for text fragments in notes according to embodiments of this disclosure is shown.

[0067] In Figure 1000a, text fragment 1004 in note 1002 is “yarn add@reduxjs / toolkit”. A large language model can be used to generate annotation suggestions 1006 for text fragment 1004 in the note, based on text fragments 202 to 210 in the target window. Annotation suggestion 1006 includes annotation 1008 for text fragment 1004 generated by the large language model based on its understanding of text fragments 202 to 210 in the target window and text fragment 1004 in the note, utilizing its knowledge base and logical reasoning capabilities; namely, “command for installing the Redux toolkit”.

[0068] Users can select the annotation suggestion 1006 by clicking predefined keyboard shortcuts and / or mouse buttons. If a selection of annotation suggestion 1006 is detected, annotation 1008 contained in annotation suggestion 1006 can be appended to text fragment 1004, as shown in Figure 1000b. Preferably, text fragment 1004 can be highlighted to remind the user that the text fragment has an appended annotation. Annotation 1008 can be automatically displayed when the cursor or other indicator hovers over text fragment 1004.

[0069] The content in notes may be fragmented and semantically disjointed. The technical benefit of annotating notes is that it helps users recall the meaning of the recorded content when reading them later, and facilitates subsequent searches within the notes. The technical benefit of using large language models to annotate notes is that it improves the efficiency and accuracy of note annotation.

[0070] It should be understood that the above text, in combination with... Figures 9 to 10 The described process for generating annotation suggestions for text snippets in notes is merely exemplary. Depending on the specific application requirements, the steps in the process for generating annotation suggestions can be replaced or modified in any way, and the process may include more or fewer steps.

[0071] The annotation 924 generated by process 900 can be stored together with its corresponding text fragment 904 in the notes as key-value pairs {key: value}, where annotation 924 can be stored as the "key" and text fragment 904 can be stored as the "value" corresponding to that key. This method facilitates subsequent searching of text fragments within the notes.

[0072] Figure 11 An exemplary process 1100 for searching text fragments in notes according to an embodiment of the present disclosure is illustrated. In process 1100, a search query 1104 for note 1102 may be received. Notes 1152 that match search query 1104 can be searched from a set of notes 1106 of note 1102. The set of notes 1106 may be... Figure 9 The process 900 generates the text. Subsequently, text fragment 1154 corresponding to annotation 1152 can be extracted from note 1102.

[0073] Search query 1104 can be provided in several ways. In one implementation, it can be done by clicking the search button in the note-taking application and entering search query 1104 in the search box that subsequently appears. In another implementation, a predefined search character can be entered directly in the notes, followed by search query 1104. As an example, the predefined search character could be the forward slash " / ".

[0074] The query vector 1112 for search query 1104 can be generated using encoder 1110. Encoder 1110 can be a machine learning model capable of generating vectors or embeddings based on text input. As an example, encoder 1110 can be a Bidirectional Encoder Representations from Transformers (BERT) model. Similarly, the annotation vector set 1122 corresponding to the annotation set 1106 can be generated using encoder 1120. Encoder 1120 can have the same model structure and parameters as encoder 1110.

[0075] Based on the query vector 1112 and the annotation vector set 1122, a subset 1132 of annotations similar to the search query 1104 can be searched from the annotation set 1106. For example, the annotation subset 1132 can be searched by performing semantic similarity matching on the query vector 1112 and the annotation vector set 1122 using the similarity matching module 1130. The similarity matching module 1130 can employ techniques such as Nearest Neighbor Search to search for several annotation vectors in the annotation vector set 1122 that are closest to the query vector 1112, thereby determining several annotations that are semantically most similar to the search query 1104. These annotations can then be combined into the annotation subset 1132.

[0076] The prompt word 1142 to be provided to the large language model 1150 can be created using the prompt word creator 1140. The large language model 1150 is, for example, a GPT model. The prompt word creator 1140 can create the prompt word 1142 based on the search query 1104 and the annotation subset 1132. Preferably, a prompt word template 1134 can be pre-designed for the prompt word creator 1140. The prompt word template 1134 may include multiple variable sections for loading the search query 1104 and the annotation subset 1132, respectively. In addition, the prompt word template 1134 may include response instructions for guiding how the large language model 1150 should respond.

[0077] The prompt word 1142 can be provided to the large language model 1150. The large language model 1150 can utilize its knowledge base, semantic understanding capabilities, logical reasoning abilities, etc., to identify annotations 1152 in the annotation subset 1132 that match the search query 1104. Subsequently, the text fragment 1154 corresponding to annotation 1152 in the notes 1102 can be extracted as the search result for the search query 1104. As mentioned earlier, annotations and their corresponding text fragments in the notes are stored together as key-value pairs. Therefore, the text fragment 1154 corresponding to annotation 1152 in the notes 1102 can be extracted quickly. The text fragment 1154 can be presented in various ways. In one embodiment, the text fragment 1154 can be presented in a pop-up window of the note-taking application. In another embodiment, the text fragment 1154 can be presented directly in the notes 1102.

[0078] Figure 12 An example of searching for a text fragment in a note according to an embodiment of the present disclosure is shown. In FIG1200a, a predefined search character " / " and search query 1204 "Redux" are entered in note 1202. Subsequently, in note 1202, a text fragment 1206 matching the search query 1204 "Redux", namely "yarn add@reduxjs / toolkit", is provided, as shown in FIG1200b.

[0079] In process 1100, upon receiving a search query for a note, a matching note can be searched from the note's annotation set using methods such as vector similarity matching and large language models. Subsequently, the text fragment corresponding to that note can be extracted from the note as the search result for the search query. The technical advantage of this method is that it enables accurate and rapid retrieval of text fragments within a note even when an accurate search query matching the text fragment is unavailable.

[0080] It should be understood that the above text, in combination with... Figures 11 to 12The described process for searching text fragments in notes is merely exemplary. Depending on the specific application requirements, the steps in the process for searching text fragments in notes can be replaced or modified in any way, and the process may include more or fewer steps. For example, in process 1100, encoders 1110 and 1120 and the similarity matching module 1130 first search for a subset 1132 of annotations similar to search query 1104 from annotation set 1106. Then, a large language model 1150 identifies annotations 1152 matching search query 1104 from annotation subset 1132. However, embodiments of this disclosure are not limited to this. In some embodiments, annotations matching search query 1104 can be directly identified from annotation set 1106 by the large language model 1150 without the processing of encoders 1110 and 1120 and the similarity matching module 1130. Furthermore, the specific order or hierarchy of the steps in process 1100 is merely exemplary, and the process for searching for text fragments in the notes may be performed in a different order than that described.

[0081] Figure 13 This is a flowchart of an exemplary method 1300 for note-taking according to an embodiment of the present disclosure.

[0082] At position 1310, a set of text fragments in the target window and a set of positions corresponding to the set of text fragments can be obtained.

[0083] At position 1320, a word graph corresponding to the target window can be constructed based on the set of text fragments and the set of positions. The word graph contains multiple words from the set of text fragments and multiple edges connecting the multiple words.

[0084] At position 1330, the current word input into the note can be detected.

[0085] At position 1340, word suggestions for the current word can be generated based on the word graph, the word suggestions including words located after the current word in the target window.

[0086] In one implementation, constructing a word graph corresponding to the target window may include: sorting the set of text segments according to the set of positions; assigning a set of indices to the set of text segments according to the sorting; splitting the set of text segments into multiple words; for each of the multiple words, generating a word index list for the word based on one or more indices of one or more text segments in which the word is located; creating multiple edges between the multiple words according to the multiple word index lists corresponding to the multiple words; and combining the multiple words and the multiple edges into the word graph.

[0087] Creating multiple edges between the plurality of words may include, for every two words in the plurality of words: determining that the two word index lists corresponding to the two words contain a common index; in response to determining that the two word index lists contain a common index, determining that the two words are adjacent in the text segment corresponding to the common index; and in response to determining that the two words are adjacent in the text segment corresponding to the common index, creating an edge between the two words.

[0088] In one implementation, generating word suggestions for the current word may include: determining a list of fragment indices of text segments containing the current word in the notes; determining a group of words that have edges with the current word based on the word graph; identifying candidate words in the group of words whose word index lists and fragment index lists share a common index; and generating word suggestions based on the identified candidate words.

[0089] In one implementation, method 1300 may further include: detecting the selection of a candidate word from the word suggestions; and automatically inputting the selected candidate word into the notes in response to detecting the selection of a candidate word from the word suggestions.

[0090] In one implementation, method 1300 may further include: detecting characters input into the notes; identifying words among the plurality of words that match the characters through string comparison; and generating word suggestions for the characters based on the identified words.

[0091] In one implementation, method 1300 may further include: detecting characters input into the notes; identifying text segments in the set of text segments that match the characters by string comparison; and generating text segment suggestions for the characters based on the identified text segments.

[0092] In one implementation, method 1300 may further include: generating annotation suggestions for text segments in the notes based on the set of text segments using a large language model, the annotation suggestions including annotations for the text segments.

[0093] Method 1300 may further include: detecting a selection of the annotation suggestion; and in response to detecting a selection of the annotation suggestion, appending the annotation to the text segment.

[0094] In one implementation, method 1300 may further include: receiving a search query for the notes; searching for notes that match the search query from a set of notes; and extracting text fragments from the notes that correspond to the notes.

[0095] It should be understood that method 1300 may also include any other steps / processes for note-taking according to embodiments of the present disclosure as described above.

[0096] Figure 14 An exemplary device 1400 for note-taking according to an embodiment of the present disclosure is shown.

[0097] The device 1400 may include: a text fragment and position acquisition module 1410, configured to acquire a set of text fragments in a target window and a set of positions corresponding to the set of text fragments; a word graph construction module 1420, configured to construct a word graph corresponding to the target window based on the set of text fragments and the set of positions, the word graph including multiple words in the set of text fragments and multiple edges connecting the multiple words; a current word detection module 1430, configured to detect the current word currently input into the notes; and a word suggestion generation module 1440, configured to generate word suggestions for the current word based on the word graph, the word suggestions including words located after the current word in the target window. Furthermore, the device 1400 may also include any other modules configured for note-taking according to embodiments of the present disclosure as described above.

[0098] Figure 15 Another exemplary device 1500 for note-taking according to an embodiment of the present disclosure is shown.

[0099] The device 1500 may include: a processor 1510; and a memory 1520 storing computer-executable instructions. When executed, the computer-executable instructions may cause the processor 1510 to: obtain a set of text fragments in a target window and a set of positions corresponding to the set of text fragments; construct a word graph corresponding to the target window based on the set of text fragments and the set of positions, the word graph including multiple words in the set of text fragments and multiple edges connecting the multiple words; detect the current word currently entered into the note; and generate word suggestions for the current word based on the word graph, the word suggestions including words located after the current word in the target window.

[0100] In one implementation, constructing a word graph corresponding to the target window may include: sorting the set of text segments according to the set of positions; assigning a set of indices to the set of text segments according to the sorting; splitting the set of text segments into multiple words; for each of the multiple words, generating a word index list for the word based on one or more indices of one or more text segments in which the word is located; creating multiple edges between the multiple words according to the multiple word index lists corresponding to the multiple words; and combining the multiple words and the multiple edges into the word graph.

[0101] Creating multiple edges between the plurality of words may include, for every two words in the plurality of words: determining that the two word index lists corresponding to the two words contain a common index; in response to determining that the two word index lists contain a common index, determining that the two words are adjacent in the text segment corresponding to the common index; and in response to determining that the two words are adjacent in the text segment corresponding to the common index, creating an edge between the two words.

[0102] In one implementation, generating word suggestions for the current word may include: determining a list of fragment indices of text segments containing the current word in the notes; determining a group of words that have edges with the current word based on the word graph; identifying candidate words in the group of words whose word index lists and fragment index lists share a common index; and generating word suggestions based on the identified candidate words.

[0103] In one embodiment, the computer-executable instructions, when executed, may also cause processor 1510 to: detect the selection of a candidate word in the word suggestions; and, in response to detecting the selection of a candidate word in the word suggestions, automatically input the selected candidate word into the notes.

[0104] In one embodiment, the computer-executable instructions, when executed, may further cause the processor 1510 to: detect characters input into the notes; identify, through string comparison, a word among the plurality of words that matches the character; and generate word suggestions for the character based on the identified word.

[0105] In one implementation, the computer-executable instructions, when executed, may also cause processor 1510 to: generate annotation suggestions for text segments in the notes based on the set of text segments using a large language model, the annotation suggestions including annotations for the text segments.

[0106] When executed, the computer-executable instructions may also cause the processor 1510 to: detect a selection of the annotation suggestion; and in response to detecting a selection of the annotation suggestion, append the annotation to the text segment.

[0107] In one embodiment, the computer-executable instructions, when executed, may further cause processor 1510 to: receive a search query for the notes; search for a note matching the search query from the note's set of notes; and extract a text fragment from the notes corresponding to the note.

[0108] It should be understood that the processor 1510 may also perform any other steps / processes of the method for note-taking according to the embodiments of the present disclosure as described above.

[0109] Embodiments of this disclosure provide a computer program product for note-taking, comprising a computer program executed by a processor to: obtain a set of text fragments in a target window and a set of positions corresponding to the set of text fragments; construct a word graph corresponding to the target window based on the set of text fragments and the set of positions, the word graph containing multiple words in the set of text fragments and multiple edges connecting the multiple words; detect a current word currently input into the note; and generate word suggestions for the current word based on the word graph, the word suggestions containing words located after the current word in the target window. Furthermore, the computer program may also be executed to implement any other steps / processes for carrying out the note-taking method according to embodiments of this disclosure as described above.

[0110] Embodiments of this disclosure can be embodied in a computer-readable medium for note-taking. The computer-readable medium may include instructions that, when executed, cause a processor to: obtain a set of text fragments in a target window and a set of positions corresponding to the set of text fragments; construct a word graph corresponding to the target window based on the set of text fragments and the set of positions, the word graph containing multiple words in the set of text fragments and multiple edges connecting the multiple words; detect a current word currently input into the note; and generate word suggestions for the current word based on the word graph, the word suggestions containing words located after the current word in the target window. Furthermore, when executed, the instructions may also cause the processor to perform any other steps / processes of the note-taking method according to embodiments of this disclosure as described above.

[0111] It should be understood that all operations in the methods described above are merely exemplary, and this disclosure is not limited to any operation in the methods or the order of such operations, but should cover all other equivalent variations under the same or similar concept. Furthermore, unless otherwise specified or clearly indicated from the context as relating to the singular form, the articles “a” and “an” as used in this specification and the appended claims should generally be interpreted as meaning “one” or “one or more”.

[0112] It should also be understood that all modules in the apparatus described above can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. Furthermore, any of these modules can be further functionally divided into sub-modules or combined together.

[0113] Processors have been described in conjunction with various devices and methods. These processors can be implemented using electronic hardware, computer software, or any combination thereof. Whether these processors are implemented as hardware or software will depend on the specific application and the overall design constraints imposed on the system. As an example, the processors, any portions of processors, or any combinations of processors given in this disclosure can be implemented using microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic units, discrete hardware circuits, and other suitable processing components configured to perform the various functions described in this disclosure. The functionality of the processors, any portions of processors, or any combinations of processors given in this disclosure can be implemented using software executed by a microprocessor, microcontroller, DSP, or other suitable platform.

[0114] Software should be broadly considered to refer to instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, procedures, functions, etc. Software may reside on a computer-readable medium. Computer-readable media may include, for example, memory, which may be, for example, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical disks, smart cards, flash memory devices, random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), registers, or removable disks. Although memory is shown as separate from the processor in several aspects set forth in this disclosure, memory may also reside within the processor, such as in a cache or register.

[0115] The above description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents of the elements of the various aspects described herein, known or likely to be known by one of ordinary skill in the art, are expressly incorporated herein and covered by the claims.

Claims

1. A method for note-taking, comprising: Obtain a set of text fragments in the target window and a set of positions corresponding to the set of text fragments; Based on the set of text fragments and the set of positions, a word graph corresponding to the target window is constructed. The word graph contains multiple words from the set of text fragments and multiple edges connecting the multiple words. Detect the current word entered into the note; as well as Based on the word graph, word suggestions are generated for the current word, and the word suggestions include words located after the current word in the target window.

2. The method according to claim 1, wherein, The construction of the word map corresponding to the target window includes: Sort the set of text fragments according to the set of positions; Based on the sorting, assign a set of indices to the set of text fragments; The set of text fragments is split into the multiple words; For each of the plurality of words, a word index list is generated based on one or more indices of one or more text segments in which the word is located; Based on a list of word indices corresponding to the multiple words, create multiple edges between the multiple words; and The multiple words and the multiple edges are combined to form the word graph.

3. The method according to claim 2, wherein, The creation of multiple edges between the plurality of words includes, for every two words in the plurality of words: The index lists of the two words corresponding to the two words are determined to contain a common index; In response to determining that the two word index lists contain a common index, it is determined that the two words are adjacent in the text segment corresponding to the common index; as well as In response to determining that the two words are adjacent in the text segment corresponding to the common index, an edge is created between the two words.

4. The method according to claim 1, wherein, The generation of word suggestions for the current word includes: Determine a list of fragment indices for the text segments in the notes that contain the current word; Based on the word graph, determine a group of words that have an edge with the current word; Identify candidate words in the set of words whose word index list and fragment index list contain a common index; and Based on the identified candidate words, the word suggestions are generated.

5. The method according to claim 1, further comprising: Detect the selection of candidate words from the word suggestions; as well as In response to detecting the selection of a candidate word from the word suggestions, the selected candidate word is automatically entered into the notes.

6. The method according to claim 1, further comprising: Detect characters input into the notes; By comparing strings, words that match the given characters among the plurality of words are identified; as well as Based on the identified words, word suggestions are generated for the characters.

7. The method according to claim 1, further comprising: Detect characters input into the notes; By comparing strings, the text fragments in the set of text fragments that match the character are identified; as well as Based on the identified text fragments, text fragment suggestions are generated for the characters.

8. The method according to claim 1, further comprising: Using a large language model, annotation suggestions are generated for the text fragments in the notes based on the set of text fragments, and the annotation suggestions include annotations for the text fragments.

9. The method according to claim 8, further comprising: Detect the selection of the suggested annotations; as well as In response to detecting a selection of the suggested annotation, the annotation is appended to the text segment.

10. The method according to claim 1, further comprising: Receive search queries for the notes; From the set of annotations in the notes, search for annotations that match the search query; as well as Extract the text segments from the notes that correspond to the annotations.

11. A device for note-taking, comprising: processor; as well as A memory storing computer-executable instructions, which, when executed, cause the processor to: Obtain a set of text fragments in the target window and a set of positions corresponding to the set of text fragments. Based on the set of text fragments and the set of positions, a word graph corresponding to the target window is constructed. The word graph contains multiple words from the set of text fragments and multiple edges connecting the multiple words. Detect the current word input into the note, and Based on the word graph, word suggestions are generated for the current word, and the word suggestions include words located after the current word in the target window.

12. The apparatus according to claim 11, wherein, The construction of the word map corresponding to the target window includes: Sort the set of text fragments according to the set of positions; Based on the sorting, assign a set of indices to the set of text fragments; The set of text fragments is split into the multiple words; For each of the plurality of words, a word index list is generated based on one or more indices of one or more text segments in which the word is located; Based on a list of word indices corresponding to the multiple words, create multiple edges between the multiple words; and The multiple words and the multiple edges are combined to form the word graph.

13. The apparatus according to claim 12, wherein, The creation of multiple edges between the plurality of words includes, for every two words in the plurality of words: The index lists of the two words corresponding to the two words are determined to contain a common index; In response to determining that the two word index lists contain a common index, it is determined that the two words are adjacent in the text segment corresponding to the common index; as well as In response to determining that the two words are adjacent in the text segment corresponding to the common index, an edge is created between the two words.

14. The apparatus according to claim 11, wherein, The generation of word suggestions for the current word includes: Determine a list of fragment indices for the text segments in the notes that contain the current word; Based on the word graph, determine a group of words that have an edge with the current word; Identify candidate words in the set of words whose word index list and fragment index list contain a common index; and Based on the identified candidate words, the word suggestions are generated.

15. The apparatus according to claim 11, wherein, When executed, the computer-executable instructions also cause the processor to: Detecting the selection of candidate words from the word suggestions; and In response to detecting the selection of a candidate word from the word suggestions, the selected candidate word is automatically entered into the notes.

16. The apparatus according to claim 11, wherein, When executed, the computer-executable instructions also cause the processor to: Detect characters input into the notes; By comparing strings, words that match the given characters among the plurality of words are identified; and Based on the identified words, word suggestions are generated for the characters.

17. The apparatus according to claim 11, wherein, When executed, the computer-executable instructions also cause the processor to: Using a large language model, annotation suggestions are generated for the text fragments in the notes based on the set of text fragments, and the annotation suggestions include annotations for the text fragments.

18. The apparatus according to claim 17, wherein, When executed, the computer-executable instructions also cause the processor to: Detecting the selection of the suggested annotations; and In response to detecting a selection of the suggested annotation, the annotation is appended to the text segment.

19. The apparatus according to claim 11, wherein, When executed, the computer-executable instructions also cause the processor to: Receive search queries for the notes; From the set of annotations in the notes, search for annotations that match the search query; and Extract the text segments from the notes that correspond to the annotations.

20. A computer program product for note-taking, comprising a computer program that is executed by a processor for: Obtain a set of text fragments in the target window and a set of positions corresponding to the set of text fragments; Based on the set of text fragments and the set of positions, a word graph corresponding to the target window is constructed. The word graph contains multiple words from the set of text fragments and multiple edges connecting the multiple words. Detect the current word entered into the note; as well as Based on the word graph, word suggestions are generated for the current word, and the word suggestions include words located after the current word in the target window.

Images