Note taking with contextual information

Abstract

The present disclosure proposes a method, apparatus and computer program products for note taking. A set of text segments in a target window and a set of positions corresponding to the set of text segments may be obtained. A word graph corresponding to the target window may be constructed according to the set of text segments and the set of positions, the word graph containing a plurality of words in the set of text segments and a plurality of edges connecting the plurality of words. A current word currently input into a note may be detected. A word suggestion for the current word may be generated according to the word graph, the word suggestion containing a word located after the current word in the target window.

Description

NOTE TAKING WITH CONTEXTUAL INFORMATIONBACKGROUND

[0001] Note taking applications such as OneNote provided by Microsoft Corporation provide a convenient way to create digital notes. Users can utilize the note taking applications to create digital notes containing text, table, picture, video, etc., to record ideas or manage tasks. The created digital notes may be shared among a plurality of users for collaboration or communication.SUMMARY

[0002] This Summary is provided to introduce a selection of concepts that are further described below in the Detailed Description. It is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] Embodiments of the present disclosure propose a method, apparatus and computer program products for note taking. A set of text segments in a target window and a set of positions corresponding to the set of text segments may be obtained. A w ord graph corresponding to the target window may be constructed according to the set of text segments and the set of positions, the word graph containing a plurality of words in the set of text segments and a plurality of edges connecting the plurality of words. A current word currently input into a note may be detected. A word suggestion for the cunent w ord may be generated according to the word graph, the word suggestion containing a w ord located after the current w ord in the target window .

[0004] It should be noted that the above one or more aspects comprise the features hereinafter fully described and particularly pointed out in the claims. The following description and the drawings set forth in detail certain illustrative features of the one or more aspects. These features are only indicative of the various ways in which the principles of various aspects may be employed, and this disclosure is intended to include all such aspects and their equivalents.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The disclosed aspects will hereinafter be described in connection with the appended drawings that are provided to illustrate and not to limit the disclosed aspects.

[0006] FIG. 1 illustrates an exemplary' process for note taking according to an embodiment of the present disclosure.

[0007] FIG. 2 illustrates an exemplary target window according to an embodiment of the present disclosure.

[0008] FIG. 3 illustrates an example of generating a w ord suggestion and a text segment suggestion for a character according to an embodiment of the present disclosure.

[0009] FIG. 4 illustrates an exemplary’ process for constructing a word graph correspondingto a target window according to an embodiment of the present disclosure.

[0010] FIG. 5 illustrates an example of assigning indexes to text segments according to an embodiment of the present disclosure.

[0011] FIG. 6 illustrates an exemplary word graph according to an embodiment of the present disclosure.

[0012] FIG. 7 illustrates an exemplary process for generating a word suggestion for a current word according to an embodiment of the present disclosure.

[0013] FIG. 8 illustrates an example of generating a word suggestion for a current word according to an embodiment of the present disclosure.

[0014] FIG. 9 illustrates an exemplar}' process for generating an annotation suggestion for a text segment in a note according to an embodiment of the present disclosure.

[0015] FIG. 10 illustrates an example of generating an annotation suggestion for a text segment in a note according to an embodiment of the present disclosure.

[0016] FIG. 11 illustrates an exemplar}' process for searching for a text segment in a note according to an embodiment of the present disclosure.

[0017] FIG. 12 illustrates an example of searching for a text segment in a note according to an embodiment of the present disclosure.

[0018] FIG. 13 is a flowchart of an exemplary method for note taking according to an embodiment of the present disclosure.

[0019] FIG. 14 illustrates an exemplary apparatus for note taking according to an embodiment of the present disclosure.

[0020] FIG. 15 illustrates another exemplary apparatus for note taking according to an embodiment of the present disclosure.DETAILED DESCRIPTION

[0021] The present disclosure will now be discussed with reference to several example implementations. It is to be understood that these implementations are discussed only for enabling those skilled in the art to better understand and thus implement the embodiments of the present disclosure, rather than suggesting any limitations on the scope of the present disclosure.

[0022] Users may create digital notes while surfing the Internet or participating in video conferences using computing devices. For the purpose of simplicity, the present disclosure will refer to a digital note as a note. Content of a note may come from information on a screen of a computing device, e.g., a web page in a browser, a presentation presented in a conference application, etc. Existing note taking applications may not be able to perceive the information on the screen, therefore the user needs to manually input the content on the screen into the note. This approach is inefficient and reduces the user's concentration when browsing the web page orparticipating in the video conference.

[0023] Embodiment of the present disclosure propose note taking with contextual information. Contextual information may be information from a screen or a target window of a computing device when a note taking application is enabled on the computing device. Herein, a target window refers to a current window of a target application whose content is to be recorded. The target application may be an application independent of the note taking application or an application integrated with the note taking application. As an example, the target application may be a browser application, a video conferencing application, a document creation application, etc. A set of text segments in the target window and a set of positions corresponding to the set of text segments may be obtained through techniques such as Optical Character Recognition (OCR) and User Interface (UI) automation. Herein, a text segment refers to a set of texts that are physically close to each other. A line of text, a paragraph of text, etc., may be referred to as a text segment. A word graph corresponding to the target window may be constructed based on the set of text segments and the set of positions. The word graph may include a plurality of words in the set of text segments and a plurality of edges connecting the plurality of words. When a current word currently input into a note is detected, a word suggestion for the current word may be generated based on the word graph. The word suggestion may contain a word located after the current word in the target window. A word contained in a word suggestion may be referred to as a candidate word. A user may select a candidate word in the word suggestion through clicking a predefined keyboard shortcut and / or mouse button. When a selection of a candidate word in the word suggestion is detected, the selected candidate word may be automatically input into the note. Additionally, a text segment suggestion for the current word may be generated. The text segment suggestion may contain a text segment containing the current word in the target window. The user may select a text segment in the text segment suggestion through clicking a keyboard shortcut key and / or a mouse button that is different from that for selecting the candidate word. Accordingly, the selected text segment may be automatically input into the note. The technical effect of the above approach is to improve the speed of note taking and enhance the user's concentration in interacting with the target application.

[0024] When it is detected that several words or a text segment is input into the note, an annotation suggestion for the text segment in the note may be generated based on a set of text segments in the target window through a Large Language Model (LLM). The annotation suggestion may contain an annotation for the text segment. Herein, a language model refers to a deep learning model that can understand meaning of natural language, generate natural language texts, or perform other natural language tasks. It should be appreciated that the large language models include multi-modal models that can perform processing tasks for natural language as wellas other modalities. The large language model may generate the annotation suggestion for the text segment in the note using its knowledge base, logical reasoning capability, etc., based on its understanding of content in the target window and content in the note. The user may select the annotation suggestion through clicking a predefined keyboard shortcut and / or mouse button. When a selection of the annotation suggestion is detected, the annotation contained in the annotation suggestion may be appended to the text segment. The content in the note may be fragmented and semantically incoherent. The technical effects of annotating the note lie in prompting the user to recall the meaning of the recorded content when reading the note later, and facilitating the search in the note in future. The technical effect of annotating the note through a large language model he in improving the efficiency and accuracy of annotating the note.

[0025] The annotation generated by the above approach may be stored together with the text segment in the note corresponding to the annotation in the form of a key-value pair, wherein the annotation may be stored as a "key" and the text segment may be stored as a "value" corresponding to the key. When a search query for the note is received, an annotation that matches the search query may be searched from the annotation set of the note through, e.g., vector similarity matching, a large language model, etc. Subsequently, a text segment corresponding to the annotation in the note may be extracted as a search result for the search query. The technical effect of the above approach is that accurate and fast search for a text segment in the note can be achieved even without an accurate search query that matches the text segment.

[0026] Various embodiments of the present disclosure will hereinafter be described in detail in connection with the appended drawings.

[0027] FIG. 1 illustrates an exemplary process 100 for note taking according to an embodiment of the present disclosure. The process 100 may be performed by a note taking application according to an embodiment of the present disclosure.

[0028] At 102, a set of text segments in a target window and a set of positions corresponding to the set of text segments may be obtained. This step may be performed when the note taking application is enabled, or when content in the target window changes. The target window may be a current window of a target application whose content is to be recorded. The target application may be an application independent of the note taking application or an application integrated with the note taking application. As an example, the target application may be a browser application, a video conferencing application, a document creation application, etc. A set of text segments in the target window and a set of positions corresponding to the set of text segments may be obtained through techniques such as OCR and UI automation. FIG. 2 illustrates an exemplary target window 200 according to an embodiment of the present disclosure. In the target window 200, a text segment 202 to a text segment 210 are shown. The position of each text segment may bedefined by a coordinate of an upper left vertex of a bounding box defining the text segment. For example, the position of the text segment 202 may be defined by a coordinate of a vertex 212.

[0029] Referring back to FIG. 1, at 104, a word graph corresponding to the target window may be constructed based on the set of text segments and the set of positions. The word graph may contain a plurality of words in the set of text segments and a plurality of edges connecting the plurality of words. An exemplary process for constructing the word graph will be described later in conjunction with FIG. 4.

[0030] At 106, a character input into the note may be detected.

[0031] At 108. a word suggestion and / or a text segment suggestion for the character may be generated through string comparison. Preferably, this step may be performed when the character is the first character of the current text segment or belongs to the first predetermined number of characters of the current text segment, that is, this step may be performed when a text segment is started to be input into the note. A word that matches the character in the plurality of words in the target window may be identified through string comparison. A word suggestion for the character may be generated based on the identified word. Alternatively or additionally, a text segment that matches the character in the set of text segments in the target window^ may be identified through string comparison. A text segment suggestion for the character may be generated based on the identified text segment. The word suggestion and the text segment suggestion may be presented in a pop-up window. Preferably, the text segment suggestion may be generated when it is detected that a predetermined number of characters are input into the current text segment. The predetermined number is, e.g., "2". That is, the text segment suggestion is presented only when the current text segment contains two or more characters. The technical effect of this approach lies in improving the accuracy of the text segment suggestion. There may be a plurality of text segments that match the character in the target window. Preferably, the text segment suggestion may only contain a predetermined number of text segments. The predetermined number may be " 1". The text segment included in the text segment suggestion may be a text segment with the highest user attention probability, such as a text segment closest to the upper left comer of the target window. The technical effects of this approach lie in reducing the number of text segments provided to the user, thereby reducing the complexity of user interaction.

[0032] FIG. 3 illustrates an example 300 of generating a word suggestion and a text segment suggestion for a character according to an embodiment of the present disclosure. In the example 300, a note taking application 302 is enabled and a note 304 is created. A character 306 "g" and a character 308 "i" input into note 304 are detected. A word suggestion 310 for the character 306 "g" and the character 308 "i" is generated, which contains a candidate word "gif'. A user may select the candidate word in the word suggestion 310 through clicking a predefined keyboardshortcut and / or mouse button. The selected candidate word may be automatically input into note 304. It should be appreciated that when there are a plurality of words in the target window that match the character 306 "g" and the character 308 "i", the word suggestion 310 may contain the plurality of words accordingly. In addition, the word suggestion 310 may be generated when only the character 306 "g" is detected and the character 308 "i" is not detected.

[0033] In addition, a text segment suggestion 312 for the character 306 "g" and the character 308 "i" is generated, which contains a text segment 202 "git add text. text". A text segment 204 and a text segment 206 also match the character 306 "g" and the character 308 "i", but the text segment suggestion 312 only contains the text segment 202, because the text segment 202 is the text segment closest to the upper left comer of the target window, which is most likely the text segment that the user is currently watching. It should be appreciated that it is also possible to contain a plurality of text segments in the text segment suggestion. The user may select a text segment in the text segment suggestion 312 through clicking a keyboard shortcut key and / or a mouse button that is different from that for selecting the candidate word. Accordingly, the selected text segment may be automatically input into the note 304.

[0034] Referring back to FIG. 1, at 110, a current word currently input into the note may be detected. The current word may be input by the user through typing letter keys, number keys, etc., on a keyboard, or through selecting a candidate word in a word suggestion.

[0035] In 112, a word suggestion for the cunent word may be generated based on the word graph. The w ord suggestion may contain a word located after the current w ord in the target window'. An exemplary' process for generating the word suggestion for the current word will be described later in conjunction with FIG. 7. In addition, a text segment suggestion for the current word may be generated through e.g., string comparison. The text segment suggestion may contain a text segment containing the current word in the target w indow. The w ord suggestion and / or the text segment suggestion may be presented in a pop-up w indow7.

[0036] At 114, a selection of a candidate word in the word suggestion may be detected. A user may select a word in the word suggestion through clicking a predefined keyboard shortcut and / or mouse button.

[0037] At 116, in response to a selection of a candidate w ord in the word suggestion is detected, the selected candidate word may be automatically input into the note. The text segment in the note may be updated accordingly.

[0038] The step 110 to the step 116 may be iteratively performed until the input of the text segment is completed.

[0039] In process 100, the word suggestion and the text segment suggestion for the character or word currently input into the note are generated with the contextual information from the targetwindow. The user may perform note taking through selecting a word or a text segment from the word suggestion and the text segment suggestion. The technical effects of the above approach lie in improving the speed of note taking and enhancing the user's concentration in interacting with the target application.

[0040] It should be appreciated that the process for note taking described above in conjunction with FIG. 1 to FIG. 3 is merely exemplary. Depending on actual application requirements, the steps in the process for note taking may be replaced or modified in any manner, and the process may comprise more or few er steps. In addition, the specific order or hierarchy of the steps in the process 100 is merely exemplary, and the process for note taking may be performed in an order different from the described one. For example, in the process 100, the step 106 is described after the step 104, but the step 104 and the step 106 may be performed simultaneously, or the step 104 may be performed after the step 106.

[0041] FIG. 4 illustrates an exemplary’ process 400 for constructing a word graph corresponding to a target window according to an embodiment of the present disclosure. The process 400 may correspond to the step 104 in FIG. 1.

[0042] At 402, a set of text segments in the target w indow may be sorted according to a set of positions corresponding to the set of text segments. As previously described, the position of each text segment may be defined by a coordinate of an upper left vertex of a bounding box defining the text segment. A text segment whose position is closer to the upper left comer of the target window may be sorted higher.

[0043] At 404, a set of indexes may be assigned to the set of text segments according to the sorting. A text segment that is sorted higher may be assigned a smaller index and accordingly have a higher priority. FIG. 5 illustrates an example 500 of assigning indexes to text segments according to an embodiment of the present disclosure. A text segment 202 to a text segment 210 are assigned with an index "0" to an index "4", respectively.

[0044] At 406, the set of text segments may be split into a plurality of words. The language of the set of text segments may be first detected, and the set of text segments may be split into the plurality of words based on a tokenization technique corresponding to the language. Taking the text segment 202 to the text segment 210 in FIG. 5 as an example, the set of text segments may be divided into a plurality of words as follows: "gif', "add", "text.text", "merge", "master", "readme.md", "yam", and "@reduxjs / toolkit".

[0045] At 408, for each word in the plurality of w ords, a word index list of the word may be generated based on one or more indexes of one or more text segments where the word is located. Herein, a word index list refers to an index list corresponding to a single word. One or more indexes of one or more text segments where a word is located may be combined into a word indexlist of the word. The word index list may indicate the one or more text segments where the word is located. The indexes in the word index list may be arranged in ascending order. Continuing with the example of the text segment 202 to the text segment 210 in FIG. 5, the word index list of the word "git" may be [0,1,2], the word index list of the word "add" may be [0.2.3.4], etc.

[0046] At 410. a plurality of edges among the plurality of words may be created according to a plurality of word index lists corresponding to the plurality of words. For example, for every two words in the plurality of w ords, it may be determined whether two word index lists corresponding to the two words contain a common index. If the two word index lists corresponding to the two words contain a common index, it indicates that the two words are located in a same text segment. In this case, it may be determined w ether the two words are adjacent in a text segment corresponding to the common index. If it is determined that the two w ords are adjacent in the text segment corresponding to the common index, i.e., there are no other words between the two words, an edge may be created between the two words. The direction of the edge may be from the previous word to the next word.

[0047] At 412, the plurality of w ords and the plurality7of edges may be combined into a word graph. Since each edge in the word graph is directional, the word graph may be referred to as a directed word graph.

[0048] FIG. 6 illustrates an exemplary word graph 600 according to an embodiment of the present disclosure. The word graph 600 may be a word graph corresponding to the target window shown in FIG. 5. The word graph 600 includes a plurality of words as follows: "git", "add", "text.text", "merge", "master", "readme.md", "yam", and "@reduxjs / toolkit". A word index list of each word is shown below the word. The word index list may indicate one or more text segments where the word is located. When two word index lists corresponding to two words contain a common index and the twn words are adjacent, an edge may be created between the twn words. The direction of the edge may be from the previous word to the next w ord. For example, the w ord index list of the word "git" is [0,1,2], the word index list of the word "add" is [0,2, 3, 4], and the word index lists of the two words contain common indexes, i.e., "0" and "2". This indicates that the word "git" and the word "add" are both located in the text segment with index "0" and the text segment with index "2". It may be further determined w hether there are other w ords betw een the word "git" and the word "add" in the text segment with index "0" and / or the text segment with index "2". The text segment with index "0" is "git add text.text". The text segment with index "2" is "git add readme.md". In these two text segments, the word "git" is adjacent to the word "add". Therefore, an edge may be created between the word "git" and the word "add". The direction of the edge is from the word "git" to the word "add". Edges among other words may be created in the same way as described above. It should be appreciated that the word graph 600 shown in FIG.6 is merely an example of the word graph. Depending on actual application requirements, the word graph may have any other structure and may include more or fewer elements.

[0049] The technical effect of process 400 is that a word graph that is consistent with the relative position of texts in a target window can be constructed. Using this word graph, a word suggestion for a word in the note may be generated quickly and accurately, thereby improving the speed of note taking.

[0050] It should be appreciated that the process for constructing the word graph corresponding to the target window described above in conjunction with FIG. 4 to FIG. 6 is merely exemplary. Depending on actual application requirements, the steps in the process for constructing the word graph may be replaced or modified in any manner, and the process may comprise more or fewer steps. In addition, the specific order or hierarchy of the steps in the process 400 is merely exemplary, and the process for constructing the word graph according to the target window may be performed in an order different from the described one.

[0051] FIG. 7 illustrates an exemplary process 700 for generating a word suggestion for a current word according to an embodiment of the present disclosure. The process 700 may correspond to the step 112 in FIG. 1.

[0052] At 702, a segment index list of a text segment containing a current word in a note may be determined. Herein, a segment index list refers to an index list corresponding to a text segment containing one or more words. A segment index list of a text segment may include a common index of a set of word index lists corresponding to a set of words in the text segment. For example, assuming that the current word is "add", a text segment containing the current word in the note is "git add", that is, the text segment contains the word "git" and the word "add". The word index list of the word "git" is [0, 1,2], and the word index list of the word "add" is [0,2,3,4], The common indexes of these two word index lists are "0" and "2". Therefore, the segment index list of the text segment "git add" is [0,2], The segment index list may be updated each time a word is input into the text segment.

[0053] At 704. a set of words that have edges with the current word may be determined according to a word graph. Preferably, the determined words should be words pointed to by the edges, that is, words located after the current word in the target window.

[0054] At 706, a candidate word in the set of words whose word index list and the segment index list contain a common index may be identified. The technical effects of the step 704 and the step 706 he in causing the word suggestion to only contain words located after the current word in the target window, and meanwhile ensuring that a text segment formed by the current text segment in the note and the candidate word is consistent with the text segment in the target window, thereby improving the accuracy of the word suggestion. This will be explained later inconjunction with an example in FIG. 8.

[0055] At 708, a word suggestion for the current word may be generated based on the identified candidate word. Preferably, when there are a plurality of candidate words, the plurality of candidate words may be sorted according to the first index in a word index list of the candidate word. The smaller the first index in the word index list, the higher it is sorted. The smaller the index of the word, the closer the position of the word in the target window is to the upper left comer of the screen, and it is more likely to be the word currently watched by the user. The technical effects of this approach lie in causing a word that is more likely to be watched by the user to appear at the top of the word suggestion, which is convenient for the user to see and select, thereby further improving the speed of note taking.

[0056] FIG. 8 illustrates an example of generating a word suggestion for a current word according to an embodiment of the present disclosure.

[0057] In a diagram 800a, a current word 804 "git" currently input into a note 802 is detected. A text segment only includes the current word 804 "git". Accordingly, a segment index list of the text segment is consistent with a word index list of the current word 804 "git", that is, [0,1,2], A set of words that have edges with the current word 804 "git" may be determined according to a word graph. For example, referring to the word graph 600 in FIG. 6, a set of words that have edges with the current word 804 "git" are the word "add" and the word "merge". Subsequently, a candidate word in the word "add" and the word "merge" whose word index list and the segment index list contain a common index may be identified. The word index list of the word "add" is [0,2, 3, 4], which contains common indexes with the segment index list [0,1,2], i.e.. the index "0" and the index "2". The word "add" may be used as a candidate word. The word index list of the word "merge" is [1], which contains a common index with the segment index list [0,1,2], i.e., the index "1". The word "merge" may be used as a candidate word. Therefore, the generated word suggestion 806 includes two candidate words, i.e., the word "add" and the word "merge". It should be appreciated that in the diagram 800a, the candidate words in the word suggestion 806 are arranged vertically, but other arrangements, such as horizontal arrangement, are also possible.

[0058] In the diagram 800a, a text segment suggestion 808 for the current word 804 is also shown. A text segment in a set of text segments in the target window that matches the text segment may be identified through string comparison. Subsequently, a text segment suggestion for the current word may be generated based on the identified text segment. For example, text segments in the text segment 202 to the text segment 210 that match the text segment "git" may include the text segment 202 "git add text. text", the text segment 204 "git merge master" and the text segment 206 "git add readme.md". However, the text segment suggestion 808 only contains the text segment 202 "git add text. text". The text segment 202 "git add text. text" is the text segment closestto the upper left comer of the target window, which may be the text segment with the highest user attention probability. The technical effects of containing only one or a small number of text segments in the text segment suggestion 808 lie in reducing the number of text segments provided to the user, thereby reducing the complexity of user interaction.

[0059] Assuming that the user selects the candidate word "add" in the word suggestion 806, it may be detected that a current word 810 currently input into the note 802 is "add". At this point, the text segment is updated to "git add". Accordingly, the segment index list of the text segment is updated to [0.2], A set of words that have edges with the current word 810 "add" may be determined according to the word graph. For example, referring to the word graph 600 in FIG. 6, a set of words that have edges with the current word 810 "add" are the word ‘'text. text’’, the word “master”, the word “readme.md”, and the word “@reduxjs / toolkit”. Subsequently, a candidate word in the word "text.text", the word "master", the word "readme.md", and the word "@reduxjs / toolkit" whose word index list and the segment index list contain a common index may be identified. The word index list of the word "text.text" is [0], which contains a common index with the segment index list [0, 2], i.e., the index "0". The word "text.text" may be used as a candidate word. The word index list of the word "master" is [1.3], which does not contain a common index with the segment index list [0,2], The word "master" cannot be used as a candidate word. The word index list of the word "readme.md" is [2], which contains a common index with the segment index list [0,2], i.e., the index "2". The word "readme.md" may be used as a candidate word. The word index list of the word "@reduxjs / toolkit" is [4], which does not contain a common index with the segment index list [0,2], The word "@reduxjs / toolkit" cannot be used as a candidate word. Therefore, the generated word suggestion 812 includes two candidate words, i.e., the word "text.text" and the word "readme.md", as shown in the diagram 800b. In addition, a text segment suggestion 814 for the current word 810 “add” still includes the text segment 202 “git add text.text”.

[0060] As can be seen from the diagram 800b, although the set of words identified according to the word graph that have edges with the current word 810 "add" include the word "master" and the word "@reduxjs / toolkit", the word "master" and the word "@reduxjs / toolkit" are not considered as candidate words for generating the word suggestion since the word index lists of the word "master" and the word "@reduxjs / toolkit" do not contain common indexes with the segment index list of the text segment "git add" in the note. This is consistent with the content in the target window, because the target window does not contain a text segment "git add master" and a text segment "git add @reduxjs / toolkit".

[0061] In the process described above, the set of words that have edges with the current word is first determined according to the word graph, and then a candidate word in the set of wordswhose word index list and the segment index list contain a common index is identified, and next, the word suggestion is generated based on the identified candidate word. The technical effects of the approach for generating the word suggestion proposed in the embodiment of the present disclosure lie in causing the word suggestion to only contain words located after the current word in the target window, and meanwhile ensuring that a text segment formed by the current text segment in the note and the candidate word is consistent with the text segment in the target window , thereby improving the accuracy of the word suggestion.

[0062] It should be appreciated that the process for generating the word suggestion for the current word described above in conjunction with FIG. 7 to FIG. 8 is merely exemplary. Depending on actual application requirements, the steps in the process for generating the word suggestion for the current word may be replaced or modified in any manner, and the process may comprise more or fewer steps. In addition, the specific order or hierarchy of the steps in the process 700 is merely exemplary, and the process for generating the word suggestion for the current word may be performed in an order different from the described one.

[0063] FIG. 9 illustrates an exemplary process 900 for generating an annotation suggestion for a text segment in a note according to an embodiment of the present disclosure. In the process 900, an annotation suggestion 922 for a text segment 904 in a note may be generated based on a set of text segments 902 in the target window through a large language model 920. The annotation suggestion 922 may contain an annotation 924 for the text segment 904 generated by the large language model 920. The large language model 920 is, e.g., a Generative Pre-trained Transformer (GPT) model.

[0064] A prompt 912 to be provided to the large language model 920 may be created through a prompt creator 910. The prompt creator 910 may create the prompt 912 based on a set of text segments 902 in the target window^ and the text segment 904 in the note. Preferably, a prompt template 906 for the prompt creator 910 may be designed in advance. The prompt template 906 may include a plurality of variable parts for loading the set of text segments 902 in the target window and the text segment 904 in the note, respectively. In addition, the prompt template 906 may include response instructions for instructing the large language model 920 on how to respond and several examples that can be learned.

[0065] The prompt 912 may be provided to the large language model 920. The large language model 920 may generate the annotation suggestion 922 for the text segment 904 in the note using its knowledge base and logical reasoning capability, etc., based on its understanding of the set of text segments 902 in the target window and the text segment 904 in the note. The annotation suggestion 922 contains the annotation 924 for the text segment 904 generated by the large language model 920.

[0066] FIG. 10 illustrates an example of generating an annotation suggestion for a text segment in a note according to an embodiment of the present disclosure.

[0067] In a diagram 1000a, a text segment 1004 in a note 1002 is "yam add @reduxj s / toolkit" . An annotation suggestion 1006 for the text segment 1004 in the note may be generated based on the text segment 202 to the text segment 210 in the target window through a large language model. The annotation suggestion 1006 contains an annotation 1008 for the text segment 1004, i.e., "Command to install Redux toolkit", which is generated by the large language model using its knowledge base, logical reasoning capability', etc., based on its understanding of the text segment 202 to the text segment 210 in the target window and the text segment 1004 in the note.

[0068] A user may select the annotation suggestion 1006 through clicking a predefined keyboard shortcut and / or mouse button. If a selection of the annotation suggestion 1006 is detected, the annotation 1008 contained in the annotation suggestion 1006 may be appended to the text segment 1004, as shown in FIG. 1000b. Preferably, the text segment 1004 may be highlighted to remind the user that the text segment is appended with an annotation. When the cursor or other indicator hovers over the text segment 1004, the annotation 1008 may be automatically displayed.

[0069] The content in the note may be fragmented and semantically incoherent. The technical effects of annotating the note he in prompting the user to recall the meaning of the recorded content when reading the note later, and facilitating the search in the note in future. The technical effect of annotating the note through a large language model lies in improving the efficiency and accuracy of annotating the note.

[0070] It should be appreciated that the process for generating the annotation suggestion for the text segment in the note described above in conjunction with FIG. 9 to FIG. 10 is merely exemplary. Depending on actual application requirements, the steps in the process for generating the annotation suggestion may be replaced or modified in any manner, and the process may comprise more or feyver steps.

[0071] The annotation 924 generated through the process 900 may be stored together with the text segment 904 in the note corresponding to the annotation 924 in the form of a key -value pair {key: value}, yvherein the annotation 924 may be stored as a "key" and the text segment 904 may be stored as a "value" corresponding to the key. The technical effect of this approach lies in facilitating the search for a text segment in the note in future.

[0072] FIG. 11 illustrates an exemplary process 1100 for searching for a text segment in a note according to an embodiment of the present disclosure. In the process 1100, a search query 1104 for a note 1102 may be received. An annotation 1152 that matches the search query 1104 may be searched from an annotation set 1106 of the note 1102. The annotation set 1106 may begenerated through the process 900 in FIG. 9. Subsequently, a text segment 1154 corresponding to the annotation 1152 in the note 1102 may be extracted.

[0073] The search query 1104 may be provided in a variety7of ways. In an implementation, the search query71104 may be provided through clicking a search button in a note taking application and inputting the search query 1 104 into a search box that is subsequently displayed. In another implementation, a predefined search character may be directly input into the note, and the search query71104 may be input after the predefined search character. As an example, the predefined search character may be a slash symbol

[0074] A query vector 1112 of the search query71104 may be generated through an encoder 1 110. The encoder 1 110 may be a machine learning model capable of generating a vector or an embedding based on a text input. As an example, the encoder 1110 may be a Bidirectional Encoder Representations from Transformers (BERT) model. Similarly, an annotation vector set 1122 corresponding to the annotation set 1106 may be generated through an encoder 1120. The encoder 1120 may have a model structure and parameters consistent with the encoder 1 110.

[0075] An annotation subset 1 132 similar to the search query 1104 may be searched from the annotation set 1106 according to the query7vector 1112 and the annotation vector set 1122. For example, an annotation subset 1132 may be searched through performing semantic similarity7matching on the query vector 1112 and the annotation vector set 1122 by a similarity matching module 1130. The similarity7matching module 1130 may employ a technology such as Nearest Neighbor Search to search several annotation vectors that are closest to the query vector 1112 from the annotation vector set 1122, thereby determining several annotations that are most semantically similar to the search query 1104. These several annotations may be combined into the annotation subset 1 132.

[0076] A prompt 1142 to be provided to a large language model 1150 may be created through a prompt creator 1140. The large language model 1150 is, e.g., a GPT model. The prompt creator 1140 may create the prompt 1142 based on the search query 1104 and the annotation subset 1132. Preferably, a prompt template 1134 for the prompt creator 1 140 may be designed in advance. The prompt template 1134 may include a plurality of variable parts for loading the search query 1104 and the annotation subset 1132, respectively. In addition, the prompt template 1134 may include a response instruction for instructing the large language model 1150 on how to respond.

[0077] The prompt 1142 may be provided to the large language model 1150. The large language model 1150 may identify an annotation 1152 in the annotation subset 1132 that matches the search query71104 using its knowledge base and semantic understanding ability, logical reasoning ability, etc. Subsequently, a text segment 1154 in the note 1102 corresponding to the annotation 1152 may be extracted as a search result for the search query 1104. As described above,the annotation and the text segment in the note corresponding to the annotation are stored together in the form of a key-value pair. Therefore, the text segment 1154 in the note 1102 corresponding to the annotation 1152 may be quickly extracted. The text segment 1154 may be presented in a variety of ways. In an implementation, the text segment 1154 may be presented in a pop-up window of the note taking application. In another implementation, the text segment 1154 may be presented directly in the note 1102.

[0078] FIG. 12 illustrates an example of searching for a text segment in a note according to an embodiment of the present disclosure. In a diagram 1200a, a predefined search characterand a search query 1204 "Redux" are input in a note 1202. Subsequently, in the note 1202, a text segment 1206 that matches the search query 1204 "Redux", i.e., "yam add @reduxjs / toolkit", is provided, as shown in a diagram 1200b.

[0079] In the process 1100, when a search query' for a note is received, an annotation that matches the search query may be searched from an annotation set of the note through, e.g., vector similarity matching, a large language model, etc. Subsequently, a text segment in the note corresponding to the annotation may be extracted as a search result for the search query. The technical effect of the above approach is that accurate and fast search for a text segment in the note can be achieved even without an accurate search query that matches the text segment.

[0080] It should be appreciated that the process for searching for the text segment in the note described above in conjunction with FIG. 11 to FIG. 12 is merely exemplary. Depending on actual application requirements, the steps in the process for searching for the text segment in the note may be replaced or modified in any manner, and the process may comprise more or fewer steps. For example, in the process 1100, the annotation subset 1132 similar to the search query 1104 is first searched from the annotation set 1106 through the encoders 1 110 and 1 120 and the similarity matching module 1130, and then the annotation 1152 that matches the search query 1104 is identified from the annotation subset 1132 through the large language model 1150, but the embodiments of the present disclosure are not limited thereto. In some embodiments, the annotation that matches the search query 1104 may be directly identified from the annotation set 1 106 through the large language model 1150, without the processes of the encoders 1110 and 1120 and the similarity matching module 1130. In addition, the specific order or hierarchy of the steps in the process 1100 is merely exemplary, and the process for searching for the text segment in the note may be performed in an order different from the described one.

[0081] FIG. 13 is a flowchart of an exemplary method 1300 for note taking according to an embodiment of the present disclosure.

[0082] At 1310, a set of text segments in a target window and a set of positions corresponding to the set of text segments may be obtained.

[0083] At 1320, a word graph corresponding to the target window may be constructed according to the set of text segments and the set of positions, the word graph containing a plurality of words in the set of text segments and a plurality of edges connecting the plurality of words.

[0084] At 1330, a current word currently input into a note may be detected.

[0085] At 1340, a word suggestion for the current word may be generated according to the word graph, the word suggestion containing a word located after the current word in the target window.

[0086] In an implementation, the constructing a word graph corresponding to the target window may comprise: sorting the set of text segments according to the set of positions; assigning a set of indexes to the set of text segments according to the sorting; splitting the set of text segments into the plurality of words; for each word in the plurality7of words, generating a word index list of the word based on one or more indexes of one or more text segments where the word is located; creating the plurality of edges among the plurality of words according to a plurality of word index lists corresponding to the plurality of words; and combining the plurality of words and the plurality of edges into the word graph.

[0087] The creating the plurality of edges among the plurality of words may comprise, for every two words in the plurality of words: determining that two word index lists corresponding to the two words contain a common index; in response to determining that the two word index lists contain a common index, determining that the two words are adjacent in a text segment corresponding to the common index; and in response to determining that the two words are adjacent in the text segment corresponding to the common index, creating an edge between the two words.

[0088] In an implementation, the generating a word suggestion for the current word may comprise: determining a segment index list of a text segment containing the current word in the note; determining a set of words that have edges with the current word according to the word graph; identifying a candidate word in the set of words whose word index list and the segment index list contain a common index; and generating the word suggestion based on the identified candidate word.

[0089] In an implementation, the method 1300 may further comprise: detecting a selection of a candidate word in the word suggestion; and in response to detecting the selection of a candidate word in the word suggestion, automatically inputting the selected candidate word into the note.

[0090] In an implementation, the method 1300 may further comprise: detecting a character input into the note; identify ing a word that matches the character in the plurality of words through string comparison; and generating a word suggestion for the character based on the identified word.

[0091] In an implementation, the method 1300 may further comprise: detecting a character input into the note; identifying a text segment that matches the character in the set of text segments through string comparison; and generating a text segment suggestion for the character based on the identified text segment.

[0092] In an implementation, the method 1300 may further comprise: generating an annotation suggestion for a text segment in the note based on the set of text segments through a large language model, the annotation suggestion containing an annotation for the text segment.

[0093] The method 1300 may further comprise: detecting a selection of the annotation suggestion; and in response to detecting the selection of the annotation suggestion, appending the annotation to the text segment.

[0094] In an implementation, the method 1300 may further comprise: receiving a search query for the note; searching for an annotation that matches the search query from an annotation set of the note; and extracting a text segment in the note corresponding to the annotation.

[0095] It should be appreciated that the method 1300 may further comprise any other steps / processes for note taking according to the embodiments of the present disclosure as mentioned above.

[0096] FIG. 14 illustrates an exemplary apparatus 1400 for note taking according to an embodiment of the present disclosure.

[0097] The apparatus 1400 may comprise: atext segment and position obtaining module 1410, for obtaining a set of text segments in a target window and a set of positions corresponding to the set of text segments; a word graph constructing module 1420, for constructing a word graph corresponding to the target window according to the set of text segments and the set of positions, the word graph containing a plurality of words in the set of text segments and a plurality of edges connecting the plurality of words; a current word detecting module 1430, for detecting a current word currently input into a note; and a word suggestion generating module 1440, for generating a word suggestion for the current word according to the word graph, the word suggestion containing a word located after the current word in the target window. Furthermore, the apparatus 1400 may further comprise any other modules configured for note taking according to the embodiments of the present disclosure as mentioned above.

[0098] FIG. 15 illustrates another exemplary apparatus 1500 for note taking according to an embodiment of the present disclosure.

[0099] The apparatus 1500 may comprise a processor 1510; and a memory 1520 storing computer-executable instructions. The computer-executable instructions, when executed, may cause the processor 1510 to: obtain a set of text segments in a target window' and a set of positions corresponding to the set of text segments; construct a word graph corresponding to the targetwindow according to the set of text segments and the set of positions, the word graph containing a plurality of words in the set of text segments and a plurality of edges connecting the plurality of words; detect a current word currently input into a note; and generate a word suggestion for the current word according to the word graph, the word suggestion containing a word located after the cunent word in the target window.

[0100] In an implementation, the constructing a word graph corresponding to the target window may comprise: sorting the set of text segments according to the set of positions; assigning a set of indexes to the set of text segments according to the sorting; splitting the set of text segments into the plurality of words; for each word in the plurality of words, generating a word index list of the word based on one or more indexes of one or more text segments where the word is located; creating the plurality of edges among the plurality of words according to a plurality of word index lists corresponding to the plurality of words; and combining the plurality of words and the plurality of edges into the word graph.

[0101] The creating the plurality of edges among the plurality of words may comprise, for every two words in the plurality of words: determining that two w ord index lists corresponding to the tw o w ords contain a common index; in response to determining that the two word index lists contain a common index, determining that the two words are adjacent in a text segment corresponding to the common index; and in response to determining that the two words are adjacent in the text segment corresponding to the common index, creating an edge between the tw o words.

[0102] In an implementation, the generating a word suggestion for the current word may comprise: determining a segment index list of a text segment containing the current word in the note; determining a set of words that have edges with the current word according to the word graph; identifying a candidate word in the set of w ords whose word index list and the segment index list contain a common index; and generating the w ord suggestion based on the identified candidate word.

[0103] In an implementation, the computer-executable instructions, when executed, may further cause the processor 1510 to: detect a selection of a candidate word in the word suggestion; and in response to detecting the selection of a candidate word in the word suggestion, automatically input the selected candidate word into the note.

[0104] In an implementation, the computer-executable instructions, when executed, may- further cause the processor 1510 to: detect a character input into the note; identify a w ord that matches the character in the plurality of words through string comparison; and generate a word suggestion for the character based on the identified word.

[0105] In an implementation, the computer-executable instructions, when executed, mayfurther cause the processor 1510 to: generate an annotation suggestion for a text segment in the note based on the set of text segments through a large language model, the annotation suggestion containing an annotation for the text segment.

[0106] The computer-executable instructions, when executed, may further cause the processor 1510 to: detect a selection of the annotation suggestion; and in response to detecting the selection of the annotation suggestion, append the annotation to the text segment.

[0107] In an implementation, the computer-executable instructions, when executed, may further cause the processor 1510 to: receive a search query for the note; search for an annotation that matches the search query from an annotation set of the note; and extract a text segment in the note corresponding to the annotation.

[0108] It should be appreciated that the processor 1510 may further perform any other steps / processes of the method for note taking according to the embodiments of the present disclosure as mentioned above.

[0109] The embodiments of the present disclosure propose a computer program product for note taking, comprising a computer program that is executed by a processor for: obtaining a set of text segments in a target window' and a set of positions corresponding to the set of text segments; constructing a word graph corresponding to the target window according to the set of text segments and the set of positions, the word graph containing a plurality of words in the set of text segments and a plurality of edges connecting the plurality of words; detecting a current word currently input into a note; and generating a word suggestion for the current w ord according to the word graph, the word suggestion containing a word located after the current word in the target window. Furthermore, the computer program may be further executed for implementing any other steps / processes of the method for note taking according to the embodiments of the present disclosure as mentioned above.

[0110] The embodiments of the present disclosure may be embodied in a computer-readable medium for note taking. The computer-readable medium may comprise instructions that, when executed, cause a processor to: obtain a set of text segments in a target window and a set of positions corresponding to the set of text segments; construct a word graph corresponding to the target window according to the set of text segments and the set of positions, the word graph containing a plurality of words in the set of text segments and a plurality of edges connecting the plurality of words; detect a current word currently input into a note; and generate a word suggestion for the current w ord according to the w ord graph, the word suggestion containing a word located after the current word in the target window. Furthermore, the instructions, when executed, may further cause the processor to perform any other steps / processes of the method for note taking according to the embodiments of the present disclosure as mentioned above.

[0111] It should be appreciated that all the operations in the methods described above are merely exemplary, and the present disclosure is not limited to any operations in the methods or sequence orders of these operations, and should cover all other equivalents under the same or similar concepts. In addition, the articles “a” and "an" as used in this specification and the appended claims should generally be construed to mean “one” or “one or more” unless specified otherwise or clear from the context to be directed to a singular form.

[0112] It should also be appreciated that all the modules in the apparatuses described above may be implemented in various approaches. These modules may be implemented as hardware, software, or a combination thereof. Moreover, any of these modules may be further functionally divided into sub-modules or combined together.

[0113] Processors have been described in connection with various apparatuses and methods. These processors may be implemented using electronic hardware, computer software, or any combination thereof. Whether such processors are implemented as hardware or software will depend upon the particular application and overall design constraints imposed on the system. By way of example, a processor, any portion of a processor, or any combination of processors presented in the present disclosure may be implemented with a microprocessor, microcontroller, digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic device (PLD). a state machine, gated logic, discrete hardware circuits, and other suitable processing components configured for performing the various functions described throughout the present disclosure. The functionality of a processor, any portion of a processor, or any combination of processors presented in the present disclosure may be implemented with software being executed by a microprocessor, microcontroller, DSP, or other suitable platform.

[0114] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, threads of execution, procedures, functions, etc. The software may reside on a computer-readable medium. A computer-readable medium may include, by way of example, memory such as a magnetic storage device (e.g., hard disk, floppy disk, magnetic strip), an optical disk, a smart card, a flash memory device, random access memory (RAM), read only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), a register, or a removable disk. Although memory is shown separate from the processors in the various aspects presented throughout the present disclosure, the memory may be internal to the processors, e.g., cache or register.

[0115] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readilyapparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents to the elements of the various aspects described throughout the present disclosure that are known or later come to be known to those of ordinary’ skilled in the art are expressly incorporated herein and intended to be encompassed by the claims.

Claims

CLAIMS1 . A method for note taking, comprising: obtaining a set of text segments in a target window and a set of positions corresponding to the set of text segments; constructing a word graph corresponding to the target window according to the set of text segments and the set of positions, the word graph containing a plurality of words in the set of text segments and a plurality of edges connecting the plurality of words; detecting a current worpd currently input into a note; and generating a word suggestion for the current word according to the word graph, the word suggestion containing a word located after the current word in the target window.

2. The method of claim 1, wherein the constructing a word graph corresponding to the target window comprises: sorting the set of text segments according to the set of positions; assigning a set of indexes to the set of text segments according to the sorting; splitting the set of text segments into the plurality of words; for each word in the plurality of words, generating a word index list of the word based on one or more indexes of one or more text segments where the word is located; creating the plurality of edges among the plurality of words according to a plurality of word index lists corresponding to the plurality of words; and combining the plurality of words and the plurality' of edges into the word graph.

3. The method of claim 2, wherein the creating the plurality of edges among the plurality of words comprises, for even’ two words in the plurality of words: determining that two word index lists corresponding to the two words contain a common index; in response to determining that the two word index lists contain a common index, determining that the two words are adjacent in a text segment corresponding to the common index; and in response to determining that the two words are adjacent in the text segment corresponding to the common index, creating an edge between the two words.

4. The method of claim 1, wherein the generating a word suggestion for the current word comprises: determining a segment index list of a text segment containing the current word in the note; determining a set of words that have edges with the current word according to the word graph; identifying a candidate word in the set of words whose word index list and the segmentindex list contain a common index; and generating the word suggestion based on the identified candidate word.

5. The method of claim 1, further comprising: detecting a selection of a candidate word in the word suggestion; and in response to detecting the selection of a candidate word in the word suggestion, automatically inputting the selected candidate word into the note.

6. The method of claim 1, further comprising: detecting a character input into the note; identifying a word that matches the character in the plurality of words through string comparison; and generating a word suggestion for the character based on the identified word.

7. The method of claim 1, further comprising: detecting a character input into the note; identifying a text segment that matches the character in the set of text segments through string comparison; and generating a text segment suggestion for the character based on the identified text segment.

8. The method of claim 1, further comprising: generating an annotation suggestion for a text segment in the note based on the set of text segments through a large language model, the annotation suggestion containing an annotation for the text segment.

9. The method of claim 8, further comprising: detecting a selection of the annotation suggestion; and in response to detecting the selection of the annotation suggestion appending the annotation to the text segment.

10. The method of claim 1, further comprising: receiving a search query for the note; searching for an annotation that matches the search query from an annotation set of the note; and extracting a text segment in the note corresponding to the annotation.

11. An apparatus for note taking, comprising: a processor; and a memory storing computer-executable instructions that, when executed, cause the processor to: obtain a set of text segments in a target window and a set of positions corresponding to the set of text segments.construct a word graph corresponding to the target window according to the set of text segments and the set of positions, the word graph containing a plurality of words in the set of text segments and a plurality' of edges connecting the plurality7of words, detect a current word currently input into a note, and generating a word suggestion for the current word according to the word graph, the word suggestion containing a word located after the current word in the target window.

12. The apparatus of claim 11, wherein the constructing a word graph corresponding to the target window comprises: sorting the set of text segments according to the set of positions; assigning a set of indexes to the set of text segments according to the sorting; splitting the set of text segments into the plurality7of words; for each word in the plurality7of words, generating a word index list of the word based on one or more indexes of one or more text segments where the word is located; creating the plurality of edges among the plurality of words according to a plurality of word index lists corresponding to the plurality7of words; and combining the plurality7of words and the plurality7of edges into the word graph.

13. The apparatus of claim 11, wherein the generating a word suggestion for the current word comprises: determining a segment index list of a text segment containing the current word in the note; determining a set of words that have edges yvith the current word according to the yvord graph; identifying a candidate word in the set of words whose word index list and the segment index list contain a common index; and generating the word suggestion based on the identified candidate word.

14. The apparatus of claim 11, yvherein the computer-executable instructions, yvhen executed, further cause the processor to: generate an annotation suggestion for a text segment in the note based on the set of text segments through a large language model, the annotation suggestion containing an annotation for the text segment.

15. A computer program product for note taking, comprising a computer program that is executed by a processor for: obtaining a set of text segments in a target window and a set of positions corresponding to the set of text segments; constructing a word graph corresponding to the target window according to the set of text segments and the set of positions, the yvord graph containing a plurality of yvords in the set of textsegments and a plurality of edges connecting the plurality of words; detecting a current word currently input into a note; and generating a word suggestion for the current word according to the word graph, the word suggestion containing a word located after the current word in the target window.

Images