Digital ink processing system, method, processing apparatus and computer program product

Abstract

The present invention provides a digital ink processing system, method, and program that make the search results valuable when searching using digital ink, and that are able to present information with high relevance to the user. A tablet (40) has a processor (46) that validates a pointing operation of a stylus (50) with respect to a stroke, after validating the pointing operation of the stylus (50), searches for associated content (C1, C2) associated with a meaning attribute of the pointed stroke or requests the search to an external server, and controls to display the associated content (C1, C2) obtained by the search together with the stroke on a display (42).

Description

Technical Field

[0001] This invention relates to a digital ink processing system, method, and procedure for processing digital ink. Background Technology

[0002] Patent document 1 discloses the following technology: performing a search by setting at least a portion of the information displayed on the screen of an information processing device including a personal computer as search keywords, and displaying the search results on the screen in the form of a pop-up box.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2015-114955 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] In digital inks, which describe a collection of strokes, meaning often arises from a series of strokes. However, in digital ink editing, when attempting a search by indicating a location equivalent to a search keyword, sometimes setting meaningless or incorrect search keywords can lead to unsuitable search results.

[0008] The purpose of this invention is to provide a digital ink processing system, method, and program that produces valuable search results and provides users with highly relevant information when performing searches using digital ink.

[0009] Technical solutions for solving the problem

[0010] The digital ink processing system of the first invention includes: an electronic pen; and a tablet computer that generates digital ink describing strokes based on input made by the electronic pen via a display. The tablet computer has a processor that enables the electronic pen to instruct the strokes. After enabling the electronic pen to instruct the strokes, the processor retrieves content associated with the meaning attributes of the instructed strokes or requests the retrieval from an external server. The processor controls the display to show the retrieved content along with the strokes on the display.

[0011] Alternatively, the processor may control the display by emphasizing the strokes on the screen after the instruction operation has been activated compared to before activation.

[0012] Alternatively, the processor may enable the electronic pen's instruction operation for strokes with the attached meaning attribute.

[0013] Alternatively, the system may also include a digital ink server that adds the meaning attribute to the stroke by parsing the digital ink sent from the tablet computer. The processor enables the electronic pen's instruction operation for strokes for which meaning data has been obtained from the digital ink server, and the meaning data represents the meaning attribute.

[0014] Alternatively, the processor may enable the electronic pen's instruction operation for strokes that have been marked or annotated by the user.

[0015] Alternatively, the system may also include a content server that stores content in correspondence with usage IDs. The processor retrieves content associated with and permitted for use by sending data including the usage ID and the meaning attribute to the content server and requesting retrieval.

[0016] Alternatively, the processor may obtain different content based on the degree of occurrence of the meaning attribute of the same usage ID.

[0017] Alternatively, the ID used may be a pen ID used to identify the electronic pen.

[0018] Alternatively, when digital ink is generated using a first electronic pen with a first ID attached, and a second electronic pen with a second ID different from the first ID is used, the processor may disable or restrict the editing of the digital ink and accept the instruction operation.

[0019] Alternatively, the content server may perform payment processing on the users and suppliers of the content in accordance with the usage status of the content.

[0020] The second digital ink processing method of the present invention includes a processor generating digital ink describing a stroke based on input made by an electronic pen via a display. The processor performs the following steps: activating the instruction operation of the electronic pen on the stroke; after activating the instruction operation of the electronic pen, retrieving content associated with the meaning attribute of the indicated stroke or requesting the retrieval from an external server; and controlling the display to show the retrieved content along with the stroke.

[0021] The third invention's digital ink processing program generates digital ink describing strokes based on input from an electronic pen via a display. The digital ink processing program causes the processor to perform the following steps: activating the electronic pen's instruction operation on the stroke; after activating the electronic pen's instruction operation, retrieving content associated with the meaning attributes of the indicated stroke or requesting the retrieval from an external server; and controlling the display to show the retrieved content along with the stroke.

[0022] Invention Effects

[0023] According to the present invention, when performing a search using digital ink, the search results are valuable and can provide users with highly relevant information. Attached Figure Description

[0024] Figure 1 This is an overall structural diagram of a digital ink processing system according to one embodiment of the present invention.

[0025] Figure 2 yes Figure 1 The diagram shows a digital ink server, a content server, and a user terminal.

[0026] Figure 3 It is a sequence diagram related to the meaning and actions of a digital ink processing system.

[0027] Figure 4A It is a schematic diagram obtained by visualizing a collection of strokes.

[0028] Figure 4B This is a diagram illustrating an example of how digital ink is constructed.

[0029] Figure 5A This is a diagram showing the classification results of the strokes.

[0030] Figure 5B It shows that it is aimed at in Figure 5A A diagram showing the additional results of the meaning attributes of the groups obtained by classification.

[0031] Figure 6A This is a transition diagram showing the first state change in the display of the user terminal.

[0032] Figure 6B This is a transition diagram showing the first state change in the display of the user terminal.

[0033] Figure 7A This is a transition diagram showing the second state change in the display of the user terminal.

[0034] Figure 7BThis is a transition diagram showing the second state change in the display of the user terminal.

[0035] Figure 8 It is a sequence diagram related to content referencing actions in a digital ink processing system.

[0036] Figure 9A This is a diagram illustrating an example of the data structure of information displayed by an electronic pen.

[0037] Figure 9B This is a diagram illustrating an example of the data structure that content management information has.

[0038] Figure 10A This is a transition diagram showing the third state change in the display of a user terminal.

[0039] Figure 10B This is a transition diagram showing the third state change in the display of a user terminal.

[0040] Figure 11A This is a diagram illustrating an example of the actions of a user terminal that differs from those of a user terminal that has performed handwriting input.

[0041] Figure 11B This is a diagram illustrating an example of the actions of a user terminal that differs from those of a user terminal that has performed handwriting input.

[0042] Figure 12A This is a diagram illustrating the first improved example of the operation of a digital ink processing system.

[0043] Figure 12B This is a diagram illustrating the first improved example of the operation of a digital ink processing system.

[0044] Figure 13A This is a diagram illustrating a second improved example of the operation of a digital ink processing system.

[0045] Figure 13B This is a diagram illustrating a second improved example of the operation of a digital ink processing system.

[0046] Figure 14 This is a schematic diagram illustrating the first example of the reimbursement process that occurs during learning support services.

[0047] Figure 15 This is a diagram illustrating the second example of the reimbursement process that occurs during learning support services. Detailed Implementation

[0048] [Structure of the Digital Ink Processing System 10]

[0049] Figure 1 This is an overall structural diagram of a digital ink processing system 10 according to one embodiment of the present invention. Figure 2yes Figure 1 The diagram shows a block diagram of a digital ink server 20, a content server 30, and a user terminal 40. The digital ink processing system 10 is configured to provide "learning support services" for users such as students to learn efficiently using electronic notebooks. Specifically, the digital ink processing system 10 is configured to include a digital ink server 20, a content server 30, one or more user terminals 40, and one or more electronic pens 50.

[0050] As a data format for digital ink (or ink data), the so-called "ink description language" includes, for example, WILL (Wacom Ink Layer Language), InkML (Ink Markup Language), and ISF (Ink Serialized Format). By using JSON (JavaScript Object Notation), a data structure that describes the digital ink, data exchange between various software and programming languages ​​becomes easy.

[0051] Digital ink server 20 is a computer that performs centralized control related to the processing of digital ink, and can be either cloud-based or on-premises. Here, digital ink server 20 is illustrated as a single computer, but alternatively, digital ink server 20 can also be a cluster of computers forming a distributed system.

[0052] Specifically, the digital ink server 20 is configured to include a communication unit 21, a control unit 22, and a storage unit 23. The communication unit 21 is an interface for transmitting and receiving electrical signals to external devices. The control unit 22 is composed of a processing unit including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The control unit 22 executes programs stored in the storage unit 23, thus functioning as the "processing engine" of the digital ink. The storage unit 23 is composed of a non-transitory, computer-readable storage medium, such as a hard disk drive (HDD) or a solid-state drive (SSD). Therefore, the storage unit 23 stores various data processed by the digital ink server 20.

[0053] Content server 30 is a computer that performs centralized control related to providing educational content, including textbooks and dictionaries, and can be either cloud-based or site-based. Here, content server 30 is illustrated as a single computer, but alternatively, content server 30 can also be a cluster of computers forming a distributed system.

[0054] Specifically, the content server 30 is configured to include a communication unit 31, a control unit 32, and a storage unit 33. The communication unit 31 is an interface for transmitting and receiving electrical signals to external devices. The control unit 32 is composed of a processing unit including a CPU and a GPU. The control unit 32 is configured to execute programs stored in the storage unit 33, thereby enabling it to retrieve and provide content suitable for users.

[0055] User terminal 40 is a terminal that functions as an electronic notebook computer, such as a tablet computer, smartphone, or personal computer. Specifically, user terminal 40 is configured to include a touch panel display 41, a touch IC (Integrated Circuit) 44, a display driver IC 45, a main processor 46, a memory 47, and a communication module 48.

[0056] The touch panel display 41 is configured to include a display panel 42 capable of visually outputting content and sensor electrodes 43 superimposed on the display screen 42. The display panel 42 can display monochrome or color images, such as a liquid crystal panel or an organic EL (Electro-Luminescence) panel. The sensor electrodes 43 are formed by arranging a plurality of X-ray electrodes for detecting the position of the X-axis in the XY sensor coordinate system and a plurality of Y-ray electrodes for detecting the position of the Y-axis in a planar configuration.

[0057] Touch IC 44 is an integrated circuit that drives and controls the sensor electrode 43. Touch IC 44 drives the sensor electrode 43 based on control signals supplied from the main processor 46. Thus, Touch IC 44 performs a "pen detection function" to detect the state of the electronic pen 50 and a "touch detection function" to detect touches made by the user's finger or the like.

[0058] The display driver IC 45 is an integrated circuit that drives the display panel 42. Based on image signals supplied from the main processor 46 in frames, the display driver IC 45 drives the display panel 42. This displays an image within the display area of ​​the display panel 42. This image may include, in addition to writing lines created by the user with the electronic pen 50, application windows, icons, and a cursor.

[0059] The main processor 46 is composed of a processing and arithmetic unit including an MPU (Micro-Processing Unit) and a CPU. The main processor 46 executes programs stored in the memory 47, thereby performing processes such as generating digital ink ink using data from the touch IC 44 and ink regeneration processing for displaying the depicted content represented by the digital ink ink.

[0060] The memory 47 is composed of a non-transitory and computer-readable storage medium. Here, the computer-readable storage medium is a portable medium such as an optical disk, ROM, CD-ROM, flash memory, or a storage device such as an HDD or SSD built into a computer system.

[0061] The communication module 48 is configured to transmit and receive electrical signals to external devices using wired or wireless communication. Thus, for example, the user terminal 40 can transmit and receive digital ink to the digital ink server 20 via the network NW, and receive associated content C1 and C2 from the content server 30.

[0062] The electronic pen 50 is a pen-shaped indicator configured to communicate unidirectionally or bidirectionally via electrostatic coupling formed between it and the user terminal 40. The user can write pictures or text onto the user terminal 40 by holding the electronic pen 50 and pressing the pen tip on the touch surface of the touch panel display 41, thereby moving the pen tip. The electronic pen 50 can be, for example, an active electrostatic coupling (AES) or electromagnetic induction (EMR) stylus.

[0063] [Operation of the Digital Ink Processing System 10]

[0064] The digital ink processing system 10 in this embodiment is configured as described above. Next, regarding the first and second operations of the digital ink processing system 10, please refer to... Figure 3 , Figure 8 The sequence diagram is used for illustration. The "first action" refers to the action related to "meaning interpretation," which involves summarizing the strokes and automatically attaching meaning attributes (or semantic attributes). Additionally, the "second action" refers to the action related to "content citation," which involves timely referencing and displaying content associated with the meaning attributes.

[0065] <Action 1: Meaning and Explanation>

[0066] First, the first operation of the digital ink processing system 10, namely the meaning interpretation operation, will be explained. This first operation is performed through the cooperation of the user terminal 40 (more specifically, the main processor 46) and the digital ink server 20 (more specifically, the control unit 22).

[0067] exist Figure 3 In step S01, the user terminal 40 authenticates the identification information (hereinafter referred to as the pen ID) of the electronic pen 50. If the pen ID received from the electronic pen 50 has been registered, the user terminal 40 allows digital ink editing by the electronic pen 50. Conversely, if the pen ID has not been registered, the user terminal 40 does not allow digital ink editing by the electronic pen 50.

[0068] In step S02, the user terminal 40 accepts input from the electronic pen 50, which was allowed to be edited in step S01, i.e., various writing operations performed by the user. These writing operations include, for example, adding, deleting, and changing strokes, selecting icons, and adding marks and annotations.

[0069] In step S03, the user terminal 40 updates the digital ink periodically or irregularly to reflect the writing operation received in step S02. An example of the former is a case where a predetermined time has elapsed since the most recent update. An example of the latter is a case where an update instruction from the user has been received.

[0070] In step S04, the user terminal 40 uses the data obtained through the writing operation in step S02 to generate digital ink in the form of, for example, WILL (Ver. 3.0).

[0071] Figure 4A This is a diagram that visualizes the collection of strokes G0. Here, collection G0 shows a handwritten shopping list, consisting of the items "milk", "bread", "eggs" and "apples" from top to bottom. Figure 4B This is a diagram illustrating an example of the data structure of a digital ink. A digital ink has a data structure consisting of [1] document metadata, [2] ink semantics, [3] devices, [4] strokes, [5] groups and [6] contexts arranged in sequence.

[0072] When generating a digital ink, the file metadata, device data, stroke data, and context data are already defined, but the meaning data and classification data are not yet defined. That is, the digital ink has not undergone the (1) stroke classification and (2) meaning attribute additions described later. Hereinafter, a digital ink in which the classification data and meaning data are blank is sometimes referred to as "pre-addition ink".

[0073] In step S05, the user terminal 40 sends the additional pre-ink generated in step S04 to the digital ink server 20 in a state associated with the pen ID.

[0074] In step S06, the digital ink server 20 obtains the additional ink and pen ID respectively by receiving the data sent from the user terminal 40 in step S05.

[0075] In step S07, the digital ink server 20 parses the stroke data included in the additional ink obtained in step S06 and classifies the strokes. Specifically, the digital ink server 20 classifies the collection of strokes G0 into one or more groups (e.g., groups G1 to G5) based on the relationship between the strokes' order, position, shape, or the pen pressure of the electronic pen 50, as determined by the stroke data. Through this classification, classification data describing the inclusion relationship between groups and stroke elements, or the inclusion relationship between groups themselves, can be obtained.

[0076] In step S08, the digital ink server 20 processes the addition of meaning attributes to groups G1 to G5 of the strokes obtained from the classification in step S07. Specifically, the digital ink server 20 uses a machine learning-enabled recognizer (e.g., a hierarchical neural network) to infer and add meaning attributes to groups G1 to G5 respectively. Through this processing, for each group obtained from the classification, classification data consisting of pairs of categories and values ​​describing the meaning attributes is obtained.

[0077] For example, the aforementioned recognizer is configured to take the feature quantities of the input stroke (e.g., coordinates of the start point, waypoints, and end point, curvature, etc.) and output a label of the meaning attribute. As an example of "category," examples could include text (including language categories), graphs (including graphic categories), formulas, chemical formulas, lists, tables, etc. As an example of "value," examples could include handwritten characters or strings (e.g., "milk"), handwritten names of objects (e.g., "pencil" for an illustration of a pencil), etc.

[0078] Figure 5A This is a diagram showing the classification results of the strokes. Based on this diagram, it can be understood that set G0 contains group G1, and group G1 consists of four groups G2 to G5. Figure 5B It shows that it is aimed at in Figure 5A A graph showing the additional results of the meaning attributes of the groups obtained from the classification. Group G1 has the category "LIST" and the value "NULL". Group G2 has the category "LIST_ITEM" and the value "milk". Group G3 has the category "LIST_ITEM" and the value "bread". Group G4 has the category "LIST_ITEM" and the value "eggs". Group G5 has the category "LIST_ITEM" and the value "apples".

[0079] exist Figure 3 In step S09, the digital ink server 20 updates the pre-attachment ink by appending the classification data obtained in step S07 and the meaning data obtained in step S08. Hereinafter, the digital ink with the appended classification and meaning data is sometimes referred to as the "post-attachment ink".

[0080] In step S10, the digital ink server 20 sends the updated additional ink from step S09 to the user terminal 40.

[0081] In step S11, the user terminal 40 obtains additional post-ink, including classification data and meaning data, by receiving data sent from the digital ink server 20 in step S10.

[0082] In step S12, the user terminal 40 stores the additional ink obtained in step S11 into the memory 47.

[0083] In step S13, the user terminal 40 confirms that the judgment condition related to the stroke classification is met, thus validating the instruction operation for at least one group. This "judgment condition" is, for example, obtaining meaning data from the digital ink server 20. In this case, groups G2 to G5, whose meaning attribute values ​​are valid (excluding NULL), are selected as valid objects. Furthermore, regarding group G1, whose meaning attribute value is invalid (NULL) and whose meaning is undetermined, and one or more strokes that have not yet formed meaning due to being written, the validity of the instruction operation is retained.

[0084] In step S14, the user terminal 40 displays the groups G2 to G5 of strokes that were activated in step S13 with emphasis compared to before activation. This "emphasis display" means displaying them in a style that is easily recognizable by the user's vision, such as changing the display color, increasing the stroke width, or adding indicator marks.

[0085] Figure 6A and Figure 6B This is a transition diagram showing the first state change in the display of user terminal 40. More specifically, Figure 6A Shows the display state before activation, and, Figure 6B This shows the display status after activation. Figure 6B In the example, underlines are added to the four parts corresponding to groups G2 to G5 in the stroke set G0. This makes it immediately clear to the user that the instructions for the four words "milk", "bread", "eggs", and "apples" have been effectively implemented.

[0086] As described above, the first operation of the digital ink processing system 10 ends. The digital ink processing system 10 can then use the end of the first operation as a trigger to begin the second operation.

[0087] Furthermore, in the above example, the determination of the instruction operation is triggered by the user terminal 40 obtaining meaning data from the digital ink server 20, but the determination condition is not limited to this example. For example, the determination condition could be that the user terminal 40 obtains classification data or meaning data by parsing the digital ink, or it could be the arrival of the time point when the digital ink is automatically or manually saved. In addition, the determination condition could also be that the user terminal 40 accepts an operation to add a mark or annotation to at least a part of the collection of strokes G0.

[0088] Figure 7A and Figure 7B This is a transition diagram showing the second state change in the display of user terminal 40. For example... Figure 7A As shown, with no part of the stroke set G0 valid, the user uses the electronic pen 50 to write the part of "eggs" surrounded by a lasso (a so-called loop). This causes a new window 60 to be displayed along with the stroke set G0 to confirm whether "eggs" needs to be valid. Here, by touching the [ON] button within window 60, the display state of the user terminal 40 changes from... Figure 7A Switch to Figure 7B .

[0089] like Figure 7B As shown, a fluorescent marker is added to the word (eggs) corresponding to group G4 in the stroke set G0. This makes it immediately clear to the user that the instruction operation for the word "eggs" has been effectively implemented.

[0090] <Action 2: Content Reference Action>

[0091] Next, the second operation of the digital ink processing system 10, namely the content referencing operation, will be described. This second operation is performed through the cooperation of the user terminal 40 (more specifically, the main processor 46) and the content server 30 (more specifically, the control unit 32).

[0092] exist Figure 8 In step S21, the user terminal 40 accepts an operation (i.e., an instruction operation) where the user pen 50 instructs one of the activated groups G2 to G5 within the set of strokes G0 displayed on the touch panel display 41. This instruction operation is, for example, in… Figure 6B or Figure 7B The middle part can also be operated by long-pressing the "eggs" (group G4) part.

[0093] In step S22, the user terminal 40 reads the additional ink stored in the memory 47 to obtain the meaning attribute of group G4 indicated in step S21. Specifically, the user terminal 40 determines the group ID corresponding to the indicated position of the electronic pen 50 by parsing the classification data of the additional ink. Then, the user terminal 40 determines the meaning attribute corresponding to the group ID by parsing the meaning data of the additional ink.

[0094] In step S23, the user terminal 40 obtains the pen ID of the electronic pen 50 by receiving the data signal sent from the electronic pen 50 through the sensor electrode 43.

[0095] In step S24, the user terminal 40 sends the meaning data obtained in step S22, associated with the pen ID obtained in step S23, to the content server 30. That is, the user terminal 40 requests content retrieval from the content server 30.

[0096] In step S25, the content server 30 obtains the pen ID and meaning attribute by receiving the data sent from the user terminal 40 in step S24.

[0097] In step S26, the content server 30 uses the meaning attribute and pen ID obtained in step S25 to retrieve content. Specifically, the content server 30 sequentially performs [1] retrieval of content using the pen ID and [2] retrieval within content using the meaning attribute. Here, the pen ID functions as identification information (hereinafter referred to as the usage ID) used to determine whether content usage is permitted. In addition to the pen ID, the usage ID can also be the identification information of the user terminal 40 (hereinafter referred to as the terminal ID), the user's identification information (hereinafter referred to as the user ID), or a combination thereof.

[0098] Figure 9A This diagram illustrates an example of the data structure of the electronic pen information 62. The electronic pen information 62 represents the correspondence between the electronic pen 50's identification information (i.e., the "pen ID"), the user's "school," and the corresponding "grade / class." The pen ID is equivalent to a usage ID used to manage content usage. Examples of "schools" include elementary school, junior high school, high school, university, preparatory school, etc. Regarding "grade / class," various grade years or classes can be selected depending on the type of school.

[0099] Figure 9Bis a diagram showing an example of the data structure of the content management information 64. The content management information 64 represents the correspondence between the "school" to which the user belongs, the "school year / class" matching the user, and the usable "content type". The "school" and "school year / class" are defined substantially in the same manner as the electronic pen information 62, respectively. Examples of content types include book names including textbooks, reference books, and exercise collections, subject names, publisher names, and the like.

[0100] The content server 30 reads out the electronic pen information 62 and the content management information 64, and performs verification using "school" and "school year / class" as search keywords, thereby acquiring at least one type of content corresponding to the pen ID. Then, the content server 30 uses various search methods to acquire information (hereinafter referred to as "associated content C1") that matches or is approximate to the "semantic attribute (value)" as the search keyword from the content serving as the search target.

[0101] In Figure 8 step S27, the content server 30 transmits the associated content C1, which is the search result of step 26, to the user terminal 40 in a state associated with the pen ID.

[0102] In step S28, the user terminal 40 acquires the associated content C1 and the pen ID respectively by receiving the data transmitted from the content server 30 in step S27.

[0103] In step S29, the user terminal 40 temporarily stores the associated content C1 acquired in step S28 in the memory 47.

[0104] In step S30, the user terminal 40 simultaneously displays the associated content C1 obtained by the search together with the stroke aggregate G0 on the touch panel display 41.

[0105] Figure 10A and Figure 10B are transition diagrams showing the third state change in the display of the user terminal 40. More specifically, Figure 10A shows the display state before the associated content C1 appears, and Figure 10B shows the display state after the associated content C1 appears. It can be understood from the two diagrams that by performing an instruction operation on the portion of "eggs", the associated content C1 is displayed together in the periphery of "eggs" (here, at the lower right). This associated content C1 is composed of "eggs" written in English and "Meaning: egg" written in Japanese.

[0106] The user can easily confirm that the Japanese translation of the English word "eggs" pointed to by the electronic pen 50 is "egg (plural form)". In this way, the learning assistance service provided by the digital ink processing system 10 further improves learning efficiency.

[0107] However, the administrator of the electronic notebook sometimes implements management where the electronic notebook is lent to users within the school, but the electronic notebook is prohibited from being taken outside the school. Therefore, by associating the content usage ID with the pen ID in advance, the user can receive the above-mentioned learning assistance service even at home.

[0108] Figure 11A This shows a case where the user terminal 40B is different from the user terminal 40A on which handwritten input was performed using the electronic pen 50. For example, the user can use the same electronic pen 50 brought back from school to edit the digital ink Ink. Alternatively, the digital ink processing system 10 may be configured to refer to the terminal ID included in the "device data" of the digital ink Ink to restrict or prohibit editing of the digital ink Ink on the user terminal 40B with a mismatched value.

[0109] On the other hand, as Figure 11B shown, when the user points to the corresponding area using the electronic pen 50, the user terminal 40B can display the associated content C1 on the touch panel display 41 in the same manner as the user terminal 40A. Accordingly, even in a situation where it is difficult to move the user terminal 40A freely, the user can receive the learning assistance service in various places by carrying their own electronic pen 50.

[0110] [Modified Example of Operation]

[0111] In the following, with reference to Figures 12A to 13B , a modified example of the operation of the digital ink processing system 10 will be described.

[0112] <First Modified Example>

[0113] As the user's learning progresses and their proficiency increases, it is estimated that the information the user wants to know will gradually change. Therefore, the user terminal 40 may be configured to display different associated content C1 and C2 according to the user's proficiency. For example, in Figure 8 the sequence diagram, an additional step S31 may be provided between steps S23 and S24.

[0114] In Figure 8In step S31, the user terminal 40 performs a search within the meaning data of the meaning attribute "values" (e.g., eggs) obtained in step S22 as keywords. The search object is not limited to the digital ink in the editing process, but may also include digital ink stored in its own memory 47 or the digital ink server 20. For example, the user terminal 40 counts the number of "values" that are consistent with or similar to the keywords, and calculates this count as the "occurrence level". Subsequently, in steps S24 and S25, data including the occurrence level is transmitted and received between the user terminal 40 and the content server 30.

[0115] In step S26, the content server 30 uses the pen ID, meaning attribute, and occurrence degree obtained in step S25 to retrieve content. Specifically, the content server 30 sequentially performs [1] retrieval of content using the pen ID, [2] selection of content using the occurrence degree, and [3] retrieval within content using the meaning attribute.

[0116] Figure 12A This is a diagram illustrating an example of the data structure of the judgment table 66. Judgment table 66 shows the correspondence between the "number of occurrences" representing the degree of occurrence and the "proficiency" of a typical user. In the example of this diagram, judgment table 66 describes the following criteria: [1] low proficiency when the number of occurrences is less than 9, [2] moderate proficiency when the number of occurrences is more than 10 but less than 20, and [3] high proficiency when the number of occurrences is more than 20. Furthermore, in this diagram, the number of occurrences can be listed as an example of the degree of occurrence, but the frequency of occurrence can also be used together with it or used independently of it.

[0117] Content server 30 uses the determination table 66 to determine the user's proficiency level based on the frequency of occurrence, and then selects content corresponding to that proficiency level from a variety of content items associated with the pen ID. For example, in the case of a user with a high proficiency level, it retrieves content related to the pen ID. Figure 10B The associated content C1 shown is different from the associated content C2. Then, steps S27 to S29 are performed in the same manner as described above.

[0118] In step S30, the user terminal 40 displays the retrieved associated content C2 and the collection of strokes G0 together on the touch panel display 41. For example... Figure 12B As shown, the associated content C2 is displayed around the "eggs" text that has been instructed. This associated content C2 consists of the string "Birdslay eggs.", an example sentence using the English word "eggs".

[0119] By displaying different related content C1 and C2 based on the frequency of occurrence of the semantic attributes within the same user ID (in this case, pen ID), information suitable for users with varying levels of proficiency can be provided. Furthermore, since there is a high correlation between frequency of occurrence and user attention, displaying the same information as for proficiency levels is effective.

[0120] <Second Improved Example>

[0121] Since electronic notebooks are personal items, it is sometimes inappropriate for others to be able to freely edit one's electronic notebook. Therefore, the user terminal 40 can also be configured to change editing permissions based on the pen ID of the electronic pens 50A and 50B.

[0122] like Figure 13A As shown, the pen ID (e.g., ID = 002) of the electronic pen 50B identified by the user terminal 40 is different from the pen ID (e.g., ID = 001) of the electronic pen 50A used for handwriting input. In this case, the user terminal 40 displays a message 70 on the touch panel display 41 indicating that editing of the digital ink is restricted. While displaying message 70, the user terminal 40 disables or restricts editing of the digital ink using the electronic pen 50B. Here, "disable" means not processing any editing-related operations. "Restrict" means making a portion of the editing-related functions unusable.

[0123] On the other hand, such as Figure 13B As shown, when a user points to a corresponding area using the electronic pen 50B, the user terminal 40 displays the associated content C1 on the touch panel display 41, just as with the electronic pen 50A. This prevents unauthorized editing of the user's electronic notebook and ensures that even those other than the user can access the content.

[0124] [Payment Process for Learning Assistance Services]

[0125] However, along with the provision of the aforementioned learning assistance services, transactions occur regarding [1] usage fees for the meaning explanation service and [2] usage fees for the content citation service. The payment process between the digital ink server 20, the content server 30, the user terminal 40, and the operator-side server 80 is described below.

[0126] Here, the operator-side server 80 is a server managed by the content provider (e.g., a publisher). Furthermore, for ease of explanation, user terminals 40 participate in the payment verification process, but the school-side server (not shown) that manages each user terminal 40 can also participate in the payment verification process together with or independently of user terminals 40.

[0127] <Example 1>

[0128] Figure 14 This is a schematic diagram illustrating the first example of the reimbursement process that occurs during learning support services.

[0129] [T11] Whenever the user terminal 40 uses the meaning interpretation service, it accesses the digital ink server 20.

[0130] [T12] For each due date / user of the usage fee, the digital ink server 20 interprets the usage status of the service in terms of total meaning and performs the payment processing for the usage fee of the service in accordance with the prescribed payment rules.

[0131] [T13] The digital ink server 20 notifies each user terminal 40 of the request for payment, indicating that the usage fee has been clearly defined through the payment processing. Users who receive this notification complete the payment of the usage fee by making payment within the payment period.

[0132] [T14] On the other hand, whenever the user terminal 40 uses the content referencing service, it accesses the content server 30.

[0133] [T15] Content server 30 records the usage status of the content associated with the content referencing service for each expiration date / per user of the usage fee.

[0134] [T16] Content server 30 notifies operator-side server 80 of the content usage status of each user.

[0135] [T17] Based on the total results of usage, the operator-side server 80 processes the payment of usage fees for the content in accordance with the prescribed payment rules.

[0136] [T18] The operator-side server 80 notifies each user terminal 40 of the request for payment, indicating that the usage fee has been clearly defined through the payment processing. Users who receive this notification complete the payment of the usage fee by making payment within the payment period.

[0137] [T19] The operator-side server 80 notifies the content server 30 that it intends to pay the service platform provision fee in accordance with the prescribed contract rules (e.g., a pro-rata system proportional to the content usage fee).

[0138] <Example 2>

[0139] Figure 15 This is a diagram illustrating the second example of the reimbursement process that occurs during learning support services.

[0140] [T21] Whenever the user terminal 40 uses the meaning interpretation service, it accesses the digital ink server 20.

[0141] [T22] The digital ink server 20 interprets the usage status of the service for each due date / each user, and performs the payment processing for the service usage fee in accordance with the prescribed payment rules.

[0142] [T23] The digital ink server 20 notifies each user terminal 40 of the request for payment, indicating that the usage fee has been clearly defined through the payment processing. Users who receive this notification complete the payment of the usage fee by making payment within the payment period.

[0143] [T24] On the other hand, whenever the user terminal 40 uses the content citation service, it accesses the content server 30.

[0144] [T25] For each expiration date of the usage fee / for each user, the content server 30, based on the usage status of the content reference service, performs the payment processing for the usage fee of the content in accordance with the prescribed payment rules.

[0145] [T26] Content server 30 notifies each user terminal 40 of the request for payment, indicating that the usage fee has been clearly defined through the payment processing. Users who receive this notification complete the payment of the usage fee by making payment within the payment period.

[0146] [T27] The content server 30 notifies the operator-side server 80 that the payment is the amount obtained by subtracting the service platform provision fee from the content usage fee.

[0147] [Variation Example]

[0148] Furthermore, the present invention is not limited to the above-described embodiments, and can be freely modified without departing from the spirit of the invention. Alternatively, the various structures can be arbitrarily combined without causing technical inconsistencies.

[0149] In the above embodiments, the digital ink processing system 10 is configured to provide learning assistance services to support education for students, but the types of services or target users are not limited to the examples described above.

[0150] In the above embodiments, the digital ink server 20 performs stroke classification and adds meaning attributes, but the user terminal 40 can also be configured to perform classification or both classification and addition in place of the digital ink server 20. For example, the user terminal 40 can obtain meaning attributes by parsing the digital ink itself, or it can obtain meaning attributes through manual input by the user.

[0151] In the above embodiment, a digital ink with defined meaning data was used as an example for explanation, but a digital ink without defined meaning data can also be used. In this case, by managing the digital ink and meaning data in a way that is interconnected, the same operation as in the above embodiment can be achieved.

[0152] [Summary of Implementation Methods]

[0153] As described above, the digital ink processing system 10 is configured to include an electronic pen 50 and a user terminal 40. The user terminal 40 is configured to receive writing operations performed by the electronic pen 50 via a touch panel display 41 and to generate digital ink describing a collection G0 of strokes. From the collection G0 displayed on the touch panel display 41, the user terminal 40 enables the electronic pen 50's instruction operation for groups G2 to G5 of strokes that meet the determination conditions related to stroke classification, while retaining the enabling of instruction operations for group G1 of strokes that do not meet the determination conditions. When an enabled instruction operation is received, the system retrieves content associated with the meaning attributes of the specified groups G2 to G5 or requests the retrieval from an external source, and displays the associated content C1 and C2 obtained through the retrieval along with the collection G0 on the touch panel display 41.

[0154] In addition, according to the digital ink processing method and procedure, the user terminal 40 performs the following steps: Step (S13), from the collection of strokes G0 displayed on the touch panel display 41, the instruction operation of the electronic pen 50 that indicates the groups G2 to G5 of strokes that meet the determination conditions related to the classification of strokes is made valid, and on the other hand, the validity of the instruction operation for the group G1 of strokes that do not meet the determination conditions is retained; and Step (S30), when a valid instruction operation is received (S21), the content associated with the meaning attributes attached to the groups G2 to G5 is retrieved or the retrieval is requested from the outside (S24), and the associated content C1 and C2 obtained by the retrieval are displayed together with the collection G0 on the touch panel display 41.

[0155] In this way, the operation of the electronic pen 50 is made effective only for strokes that meet the criteria related to the stroke classification. Therefore, by appropriately setting the criteria, content retrieval is limited to groups G2 to G5 obtained by classification in a way that maximizes the accuracy of the meaning formed by summarization. As a result, when performing searches using digital ink, the search results are valuable and can provide the user with highly relevant information.

[0156] Additionally, the user terminal 40 can also emphasize the groups G2 to G5 where the instruction operation has been activated compared to before activation. This allows the user to easily visually recognize the meaning and location of the activated instruction operation.

[0157] Alternatively, the judgment criteria can also be based on adding meaning attributes to stroke groups G2 to G5. This allows the added meaning attributes to be used as search keywords, resulting in more valuable search results for users.

[0158] Furthermore, the digital ink server 20 adds meaning attributes to groups G2 to G5 by parsing the digital ink sent from the user terminal 40. The determination condition can also be that the meaning data representing the meaning attributes is obtained from the digital ink server 20. By having the digital ink server 20 add meaning attributes instead of the user terminal 40, the parsing load performed by the user terminal 40 is reduced.

[0159] Alternatively, the determination criterion can be the user's action of adding tags or annotations to groups G2 to G5. Thus, by using tags or annotations, it is possible to specify a summary of strokes whose meanings were found by the user who performed the action, making it easier to obtain more accurate meaning attributes.

[0160] Alternatively, the content server 30 can store content corresponding to usage IDs, and the user terminal 40 can retrieve associated content C1 and C2 that are associated with the meaning attributes and are allowed to be used by sending data including usage IDs and meaning attributes to the content server 30 and requesting retrieval.

[0161] In addition, user terminal 40 can also obtain different associated content C1 and C2 based on the frequency of occurrence of the meaning attributes of the same user ID. By considering the frequency of occurrences with high relevance to the user's proficiency or attention, it is possible to select information suitable for users with different levels of proficiency or attention to provide prompts.

[0162] Alternatively, the ID used can also be a pen ID used to identify the electronic pen 50. This allows for the association of usable content with the electronic pen 50, enabling the same associated content C1 and C2 to be displayed using the same electronic pen 50, regardless of the user terminals 40A and 40B.

[0163] Alternatively, when a digital ink ink is generated using an electronic pen 50A with a first ID, and a second ID (different from the first ID) is used using an electronic pen 50B, the user terminal 40 may prohibit or restrict editing of the digital ink ink and be subject to instructions from the electronic pen 50B. This prevents the user-generated digital ink ink from being edited arbitrarily, and ensures that even to someone other than the user, the same associated content C1 and C2 is displayed as in the user's case.

[0164] In addition, the content server 30 can also process payments to content users and providers based on content usage. Thus, content management and usage fee management can be unified through a single device.

[0165] Label Explanation

[0166] 10 Digital ink processing system, 20 Digital ink server, 30 Content server, 40, 40A, 40B User terminal, 41 Touch panel display, 50, 50A, 50B Electronic pen, 80 Operator-side server, C1, C2 Related content, G0 Collection, G1~G5 Group, Ink digital ink.

Claims

1. A digital ink processing system, wherein, have: Electronic pen; and The computer generates digital ink describing the strokes based on input made by the electronic pen via a display. The computer has a processor. The processor enables the electronic pen to perform instruction operations on the strokes. After activating the pointer operation of the electronic pen, the processor retrieves content associated with the semantic attributes attached by using a machine learning-based recognizer to interpret the summary of the indicated strokes, or requests the retrieval from an external server. The processor controls the display to show the retrieved content along with the strokes. The processor enables the electronic pen to indicate strokes that have been marked or annotated by the user.

2. A digital ink processing system, wherein, have: Electronic pen; and The computer generates digital ink describing the strokes based on input made by the electronic pen via a display. The computer has a processor. The processor enables the electronic pen to perform instruction operations on the strokes. After activating the pointer operation of the electronic pen, the processor retrieves content associated with the semantic attributes attached by using a machine learning-based recognizer to interpret the summary of the indicated strokes, or requests the retrieval from an external server. The processor controls the display to show the retrieved content along with the strokes. The meaning attribute includes the category of the stroke summary and the value of the stroke summary, and the value is used as a search keyword for the search.

3. The digital ink processing system according to claim 1 or 2, wherein, The processor controls the display by emphasizing the strokes on the screen after the electronic pen's instruction on the strokes has been activated, compared to before activation.

4. The digital ink processing system according to claim 2, wherein, The processor enables the electronic pen to perform instruction operations for strokes with the attached meaning attributes.

5. The digital ink processing system according to claim 4, wherein, The digital ink processing system also includes a digital ink server, which adds the meaning attribute to the stroke by parsing the digital ink sent from the computer. The processor enables the electronic pen to perform instruction operations on strokes for which meaning data has been obtained from the digital ink server, wherein the meaning data represents the meaning attribute.

6. The digital ink processing system according to claim 1 or 2, wherein, The digital ink processing system also includes a content server, which stores content by establishing a correspondence between content and usage ID. The processor retrieves content associated with and permitted for use from the content server by sending data including the usage ID and the meaning attribute to the content server and requesting retrieval.

7. The digital ink processing system according to claim 6, wherein, The processor retrieves different content based on the frequency of occurrence of the semantic attribute for the same usage ID.

8. The digital ink processing system according to claim 6, wherein, The usage ID is the pen ID used to identify the electronic pen.

9. The digital ink processing system according to claim 7, wherein, The usage ID is the pen ID used to identify the electronic pen.

10. The digital ink processing system according to claim 8 or 9, wherein, In the case of generating digital ink using a first electronic pen with a first ID attached, and using a second electronic pen with a second ID attached that is different from the first ID. The processor disables or restricts the editing of the digital ink and accepts the pen's instructions for the strokes.

11. The digital ink processing system according to claim 6, wherein, The content server performs payment processing on users and suppliers of the content in accordance with the usage status of the content.

12. The digital ink processing system according to claim 2, wherein, The types of stroke summaries include text, graphics, formulas, chemical formulas, lists, and tables.

13. The digital ink processing system according to claim 2, wherein, The total value of the strokes includes the name of the handwritten object, the handwritten text, and the handwritten string.

14. A digital ink processing method, wherein, The processor generates digital ink describing the strokes based on input from the electronic pen via the display, and the processor performs the following steps: This enables the electronic pen to effectively instruct the strokes. After enabling the indication operation of the electronic pen, retrieve content associated with the semantic attributes attached by using a machine learning-based recognizer to interpret the summary of the indicated strokes, or request the retrieval from an external server; and The control is performed by displaying the retrieved content along with the strokes on the display screen. The processor enables the electronic pen to indicate strokes that have been marked or annotated by the user.

15. A digital ink processing method, wherein, The processor generates digital ink describing the strokes based on input from the electronic pen via the display, and the processor performs the following steps: This enables the electronic pen to effectively instruct the strokes. After enabling the indication operation of the electronic pen, retrieve content associated with the semantic attributes attached by using a machine learning-based recognizer to interpret the summary of the indicated strokes, or request the retrieval from an external server; and The control is performed by displaying the retrieved content along with the strokes on the display screen. The meaning attribute includes the category of the stroke summary and the value of the stroke summary, and the value is used as a search keyword for the search.

16. A processing apparatus that generates strokes based on input from an electronic pen via a display, wherein, The processing device has a processor. The processor enables the electronic pen to perform instruction operations on the strokes. After activating the pointer operation of the electronic pen, the processor retrieves content associated with the semantic attributes attached by using a machine learning-based recognizer to interpret the summary of the indicated strokes, or requests the retrieval from an external server. The processor controls the display to show the retrieved content along with the strokes. The processor enables the electronic pen to indicate strokes that have been marked or annotated by the user.

17. A processing apparatus that generates strokes based on input from an electronic pen via a display, wherein, The processing device has a processor. The processor enables the electronic pen to perform instruction operations on the strokes. After activating the pointer operation of the electronic pen, the processor retrieves content associated with the semantic attributes attached by using a machine learning-based recognizer to interpret the summary of the indicated strokes, or requests the retrieval from an external server. The processor controls the display to show the retrieved content along with the strokes. The meaning attribute includes the category of the stroke summary and the value of the stroke summary, and the value is used as a search keyword for the search.

18. A computer program product comprising a digital ink processing program, wherein, The processor generates digital ink describing the strokes based on input from the electronic pen via the display. The digital ink processing program causes the processor to perform the following steps: This enables the electronic pen to effectively instruct the strokes. After enabling the indication operation of the electronic pen, retrieve content associated with the semantic attributes attached by using a machine learning-based recognizer to interpret the summary of the indicated strokes, or request the retrieval from an external server; and The control is performed by displaying the retrieved content along with the strokes on the display screen. The processor enables the electronic pen to indicate strokes that have been marked or annotated by the user.

19. A computer program product comprising a digital ink processing program, wherein, The processor generates digital ink describing the strokes based on input from the electronic pen via the display. The digital ink processing program causes the processor to perform the following steps: This enables the electronic pen to effectively instruct the strokes. After enabling the indication operation of the electronic pen, retrieve content associated with the semantic attributes attached by using a machine learning-based recognizer to interpret the summary of the indicated strokes, or request the retrieval from an external server; and The control is performed by displaying the retrieved content along with the strokes on the display screen. The meaning attribute includes the category of the stroke summary and the value of the stroke summary, and the value is used as a search keyword for the search.

Images