Processing device and display method

Abstract

To provide a processing device and a display method capable of presenting a user with information having a significant retrieval result and high relevance in retrieving content.SOLUTION: A processor included in a processing device controls a display so as to display a group of strokes, retrieves content related to a semantic attribute outputted by using a discriminator undergoing machine learning to interpret the group of strokes or requests the retrieval to an external server, and controls the content acquired by the retrieval so as to be displayed in the display.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to a processing device and a display method. [Background technology]

[0002] Patent Document 1 discloses a technique for performing a search using at least a part of information displayed on the screen of an information processing device including a personal computer as a search key, and displaying the search results in a pop-up display on the screen. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-114955 A Summary of the Invention [Problem to be solved by the invention]

[0004] In digital ink, which describes a collection of strokes, meaning often first emerges from a group of strokes. However, when attempting to perform a search by specifying a location that corresponds to a search key while editing digital ink, a meaningless search key or a search key with an incorrect meaning may be set, resulting in a problem of not being able to obtain appropriate search results.

[0005] An object of the present invention is to provide a digital ink processing system, method, and program capable of presenting meaningful search results and highly relevant information to a user when searching using digital ink. [Means for solving the problem]

[0006] The digital ink processing system of the first invention comprises an electronic pen and a tablet that generates digital ink describing strokes in response to input by the electronic pen via a display, the tablet having a processor that enables a pointing operation of the electronic pen on the strokes, and after enabling the pointing operation of the electronic pen, searches for content related to semantic attributes of the pointed strokes or requests the search from an external server, and controls the content obtained by the search to be displayed on the display together with the strokes.

[0007] The processor may also perform control so that the stroke for which the pointing operation has been validated is displayed on the display in an emphasized manner compared to before the pointing operation was validated.

[0008] The processor may also validate a pointing operation of the electronic pen for a stroke to which the semantic attribute has been assigned.

[0009] The system may further include a digital ink server that assigns the semantic attributes to the strokes by analyzing the digital ink transmitted from the tablet, and the processor may enable a pointing operation of the electronic pen for strokes for which semantic data indicating the semantic attributes has been obtained from the digital ink server.

[0010] The processor may also validate a pointing operation of the electronic pen for a stroke where a user has operated to add a mark or annotation.

[0011] In addition, the system may further include a content server that stores content in association with a usage ID, and the processor may obtain content that is related to the semantic attribute and is permitted for use from the content server by transmitting data including the usage ID and the semantic attribute to the content server and requesting a search.

[0012] The processor may also acquire different content depending on the occurrence rate of the semantic attribute for the same usage ID.

[0013] The usage ID may be a pen ID for identifying the electronic pen.

[0014] In addition, when digital ink is generated using a first electronic pen assigned a first pen ID, and a second electronic pen assigned a second pen ID different from the first pen ID is used, the processor may prohibit or restrict editing of the digital ink while accepting the instruction operation.

[0015] The content server may also perform a billing process for the content user and the content provider according to the usage status of the content.

[0016] A digital ink processing method in a second aspect of the present invention includes a processor that generates digital ink describing strokes in response to input by an electronic pen via a display, and executes the steps of: enabling a pointing operation of the electronic pen on the strokes; searching for content related to semantic attributes of the indicated strokes or requesting a search from an external server after enabling the pointing operation of the electronic pen; and controlling the content obtained by the search to be displayed on the display together with the strokes.

[0017] The digital ink processing program of the third invention causes a processor to generate digital ink describing a stroke in response to input by an electronic pen via a display, and executes the steps of: enabling a pointing operation of the electronic pen on the stroke; searching for content related to semantic attributes of the indicated stroke or requesting a search from an external server after enabling the pointing operation of the electronic pen; and controlling the content obtained by the search to be displayed on the display together with the stroke. Effect of the Invention

[0018] According to the present invention, when searching using digital ink, search results can be presented to a user with meaningful and highly relevant information. [Brief description of the drawings]

[0019] [Figure 1] 1 is a diagram illustrating the overall configuration of a digital ink processing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram of the digital ink server, the content server, and the user terminal shown in FIG. [Diagram 3] FIG. 2 is a sequence diagram of a semantic interpretation operation of the digital ink processing system. [Figure 4A] FIG. 1 is a schematic diagram illustrating a visualization of a collection of strokes. [Figure 4B] FIG. 2 is a diagram illustrating an example of a data structure of digital ink. [Figure 5A] FIG. 13 is a diagram showing a stroke classification result. [Figure 5B] FIG. 5B is a diagram showing the results of assigning semantic attributes to the groups classified in FIG. 5A. [Figure 6A] FIG. 11 is a transition diagram showing a first state change in the display of the user terminal. [Figure 6B] FIG. 11 is a transition diagram showing a first state change in the display of the user terminal. [Figure 7A] FIG. 11 is a transition diagram showing a second state change in the display of the user terminal. [Figure 7B] FIG. 11 is a transition diagram showing a second state change in the display of the user terminal. [Figure 8] FIG. 1 is a sequence diagram of a content citation operation of the digital ink processing system. [Figure 9A] FIG. 4 is a diagram showing an example of a data structure of electronic pen information. [Figure 9B] FIG. 2 is a diagram illustrating an example of a data structure of content management information. [Figure 10A] FIG. 13 is a transition diagram showing a third state change in the display of the user terminal. [Figure 10B] FIG. 13 is a transition diagram showing a third state change in the display of the user terminal. [Figure 11A] FIG. 11 is a diagram illustrating an example of the operation of a user terminal different from the user terminal that has performed handwriting input. [Figure 11B] FIG. 11 is a diagram illustrating an example of the operation of a user terminal different from the user terminal that has performed handwriting input. [Figure 12A] FIG. 1 illustrates a first improved example of the operation of the digital ink processing system. [Figure 12B] FIG. 1 illustrates a first improved example of the operation of the digital ink processing system. [Figure 13A] FIG. 13 illustrates a second refinement of the operation of the digital ink processing system. [Figure 13B] FIG. 13 illustrates a second refinement of the operation of the digital ink processing system. [Figure 14] FIG. 11 is a schematic diagram showing a first example of a billing flow that occurs during a learning support service. [Figure 15] FIG. 11 is a schematic diagram showing a second example of a billing flow that occurs during the learning support service. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] [Configuration of digital ink processing system 10] Fig. 1 is an overall configuration diagram of a digital ink processing system 10 according to an embodiment of the present invention. Fig. 2 is a block diagram of the digital ink server 20, content server 30, and user terminal 40 shown in Fig. 1. The digital ink processing system 10 is configured to be able to provide a "learning support service" that enables users such as students to study efficiently using electronic notebooks. Specifically, the digital ink processing system 10 is configured to include the digital ink server 20, the content server 30, one or more user terminals 40, and one or more electronic pens 50.

[0021] Examples of data formats for digital ink (or ink data), so-called "ink description languages," include WILL (Wacom Ink Layer Language), InkML (Ink Markup Language), and ISF (Ink Serialized Format). By describing this digital ink using the data structure format of JSON (JavaScript (registered trademark) Object Notation), data can be easily exchanged between various software and programming languages.

[0022] The digital ink server 20 is a computer that performs overall control over the processing of digital ink, and may be either a cloud-based or on-premise type. Here, the digital ink server 20 is illustrated as a single computer, but instead, the digital ink server 20 may be a group of computers that form a distributed system.

[0023] Specifically, the digital ink server 20 includes a communication unit 21, a control unit 22, and a storage unit 23. The communication unit 21 is an interface that transmits and receives electrical signals to and from external devices. The control unit 22 is configured with a processing and arithmetic device that includes a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The control unit 22 functions as a "processing engine" of the digital ink Ink by reading and executing a program stored in the storage unit 23. The storage unit 23 is configured with a non-transient and computer-readable storage medium, such as a hard disk drive (HDD) or a solid state drive (SSD). As a result, the storage unit 23 stores various data handled by the digital ink server 20.

[0024] The content server 30 is a computer that performs overall control over the provision of educational content, including textbooks and dictionaries, and may be either a cloud-based or on-premise type. Here, the content server 30 is illustrated as a single computer, but instead, the content server 30 may be a group of computers that form a distributed system.

[0025] Specifically, the content server 30 includes a communication unit 31, a control unit 32, and a storage unit 33. The communication unit 31 is an interface that transmits and receives electrical signals to and from external devices. The control unit 32 is configured to be able to search for and provide content suitable for a user by reading and executing a program stored in the storage unit 33.

[0026] The user terminal 40 is a terminal that functions as an electronic notebook, and is configured, for example, as a tablet, a smartphone, or a personal computer. Specifically, the user terminal 40 includes a touch panel display 41, a touch IC (Integrated Circuit) 44, a display driving IC 45, a host processor 46, a memory 47, and a communication module 48.

[0027] The touch panel display 41 includes a display panel 42 capable of visibly outputting content, and a sensor electrode 43 arranged to overlap the display screen of the display panel 42. The display panel 42 is capable of displaying monochrome or color images, and may be, for example, a liquid crystal panel or an organic EL (Electro-Luminescence) panel. The sensor electrode 43 is formed by arranging a plurality of X-line electrodes for detecting an X-axis position in an XY sensor coordinate system and a plurality of Y-line electrodes for detecting a Y-axis position in a planar manner.

[0028] The touch IC 44 is an integrated circuit that controls the driving of the sensor electrodes 43. The touch IC 44 drives the sensor electrodes 43 based on a control signal supplied from the host processor 46. In this way, the touch IC 44 executes a "pen detection function" for detecting the state of the electronic pen 50 and a "touch detection function" for detecting a touch by a user's finger or the like.

[0029] The display drive IC 45 is an integrated circuit that controls the drive of the display panel 42. The display drive IC 45 drives the display panel 42 based on an image signal in frame units supplied from the host processor 46. As a result, an image is displayed within the display area of ​​the display panel 42. This image may include not only lines drawn by the user using the electronic pen 50, but also application windows, icons, and a cursor.

[0030] The host processor 46 is configured with a processing operation device including an MPU (Micro-Processing Unit) and a CPU. The host processor 46 reads and executes a program stored in the memory 47 to perform a process of generating digital ink (Ink) using data from the touch IC 44, an ink regeneration process for displaying drawing contents indicated by the digital ink (Ink), and the like.

[0031] The memory 47 is non-transient and is configured with a computer-readable storage medium. Here, the computer-readable storage medium is a portable medium such as a magneto-optical disk, a ROM, a CD-ROM, or a flash memory, or a storage device such as a HDD or SSD built into a computer system.

[0032] The communication module 48 is configured to be capable of transmitting and receiving electrical signals to and from an external device using wired or wireless communication. This allows, for example, the user terminal 40 to transmit and receive digital ink Ink to and from the digital ink server 20 via the network NW, and to receive related contents C1 and C2 from the content server 30.

[0033] The electronic pen 50 is a pen-shaped pointing device, and is configured to be capable of one-way or two-way communication via electrostatic coupling formed with the user terminal 40. A user can write pictures or characters on the user terminal 40 by holding the electronic pen 50 and moving the pen tip while pressing it against the touch surface of the touch panel display 41. The electronic pen 50 is, for example, an active electrostatic coupling type (AES) or electromagnetic induction type (EMR) stylus.

[0034] [Operation of digital ink processing system 10] The digital ink processing system 10 in this embodiment is configured as described above. Next, the first and second operations of this digital ink processing system 10 will be described mainly with reference to the sequence diagrams of Fig. 3 and Fig. 8. The "first operation" refers to an operation related to "semantic interpretation" that interprets a group of strokes and automatically assigns a semantic attribute (or a semantic attribute). The "second operation" refers to an operation related to "content citation" that timely cites and displays content related to the semantic attribute.

[0035] <First action: Semantic interpretation action> First, a description will be given of the first operation, i.e., the semantic interpretation operation, of the digital ink processing system 10. This first operation is executed by the user terminal 40 (more specifically, the host processor 46) and the digital ink server 20 (more specifically, the control unit 22) in cooperation with each other.

[0036] 3, the user terminal 40 performs authentication using identification information (hereinafter, referred to as pen ID) of the electronic pen 50. If the pen ID received from the electronic pen 50 has already been registered, the user terminal 40 allows editing of the digital ink Ink by the electronic pen 50. On the other hand, if the pen ID has not been registered, the user terminal 40 does not allow editing of the digital ink Ink by the electronic pen 50.

[0037] In step S02, the user terminal 40 accepts input using the electronic pen 50 for which editing is permitted in step S01, that is, various writing operations by the user. These writing operations include, for example, adding, deleting, and changing strokes, selecting icons, and adding marks and annotations.

[0038] In step S03, the user terminal 40 updates the digital ink Ink periodically or irregularly to reflect the writing operation received in step S02. An example of the former is when a predetermined time has passed since the most recent update. An example of the latter is when an instruction operation to update is received from the user.

[0039] In step S04, the user terminal 40 generates digital ink Ink in, for example, the WILL (Ver. 3.0) format, using the data obtained through the writing operation in step S02.

[0040] FIG. 4A is a schematic diagram visualizing a collection G0 of strokes. Here, the collection G0 shows a handwritten shopping list, which is composed of the items "milk," "bread," "eggs," and "apples" from the top. FIG. 4B is a diagram showing an example of the data structure of digital ink Ink. Digital ink Ink has a data structure in which [1] document metadata, [2] ink semantics, [3] device data, [4] stroke data, [5] group data, and [6] context data are sequentially arranged.

[0041] When digital ink Ink is generated, the document metadata, device data, stroke data, and context data have already been determined, but the semantic data and classification data have not yet been determined. In other words, (1) stroke classification and (2) semantic attribute assignment, which will be described later, have not been performed on this digital ink Ink. Hereinafter, digital ink Ink in which classification data and semantic data are blank may be referred to as "pre-assigned ink."

[0042] In step S05, the user terminal 40 transmits the pre-applied ink generated in step S04 to the digital ink server 20 in a state in which the pre-applied ink is linked to the pen ID.

[0043] In step S06, the digital ink server 20 receives the data transmitted from the user terminal 40 in step S05, and thereby acquires the pre-applied ink and the pen ID.

[0044] In step S07, the digital ink server 20 analyzes the stroke data contained in the pre-applied ink acquired in step S06, and classifies the strokes. Specifically, the digital ink server 20 classifies the collection of strokes G0 into one or more groups (for example, five groups G1 to G5) based on the order, position, and shape of the strokes specified by the stroke data, or on the relationship between the pen pressure of the electronic pen 50, etc. This classification provides classification data that describes the inclusion relationships between the groups and the stroke elements, or the inclusion relationships between the groups.

[0045] In step S08, the digital ink server 20 performs a process of assigning semantic attributes to the stroke groups G1 to G5 classified in step S07. Specifically, the digital ink server 20 estimates and assigns semantic attributes to each of the groups G1 to G5 using a classifier (e.g., a hierarchical neural network) that has undergone machine learning. This process results in classification data that describes semantic attributes consisting of type and value pairs for each classified group.

[0046] For example, the above-mentioned classifier is configured to input stroke feature quantities (e.g., coordinates of the start point, intermediate points, and end point, curvature, etc.) and output labels of semantic attributes. Examples of "types" include text (including types of languages), figures (including types of shapes), mathematical formulas, chemical formulas, lists, tables, etc. Examples of "values" include handwritten characters or character strings (e.g., "milk"), handwritten names of objects (e.g., "pencil" for an illustration of a pencil), etc.

[0047] FIG. 5A is a diagram showing the classification results of strokes. As can be seen from this diagram, the collection G0 includes the group G1, which is composed of four groups G2 to G5. FIG. 5B is a diagram showing the result of assigning semantic attributes to the groups classified in FIG. 5A. The type of group G1 is "LIST" and the value is "NULL". The type of group G2 is "LIST_ITEM" and the value is "milk". The type of group G3 is "LIST_ITEM" and the value is "bread". The type of group G4 is "LIST_ITEM" and the value is "eggs". The type of group G5 is "LIST_ITEM" and the value is "apples".

[0048] 3, the digital ink server 20 updates the pre-applied ink so as to add the classification data obtained in step S07 and the semantic data obtained in step S08. Hereinafter, the digital ink Ink to which the classification data and semantic data have been added may be referred to as "applied ink."

[0049] In step S10, the digital ink server 20 transmits the post-application ink updated in step S09 to the user terminal 40.

[0050] In step S11, the user terminal 40 receives the data transmitted from the digital ink server 20 in step S10, thereby acquiring the applied ink including the classification data and the semantic data.

[0051] In step S12, the user terminal 40 stores in the memory 47 the post-application ink acquired in step S11.

[0052] In step S13, the user terminal 40 confirms that the judgment condition regarding the classification of strokes is satisfied, and validates the instruction operation for at least one group. This "judgment condition" is, for example, that semantic data has been acquired from the digital ink server 20. In this case, the groups G2 to G5 whose semantic attribute values ​​are valid (other than NULL) are selected as targets for validation. Note that the validation of the instruction operation is withheld for the group G1 whose semantic attribute value is invalid (NULL) and whose meaning is not specified, and for one or more strokes that are not meaningful because they are in the middle of being written.

[0053] In step S14, the user terminal 40 displays the stroke groups G2 to G5 that were enabled in step S13 in a more emphasized manner than before they were enabled. This "emphasis display" means displaying the groups in a manner that is easily recognized by the user, and examples of this include changing the display color, enlarging the stroke width, and adding an instruction mark.

[0054] 6A and 6B are transition diagrams showing a first state change in the display of the user terminal 40. More specifically, FIG. 6A shows the display state before activation, and FIG. 6B shows the display state after activation. In the example of FIG. 6B, underlines are added to four locations of the stroke collection G0 that correspond to groups G2 to G5. This allows the user to know at a glance that the pointing operations for the four words "milk," "bread," "eggs," and "apples" have been activated.

[0055] This is how the first operation is completed by the digital ink processing system 10. The digital ink processing system 10 can start a second operation upon completion of the first operation.

[0056] In the above example, the instruction operation is enabled when the user terminal 40 acquires semantic data from the digital ink server 20, but the determination condition is not limited to this example. For example, the determination condition may be that the user terminal 40 itself acquires classification data or semantic data by analyzing the digital ink Ink, or that the timing for automatically or manually saving the digital ink Ink has arrived. In addition, the determination condition may be that the user terminal 40 accepts an operation to add a mark or annotation to at least a part of the collection G0 of strokes.

[0057] 7A and 7B are transition diagrams showing a second state change in the display of the user terminal 40. As shown in FIG. 7A, in a state in which none of the stroke collection G0 is enabled, the user performs a writing operation using the electronic pen 50 to encircle the word "eggs" with a throwing lasso (a so-called lasso). Then, a new window 60 is displayed along with the stroke collection G0, prompting the user to confirm whether or not "eggs" needs to be enabled. Here, by touching the [ON] button in the window 60, the display state of the user terminal 40 transitions from FIG. 7A to FIG. 7B.

[0058] As shown in Fig. 7B, a highlighter is added to one word (eggs) that corresponds to group G4 in the stroke collection G0. This allows the user to see at a glance that only the pointing operation for the word "eggs" has been enabled.

[0059] <Second action: Content quoting action> Next, a description will be given of a second operation, that is, a content quoting operation, of the digital ink processing system 10. This second operation is executed by the user terminal 40 (more specifically, the host processor 46) and the content server 30 (more specifically, the control unit 32) in cooperation with each other.

[0060] 8, the user terminal 40 accepts an operation (i.e., an instruction operation) of using the electronic pen 50 to point to any one of the enabled groups G2 to G5 among the collection G0 of strokes displayed on the touch panel display 41. This instruction operation may be, for example, an operation of pressing and holding the location of "eggs" (group G4) in FIG. 6B or 7B.

[0061] In step S22, the user terminal 40 reads the post-applied ink stored in the memory 47 and acquires the semantic attribute of the group G4 designated in step S21. Specifically, the user terminal 40 analyzes the classification data of the post-applied ink to identify the group ID corresponding to the designated position of the electronic pen 50. Then, the user terminal 40 analyzes the semantic data of the post-applied ink to identify the semantic attribute corresponding to the group ID.

[0062] In step S23, the user terminal 40 receives the data signal transmitted from the electronic pen 50 via the sensor electrode 43, thereby acquiring the pen ID of the electronic pen 50.

[0063] In step S24, the user terminal 40 transmits the meaning data acquired in step S22 to the content server 30 in a state where the meaning data is linked to the pen ID acquired in step S23. That is, the user terminal 40 requests the content server 30 to search for content.

[0064] In step S25, the content server 30 receives the data transmitted from the user terminal 40 in step S24, and thereby acquires the pen ID and the semantic attribute.

[0065] In step S26, the content server 30 searches for content using the semantic attribute and the pen ID acquired in step S25. Specifically, the content server 30 sequentially performs [1] a search for content using the pen ID, and [2] a search within the content using the semantic attribute. Here, the pen ID functions as identification information (hereinafter, usage ID) for determining whether or not the content is permitted to be used. This usage ID may be, in addition to the pen ID, identification information of the user terminal 40 (hereinafter, terminal ID), identification information of the user (hereinafter, user ID), or a combination of these.

[0066] 9A is a diagram showing an example of a data structure of the electronic pen information 62. The electronic pen information 62 shows the correspondence between a "pen ID" which is identification information of the electronic pen 50, a "school" to which the user belongs, and a "grade / class" corresponding to the user. This pen ID corresponds to a usage ID for managing the usage status of content. Examples of "school" include elementary schools, junior high schools, high schools, universities, and prep schools. Various grades or classes can be selected as the "grade / class" depending on the type of school.

[0067] 9B is a diagram showing an example of a data structure of the content management information 64. The content management information 64 indicates the correspondence between the "school" to which the user belongs, the "grade / class" corresponding to the user, and the "type of content" that can be used. The "school" and "grade / class" are basically defined in the same way as the electronic pen information 62. Examples of types of content include book titles, including textbooks, reference books, and workbooks, subject names, and publisher names.

[0068] The content server 30 reads out the electronic pen information 62 and the content management information 64, and compares them using "school" and "grade / class" as search keys to acquire at least one type of content corresponding to the pen ID. Then, the content server 30 uses various search methods to acquire information (hereinafter referred to as "related content C1") that matches or is similar to the "semantic attribute (value)" as the search key from among the contents to be searched.

[0069] In step S27 of FIG. 8, the content server 30 transmits the related content C1 obtained as a result of the search in step S26 to the user terminal 40 in a state in which the related content C1 is linked to the pen ID.

[0070] In step S28, the user terminal 40 receives the data transmitted from the content server 30 in step S27, thereby acquiring the related content C1 and the pen ID.

[0071] In step S29, the user terminal 40 temporarily stores in the memory 47 the related content C1 acquired in step S28.

[0072] In step S30, the user terminal 40 simultaneously displays on the touch panel display 41 the related content C1 obtained by the search together with the collection G0 of strokes.

[0073] 10A and 10B are transition diagrams showing a third state change in the display of the user terminal 40. More specifically, FIG. 10A shows the display state before related content C1 appears, and FIG. 10B shows the display state after related content C1 appears. As can be seen from both figures, by performing a pointing operation on "eggs", related content C1 is also displayed around "eggs" (here, at the lower right). This related content C1 consists of "eggs" written in English and "meaning: egg" written in Japanese.

[0074] The user can easily confirm that the Japanese translation of the English word "eggs" that is pointed to using the electronic pen 50 is "eggs (plural)." In this way, the learning support service provided by the digital ink processing system 10 can further improve the efficiency of learning.

[0075] By the way, the administrator of the electronic notebook may lend the electronic notebook to the user at school, but may prohibit the user from taking the electronic notebook outside the school. In this case, by associating the content usage ID with the pen ID, the user can receive the above-mentioned learning support service even at home.

[0076] 11A illustrates a case where the user terminal 40B is different from the user terminal 40A that performed handwritten input using the electronic pen 50. For example, the user can edit the digital ink Ink using the same electronic pen 50 that the user brought home from school. Alternatively, the digital ink processing system 10 may be configured to refer to a terminal ID included in the "device data" of the digital ink Ink, and to restrict or prohibit editing of the digital ink Ink using a user terminal 40B with a mismatched value.

[0077] 11B, when the user designates a corresponding area using the electronic pen 50, the user terminal 40B can display the related content C1 on the touch panel display 41, similarly to the case of the user terminal 40A. This allows the user to receive learning support services in various places by carrying his / her own electronic pen 50, even in a situation where it is difficult to freely move the user terminal 40A.

[0078] [Example of improved operation] An improved operation of the digital ink processing system 10 will now be described with reference to Figures 12A to 13B.

[0079] <First improvement example> It is assumed that as the user's learning progresses and his / her proficiency level increases, the information the user wants to know will gradually change. Therefore, the user terminal 40 may be configured to display different related contents C1, C2 according to the user's proficiency level. For example, in the sequence diagram of FIG. 8, an additional step S31 may be provided between steps S23 and S24.

[0080] In step S31 of Fig. 8, the user terminal 40 performs a search in the semantic data using the "value" of the semantic attribute acquired in step S22 (e.g., eggs) as a keyword. The target of this search is not limited to the digital ink Ink being edited, but may include digital ink Ink stored in its own memory 47 or the digital ink server 20. For example, the user terminal 40 counts the number of "values" that match or are close to the keyword, and calculates the number as the "degree of occurrence". After that, in steps S24 and S25, data including the degree of occurrence is transmitted and received between the user terminal 40 and the content server 30.

[0081] In step S26, the content server 30 searches for content using the pen ID, semantic attribute, and occurrence degree acquired in step S25. Specifically, the content server 30 sequentially performs the following steps: [1] searching for content using the pen ID, [2] selecting content using the occurrence degree, and [3] searching within the content using the semantic attribute.

[0082] FIG. 12A is a diagram showing an example of a data structure of the judgment table 66. The judgment table 66 shows the correspondence between the "number of occurrences" indicating the degree of occurrence and the "proficiency" of a typical user. In the example of this figure, the judgment table 66 describes the following judgment criteria: [1] if the number of occurrences is less than 9 times, the proficiency is low; [2] if the number of occurrences is 10 times or more but less than 20 times, the proficiency is medium; and [3] if the number of occurrences is 20 times or more, the proficiency is high. Note that in this figure, the number of occurrences is given as an example of the degree of occurrence, but the frequency of occurrence may be used in addition to or separately from this.

[0083] The content server 30 identifies the user's level of proficiency from the frequency of appearance using this determination table 66, and then selects content appropriate to the level of proficiency from among multiple types of content corresponding to the pen ID. For example, in the case of a user with a higher level of proficiency, it is assumed that related content C2 different from related content C1 shown in Fig. 10B is acquired. Thereafter, steps S27 to S29 are executed in the same manner as above.

[0084] In step S30, the user terminal 40 displays the related content C2 obtained by the search together with the collection of strokes G0 on the touch panel display 41. As shown in FIG. 12B, the related content C2 is displayed in a position surrounding "eggs" where the pointing operation was performed. This related content C2 is made up of a character string of an example sentence "Birds lay eggs." using the English word "eggs."

[0085] In this way, by displaying different related contents C1, C2 according to the occurrence frequency of semantic attributes for the same usage ID (here, pen ID), it is possible to present information suitable for individual users with different levels of proficiency. Note that, since there is a high correlation between the occurrence frequency and the user's interest level, it is effective to display the information in the same way as in the case of proficiency described above.

[0086] <Second improvement example> Since electronic notebooks are for personal use, it may not be desirable for others to be able to edit one's electronic notebook without permission. Therefore, the user terminal 40 may be configured to be able to change editing authority according to the pen IDs of the electronic pens 50A and 50B.

[0087] As shown in FIG. 13A, the pen ID (e.g., ID=002) of the electronic pen 50B recognized by the user terminal 40 is different from the pen ID (e.g., ID=001) of the electronic pen 50A used for handwritten input. In this case, the user terminal 40 displays a message 70 on the touch panel display 41 to the effect that editing of the digital ink Ink is restricted. With the message 70 displayed, the user terminal 40 prohibits or restricts editing of the digital ink Ink using the electronic pen 50B. Here, "prohibit" refers to an operation of not accepting operations related to editing. Also, "restrict" refers to an operation of making it impossible to execute some of the editing functions.

[0088] 13B, when the user designates the relevant area using the electronic pen 50B, the user terminal 40 displays the related content C1 on the touch panel display 41, similar to the case of the electronic pen 50A. This prevents unauthorized editing of one's electronic notebook, and allows anyone other than the user to receive the content quoting service.

[0089] [Learning support service billing flow] In the course of providing the above-mentioned learning support service, the following two services are involved: (1) a usage fee for the semantic interpretation service; and (2) a usage fee for the content citation service. The following describes the billing flow that takes place between the digital ink server 20, the content server 30, the user terminal 40, and the provider server 80.

[0090] Here, the business-side server 80 is a server managed by a content provider (e.g., a publisher). For the sake of convenience, the user terminal 40 is involved in the billing flow, but in addition to or separately from this, a school-side server (not shown) that manages each individual user terminal 40 may also be involved in this billing flow.

[0091] <Example 1> FIG. 14 is a schematic diagram showing a first example of a billing flow that occurs during the learning support service. [T11] The user terminal 40 accesses the digital ink server 20 every time the user uses the semantic interpretation service. [T12] The digital ink server 20 tallyes up the usage status of the semantic interpretation service for each user / each usage fee due date, and performs a charging process to charge the usage fee for the service in accordance with a predetermined charging rule. [T13] The digital ink server 20 notifies each user terminal 40 that the usage fee determined by the billing process will be charged. When the user receives this notification and pays the fee within the payment period, the user's payment of the usage fee is completed.

[0092] [T14] Meanwhile, the user terminal 40 accesses the content server 30 every time the user uses the content quoting service. [T15] The content server 30 tallyes up the usage status of the content associated with the content quotation service for each fee settlement date / user. [T16] The content server 30 notifies the provider's server 80 of the usage status of the content for each user. [T17] The provider's server 80 performs a charging process to charge the content usage fee according to a predetermined charging rule based on the results of the usage status calculation. [T18] The provider's server 80 notifies each user terminal 40 that the usage fee determined by the billing process will be charged. When the user receives this notification and pays the fee within the payment period, the user's payment of the usage fee is completed. [T19] The provider-side server 80 notifies the content server 30 that it will pay the service platform provision fee in accordance with a predetermined contract rule (for example, a pay-per-use system proportional to the content usage fee).

[0093] <Second Example> FIG. 15 is a schematic diagram showing a second example of the billing flow that occurs during the learning support service. [T21] The user terminal 40 accesses the digital ink server 20 every time the user terminal 40 uses the semantic interpretation service. [T22] The digital ink server 20 tallyes up the usage status of the semantic interpretation service for each user / each usage fee due date, and performs a charging process to charge the usage fee for the service according to a predetermined charging rule. [T23] The digital ink server 20 notifies each user terminal 40 that the usage fee determined by the billing process will be charged. When the user receives this notification and pays the fee within the payment period, the user's payment of the usage fee is completed.

[0094] [T24] Meanwhile, the user terminal 40 accesses the content server 30 every time the user uses the content quoting service. [T25] The content server 30 tallyes up the usage status of the content associated with the content quotation service for each user / each usage fee due date, and performs a charging process to charge the usage fee for the content in accordance with a predetermined charging rule. [T26] The content server 30 notifies each user terminal 40 that the usage fee determined by the charging process will be charged. When the user receives this notification and pays the fee within the payment period, the user's payment of the usage fee is completed. [T27] The content server 30 notifies the provider's server 80 that it will pay the amount obtained by deducting the service platform provision fee from the content usage fee.

[0095] [Variations] The present invention is not limited to the above-described embodiment, and can be freely modified without departing from the spirit and scope of the present invention. Alternatively, the respective configurations may be arbitrarily combined without causing any technical contradiction.

[0096] In the above embodiment, the digital ink processing system 10 is configured to provide a learning support service that supports education for students, but the type of service or target users are not limited to the above example.

[0097] In the above embodiment, the digital ink server 20 classifies strokes and assigns semantic attributes, but the user terminal 40 may be configured to be able to perform classification or both classification and assignment instead of the digital ink server 20. For example, the user terminal 40 may analyze the digital ink Ink by itself to acquire semantic attributes, or may acquire semantic attributes through a manual input operation by the user.

[0098] In the above embodiment, the digital ink with defined meaning data is used as an example, but the digital ink without defined meaning data may be used. In this case, the digital ink and the meaning data are linked to each other and managed, thereby realizing the same operation as the above embodiment.

[0099] [Summary of the embodiment] As described above, the digital ink processing system 10 includes the electronic pen 50 and the user terminal 40 configured to receive writing operations by the electronic pen 50 via the touch panel display 41 and generate digital ink Ink describing a collection G0 of strokes. The user terminal 40 enables a pointing operation of the electronic pen 50 for groups G2 to G5 of strokes that satisfy a determination condition related to stroke classification from among the collection G0 displayed on the touch panel display 41, while suspending the enablement of a pointing operation for a group G1 of strokes that does not satisfy the determination condition. Upon receiving the enabled pointing operation, the user terminal 40 searches for content related to semantic attributes of the specified groups G2 to G5 or requests the search from an external source, and displays related content C1, C2 obtained by the search together with the collection G0 on the touch panel display 41.

[0100] Furthermore, according to the digital ink processing method and program, the user terminal 40 executes a step (S13) of enabling a pointing operation of the electronic pen 50 to point to groups G2 to G5 of strokes that satisfy a judgment condition for stroke classification from among the collection G0 of strokes displayed on the touch panel display 41, while suspending the enabling of the pointing operation for group G1 of strokes that does not satisfy the judgment condition, and, upon receiving the enabled pointing operation (S21), searching for content related to the semantic attributes assigned to groups G2 to G5 or requesting such a search from an external source (S24), and displaying the related content C1, C2 obtained by the search together with the collection G0 on the touch panel display 41 (S30).

[0101] In this way, only the pointing operation of the electronic pen 50 that points to the stroke groups G2 to G5 that satisfy the judgment conditions regarding the classification of the strokes is enabled, so that the search for content is permitted only for the groups G2 to G5 that are classified so as to have a high probability of being meaningful as a whole by appropriately setting the judgment conditions. This makes it possible to present the user with meaningful search results and highly relevant information when searching using digital ink Ink.

[0102] Furthermore, the user terminal 40 may display the groups G2 to G5 for which the instruction operation has been validated in a more emphasized manner than before the validation, which allows the user to easily visually recognize that the instruction operation has been validated and its position.

[0103] The determination condition may be that a semantic attribute has been assigned to each of the stroke groups G2 to G5, which allows a search to be performed using the assigned semantic attribute as a search key, thereby providing a search result that is more meaningful to the user.

[0104] Furthermore, the digital ink server 20 may assign semantic attributes to the groups G2 to G5 by analyzing the digital ink Ink transmitted from the user terminal 40, and the determination condition may be that semantic data indicating the semantic attributes is acquired from the digital ink server 20. By the digital ink server 20 assigning the semantic attributes instead of the user terminal 40, the analysis load on the user terminal 40 is reduced.

[0105] The judgment condition may be that a user operation to add a mark or annotation to groups G2 to G5 has been received. This allows the user to specify a group of strokes that the user found meaningful through the mark or annotation, making it easier to obtain more accurate semantic attributes.

[0106] In addition, the content server 30 may store content in association with a usage ID, and the user terminal 40 may transmit data including the usage ID and semantic attributes to the content server 30 and request a search, thereby obtaining related content C1, C2 that is related to the semantic attributes and is permitted for use from the content server 30.

[0107] In addition, the user terminal 40 may acquire different related contents C1, C2 according to the occurrence frequency of semantic attributes for the same usage ID. By considering the occurrence frequency that is highly correlated with the user's proficiency or interest level, it is possible to select and present information suitable for users with different proficiency or interest levels.

[0108] The usage ID may be a pen ID for identifying the electronic pen 50. This makes it possible to associate available content with the electronic pen 50, and by using the same electronic pen 50, the same related content C1, C2 is presented without depending on the user terminals 40A, 40B.

[0109] Furthermore, when digital ink Ink is generated using electronic pen 50A to which a first pen ID has been assigned, and electronic pen 50B to which a second pen ID different from the first pen ID has been assigned is used, the user terminal 40 may prohibit or limit editing of the digital ink Ink, while accepting instruction operations by electronic pen 50B. The digital ink Ink generated by the user is prevented from being edited without permission, and the same related contents C1 and C2 as those for the user are presented to persons other than the user.

[0110] The content server 30 may also perform billing processing for content users and content providers according to the usage status of the content. This allows content management and usage fee management to be performed in an integrated manner by a single device. [Explanation of symbols]

[0111] 10 digital ink processing system, 20 digital ink server, 30 content server, 40, 40A, 40B user terminal, 41 touch panel display, 50, 50A, 50B electronic pen, 80 business server, C1, C2 related content, G0 collection, G1 to G5 groups, Ink digital ink

Claims

1. a processing device that generates strokes in response to input by an electronic pen; A processor is included. The processor, Controlling a display to display said collection of strokes; A search is performed for content related to the semantic attributes output by interpreting the stroke clusters using a machine learning classifier, or the search is requested from an external server; A processing device that controls the display of the content obtained by the search on the display.

2. The semantic attribute includes a type of the stroke group. The processing device of claim 1 .

3. The types of stroke groups include text, graphics, mathematical expressions, chemical expressions, lists, and tables. The processing device according to claim 2 .

4. The semantic attribute includes a value of the cluster of the strokes. The processing device of claim 1 .

5. The stroke group value includes a character, a character string, or a name of an object. The processing device according to claim 4 .

6. The processor, and obtaining the semantic attributes from a digital ink server that uses the machine learning classifier to interpret the stroke clusters. The processing device of claim 1 .

7. the semantic attributes are stored in a data structure for the stroke chunk, The processing device of claim 1 .

8. The machine-learned classifier is a neural network. The processing device of claim 1 .

9. A display method for displaying strokes according to input by an electronic pen, a display displays the collection of strokes; A processor searches for content related to semantic attributes output by interpreting the stroke clusters using a machine learning classifier, or requests such a search from an external server; A display method, wherein the display displays the content obtained by the search.

10. The semantic attribute includes a type of the stroke group. The display method according to claim 9.

11. The types of stroke groups include text, graphics, mathematical expressions, chemical expressions, lists, and tables. The display method according to claim 10.

12. The semantic attribute includes a value of the cluster of the strokes. The display method according to claim 9.

13. The stroke group value includes a character, a character string, or a name of an object. The display method according to claim 12.

14. The processor, and obtaining the semantic attributes from a digital ink server that uses the machine learning classifier to interpret the stroke clusters. The display method according to claim 9.

15. the semantic attributes are stored in a data structure for the stroke chunk, The display method according to claim 9.

16. The machine-learned classifier is a neural network. The display method according to claim 9.

Images