Processing apparatus and display method

Abstract

To provide a processing device and a display method capable of presenting a user with information having a significant retrieval result and high relevance in retrieving content.SOLUTION: A processor included in a processing device controls a display so as to display a group of strokes, retrieves content related to a semantic attribute outputted by using a discriminator undergoing machine learning to interpret the group of strokes or requests the retrieval to an external server, and controls the content acquired by the retrieval so as to be displayed in the display.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a processing apparatus and a display method. [Background Art]

[0002] Patent Document 1 discloses a technique for performing a search in which at least a part of information displayed on a screen of an information processing apparatus including a personal computer is set as a search key, and displaying the search result in a pop-up on the screen. [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2015-114955 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] In digital ink that describes an aggregate of strokes, meaning is often generated only by a group of strokes. However, when an attempt is made to perform a search by designating a portion corresponding to a search key during editing of digital ink, a meaningless search key or a search key with incorrect meaning may be set, resulting in a problem that appropriate search results cannot be obtained.

[0005] An object of the present invention is to provide a digital ink processing system, method and program capable of presenting information with meaningful and highly relevant search results to a user when performing a search using digital ink. [Means for Solving the Problem]

[0006] The first digital ink processing system of the present invention comprises an electronic pen and a tablet that generates digital ink for writing strokes in response to input from the electronic pen via a display, wherein the tablet has a processor, the processor enables instruction operations of the electronic pen for the stroke, and after enabling instruction operations of the electronic pen, searches for content related to the semantic attributes of the instructed stroke or requests such search from an external server, and controls the display to show the content obtained by the search together with the stroke.

[0007] Furthermore, the processor may control the display to show the stroke, for which the instruction operation has been activated, with greater emphasis compared to before activation.

[0008] Furthermore, the processor may enable instruction operations of the electronic pen for strokes to which the semantic attributes have been assigned.

[0009] Furthermore, the system may further include a digital ink server that analyzes the digital ink transmitted from the tablet and assigns the semantic attributes to the strokes, and the processor may enable instruction operations of the electronic pen for strokes for which semantic data indicating the semantic attributes has been obtained from the digital ink server.

[0010] Furthermore, the processor may enable the electronic pen's instruction operation in response to strokes that have been operated by the user to add marks or annotations.

[0011] Furthermore, the system may further include a content server that stores content in association with user IDs, and the processor may obtain content from the content server that is related to the semantic attributes and whose use is permitted by sending data including the user ID and the semantic attributes to the content server to request a search.

[0012] Furthermore, the processor may acquire different content depending on the frequency of occurrence of the semantic attribute in the same user ID.

[0013] Furthermore, the user ID may also be a pen ID used to identify the electronic pen.

[0014] Furthermore, when digital ink is generated using a first electronic pen assigned a first pen ID, and a second electronic pen assigned a second pen ID different from the first pen ID is used, the processor may prohibit or restrict editing of the digital ink, while accepting the instruction operation.

[0015] Furthermore, the content server may charge users and providers of content based on their usage of the content.

[0016] The second digital ink processing method of the present invention involves a processor that generates digital ink for writing a stroke in response to input from an electronic pen via a display, performing the steps of: enabling instruction operations of the electronic pen for the stroke; searching for content related to the semantic attributes of the instructed stroke or requesting such search from an external server after enabling the instruction operations of the electronic pen; and controlling the display to show the content obtained by the search together with the stroke.

[0017] The third digital ink processing program of the present invention causes a processor that generates digital ink to describe a stroke in response to input from an electronic pen via a display to perform the following steps: enable instruction operations of the electronic pen for the stroke; after enabling instruction operations of the electronic pen, search for content related to the semantic attributes of the instructed stroke or request such search from an external server; and control the program to display the content obtained by the search together with the stroke on the display. [Effects of the Invention]

[0018] According to the present invention, when performing a search using digital ink, it is possible to present a user with information having meaningful and highly relevant search results. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] [Figure 1] FIG. 1 is an overall configuration diagram of a digital ink processing system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram of a digital ink server, a content server, and a user terminal shown in FIG. 1. [Figure 3] FIG. 3 is a sequence diagram related to a semantic interpretation operation of the digital ink processing system. [Figure 4A] FIG. 4 is a schematic diagram visualizing an aggregate of strokes. [Figure 4B] FIG. 5 is a diagram showing an example of a data structure of digital ink. [Figure 5A] FIG. 5A is a diagram showing classification results of strokes. [Figure 5B] FIG. 5B is a diagram showing semantic attribute assignment results for the groups classified in FIG. 5A. [Figure 6A] FIG. 6A is a transition diagram showing a first state change in display on a user terminal. [Figure 6B] FIG. 6B is a transition diagram showing the first state change in display on the user terminal. [Figure 7A] FIG. 7A is a transition diagram showing a second state change in display on the user terminal. [Figure 7B] FIG. 7B is a transition diagram showing the second state change in display on the user terminal. [Figure 8] FIG. 8 is a sequence diagram related to a content citation operation of the digital ink processing system. [Figure 9A] FIG. 9 is a diagram showing an example of a data structure included in electronic pen information. [Figure 9B] FIG. 10 is a diagram showing an example of a data structure included in content management information. [Figure 10A] FIG. 11 is a transition diagram showing a third state change in display on the user terminal. [Figure 10B] This is a transition diagram showing the third state change in the user terminal display. [Figure 11A] This figure shows an example of the operation of a user terminal different from the one used for handwritten input. [Figure 11B] This figure shows an example of the operation of a user terminal different from the one used for handwritten input. [Figure 12A] This figure shows a first example of improved operation of the digital ink processing system. [Figure 12B] This figure shows a first example of improved operation of the digital ink processing system. [Figure 13A] This figure shows a second example of improved operation of the digital ink processing system. [Figure 13B] This figure shows a second example of improved operation of the digital ink processing system. [Figure 14] This is a schematic diagram illustrating the first example of the billing flow that occurs when providing learning support services. [Figure 15] This is a schematic diagram illustrating a second example of the billing flow that occurs when providing learning support services. [Modes for carrying out the invention]

[0020] [Configuration of Digital Ink Processing System 10] Figure 1 is an overall configuration diagram of a digital ink processing system 10 in one embodiment of the present invention. Figure 2 is a block diagram of the digital ink server 20, content server 30, and user terminal 40 shown in Figure 1. The digital ink processing system 10 is configured to provide a "learning support service" that enables users, such as students, to learn efficiently using electronic notebooks. Specifically, the digital ink processing system 10 comprises a digital ink server 20, a content server 30, one or more user terminals 40, and one or more electronic pens 50.

[0021] Examples of digital ink (or ink data) data formats, or so-called "ink description languages," include WILL (Wacom Ink Layer Language), InkML (Ink Markup Language), and ISF (Ink Serialized Format). By describing this digital ink using the JSON (JavaScript® Object Notation) data structure format, data exchange between various software and programming languages ​​becomes easier.

[0022] The digital ink server 20 is a computer that provides overall control over the processing of digital ink, and may be either cloud-based or on-premise. Here, the digital ink server 20 is illustrated as a single computer, but instead, the digital ink server 20 may be a group of computers forming a distributed system.

[0023] The digital ink server 20 specifically comprises a communication unit 21, a control unit 22, and a storage unit 23. The communication unit 21 is an interface for sending and receiving electrical signals to and from external devices. The control unit 22 is composed of a processing unit including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The control unit 22 functions as the "processing engine" for the digital ink by reading and executing programs stored in the storage unit 23. The storage unit 23 is composed of a non-transient, computer-readable storage medium, such as a hard disk drive (HDD) or a solid-state drive (SSD). As a result, the storage unit 23 stores various data handled by the digital ink server 20.

[0024] The content server 30 is a computer that provides overall control over the provision of educational content, including textbooks and dictionaries, and may be either cloud-based or on-premise. Here, the content server 30 is illustrated as a single computer, but instead, the content server 30 may be a group of computers forming a distributed system.

[0025] The content server 30 specifically comprises a communication unit 31, a control unit 32, and a storage unit 33. The communication unit 31 is an interface for sending and receiving electrical signals to and from external devices. The control unit 32 is composed of a processing unit including a CPU and a GPU. The control unit 32 is configured to search for and provide content suitable for the user by reading and executing programs stored in the storage unit 33.

[0026] The user terminal 40 is a device that functions as an electronic notebook and can be, for example, a tablet, a smartphone, or a personal computer. Specifically, the user terminal 40 includes a touch panel display 41, a touch IC (Integrated Circuit) 44, a display driver IC 45, a host processor 46, memory 47, and a communication module 48.

[0027] The touch panel display 41 comprises a display panel 42 capable of visually outputting content, and sensor electrodes 43 arranged on top of the display screen of the display panel 42. The display panel 42 is capable of displaying monochrome or color images and may be, for example, a liquid crystal panel or an organic EL (Electro-Luminescence) panel. The sensor electrodes 43 consist of a plurality of X-line electrodes for detecting the position on the X axis in an XY sensor coordinate system and a plurality of Y-line electrodes for detecting the position on the Y axis, arranged in a planar manner.

[0028] The touch IC 44 is an integrated circuit that controls the driving of the sensor electrode 43. The touch IC 44 drives the sensor electrode 43 based on control signals supplied from the host processor 46. As a result, the touch IC 44 performs a "pen detection function" to detect the state of the electronic pen 50 and a "touch detection function" to detect touches made by the user's finger or the like.

[0029] The display driver IC 45 is an integrated circuit that controls the driving of the display panel 42. The display driver IC 45 drives the display panel 42 based on frame-by-frame image signals supplied from the host processor 46. As a result, an image is displayed within the display area of ​​the display panel 42. This image may include not only writing lines made by the user using the electronic pen 50, but also application windows, icons, and cursors.

[0030] The host processor 46 is composed of a processing unit including an MPU (Micro-Processing Unit) and a CPU. The host processor 46 reads and executes a program stored in memory 47, performing processes such as generating digital ink using data from the touch IC 44, and ink regeneration to display the drawing content indicated by the digital ink.

[0031] Memory 47 consists of a non-transient and computer-readable storage medium. Here, the computer-readable storage medium is a portable medium such as a magneto-optical disk, ROM, CD-ROM, or flash memory, or a storage device such as an HDD or SSD built into a computer system.

[0032] The communication module 48 is configured to send and receive electrical signals to and from external devices using wired or wireless communication. This allows, for example, the user terminal 40 to send and receive digital ink to and from the digital ink server 20 via the network NW, and to receive related content C1 and C2 from the content server 30.

[0033] The electronic pen 50 is a pen-shaped pointing device configured to communicate unidirectionally or bidirectionally via electrostatic coupling formed with the user terminal 40. The user can draw pictures or write characters on the user terminal 40 by holding the electronic pen 50 and moving it while pressing the pen tip against the touch surface of the touch panel display 41. This electronic pen 50 is, for example, an active electrostatic coupling (AES) or electromagnetic induction (EMR) stylus.

[0034] [Operation of Digital Ink Processing System 10] The digital ink processing system 10 in this embodiment is configured as described above. Next, the first and second operations of this digital ink processing system 10 will be explained, mainly with reference to the sequence diagrams in Figures 3 and 8. The "first operation" refers to the operation related to "semantic interpretation," which interprets a group of strokes and automatically assigns semantic attributes (or meaning attributes). The "second operation" refers to the operation related to "content citation," which promptly quotes and displays content related to the semantic attributes.

[0035] <First action: Meaning interpretation action> First, the first operation of the digital ink processing system 10, namely the semantic interpretation operation, will be described. This first operation is performed through the cooperation of the user terminal 40 (more specifically, the host processor 46) and the digital ink server 20 (more specifically, the control unit 22).

[0036] In step S01 of Figure 3, the user terminal 40 performs authentication using the identification information of the electronic pen 50 (hereinafter referred to as the pen ID). If the pen ID received from the electronic pen 50 is already registered, the user terminal 40 allows editing of the digital ink with the electronic pen 50. On the other hand, if the pen ID is not registered, the user terminal 40 does not allow editing of the digital ink with the electronic pen 50.

[0037] In step S02, the user terminal 40 accepts input using the electronic pen 50, which was permitted to be edited in step S01, that is, various writing operations by the user. These writing operations include, for example, adding, deleting, and changing strokes, selecting icons, and adding marks and annotations.

[0038] In step S03, the user terminal 40 periodically or irregularly updates the digital ink to reflect the writing operations received in step S02. An example of the former is when a predetermined amount of time has elapsed since the most recent update. An example of the latter is when the user has given an instruction to update.

[0039] In step S04, the user terminal 40 uses the data obtained through the writing operation in step S02 to generate, for example, digital ink in WILL (Ver.3.0) format.

[0040] Figure 4A is a schematic diagram visualizing the stroke collection G0. Here, collection G0 represents a handwritten shopping list, consisting of the items "milk," "bread," "eggs," and "apples" from top to bottom. Figure 4B is a diagram showing an example of the data structure of digital ink. Digital ink has a data structure consisting of [1] document metadata, [2] ink semantics, [3] devices, [4] stroke data, [5] groups, and [6] context data arranged in sequence.

[0041] When digital ink is generated, document metadata, device data, stroke data, and context data are already determined, but semantic data and classification data have not yet been determined. In other words, this digital ink has not undergone (1) stroke classification and (2) semantic attribute assignment, as described later. Hereinafter, digital ink with blank classification and semantic data may be referred to as "pre-assignment ink."

[0042] In step S05, the user terminal 40 sends the pre-assigned ink generated in step S04, linked to the pen ID, to the digital ink server 20.

[0043] In step S06, the digital ink server 20 receives the data sent from the user terminal 40 in step S05 and obtains the ink to be assigned and the pen ID, respectively.

[0044] In step S07, the digital ink server 20 analyzes the stroke data contained in the pre-application ink acquired in step S06 and classifies the strokes. Specifically, the digital ink server 20 classifies the stroke collection G0 into one or more groups (for example, five groups from G1 to G5) based on the order, position, and shape of the strokes identified by the stroke data, or the relationship of the pressure applied by the electronic pen 50. This classification provides classification data that describes the inclusion relationships between groups and stroke elements, or between groups themselves.

[0045] In step S08, the digital ink server 20 performs a process to assign semantic attributes to the stroke groups G1 to G5 classified in step S07. Specifically, the digital ink server 20 uses a machine learning classifier (e.g., a hierarchical neural network) to estimate and assign semantic attributes to each of the groups G1 to G5. This process yields classification data for each classified group, describing semantic attributes consisting of type-value pairs.

[0046] For example, the classifier described above is configured to take stroke features (e.g., coordinates of the start point, intermediate points, and end point, curvature, etc.) as input and output labels of semantic attributes. Examples of "types" include text (including language type), figures (including shape type), mathematical formulas, chemical formulas, lists, and tables. Examples of "values" include handwritten letters or strings of characters (e.g., "milk") and handwritten names of objects (e.g., "pencil" for an illustration of a pencil).

[0047] Figure 5A shows the results of the stroke classification. As can be seen from this figure, aggregate G0 encompasses group G1, and group G1 consists of four groups G2 to G5. Figure 5B shows the results of assigning semantic attributes to the groups classified in Figure 5A. The type of group G1 is "LIST" and the value is "NULL". The type of group G2 is "LIST_ITEM" and the value is "milk". The type of group G3 is "LIST_ITEM" and the value is "bread". The type of group G4 is "LIST_ITEM" and the value is "eggs". The type of group G5 is "LIST_ITEM" and the value is "apples".

[0048] In step S09 of Figure 3, the digital ink server 20 updates the pre-application ink to add the classification data obtained in step S07 and the semantic data obtained in step S08. Hereinafter, the digital ink with the added classification and semantic data may be referred to as "post-application ink."

[0049] In step S10, the digital ink server 20 sends the updated ink to the user terminal 40 in step S09.

[0050] In step S11, the user terminal 40 receives the data transmitted from the digital ink server 20 in step S10 and acquires the assigned ink containing classification data and semantic data.

[0051] In step S12, the user terminal 40 stores the granted ink acquired in step S11 in the memory 47.

[0052] In step S13, the user terminal 40 confirms that the criteria for classifying strokes are met and enables the instruction operation for at least one group. This "criteria" is, for example, that semantic data has been obtained from the digital ink server 20. In this case, groups G2 to G5, whose semantic attribute values ​​are valid (not NULL), are selected as targets for activation. Note that the activation of the instruction operation is withheld for group G1, whose semantic attribute values ​​are invalid (NULL) and whose meaning has not been identified, and for one or more strokes that do not have meaning because they are in the middle of writing.

[0053] In step S14, the user terminal 40 displays stroke groups G2 to G5, which were enabled in step S13, with increased emphasis compared to before they were enabled. This "emphasis" means displaying them in a manner that is easily visible to the user, and examples include changing the display color, increasing the stroke width, and adding indicator marks.

[0054] Figures 6A and 6B are transition diagrams showing the first state change in the display of the user terminal 40. More specifically, Figure 6A shows the display state before activation, and Figure 6B shows the display state after activation. In the example in Figure 6B, underlines have been added to four locations in the stroke set G0 corresponding to groups G2 to G5. This allows the user to understand at a glance that instruction operations for the four words "milk," "bread," "eggs," and "apples" have been activated.

[0055] As described above, the first operation of the digital ink processing system 10 is completed. The digital ink processing system 10 can then start the second operation upon completion of this first operation.

[0056] In the example described above, the instruction operation is activated when the user terminal 40 acquires semantic data from the digital ink server 20, but the determination conditions are not limited to this example. For example, the determination conditions may be that the user terminal 40 itself acquires classification data or semantic data by analyzing the digital ink, or that the time has come to save the digital ink automatically or manually. Alternatively, the determination conditions may be that the user terminal 40 accepts an operation to add a mark or annotation to at least a portion of the stroke collection G0.

[0057] Figures 7A and 7B are transition diagrams showing the second state change in the display of the user terminal 40. As shown in Figure 7A, with no part of the stroke set G0 activated, the user performs a writing operation using the electronic pen 50 to draw a lasso around the word "eggs". Then, a new window 60 appears, which asks whether or not to activate "eggs" along with the stroke set G0. By touching the [ON] button in window 60, the display state of the user terminal 40 transitions from Figure 7A to Figure 7B.

[0058] As shown in Figure 7B, a fluorescent marker has been added to one word (eggs) corresponding to group G4 within the stroke set G0. This allows the user to see at a glance that only the instruction operation for the word "eggs" is enabled.

[0059] <Second action: Content citation action> Next, the second operation of the digital ink processing system 10, namely the content retrieval operation, will be described. This second operation is performed through the cooperation of the user terminal 40 (more specifically, the host processor 46) and the content server 30 (more specifically, the control unit 32).

[0060] In step S21 of Figure 8, the user terminal 40 receives an operation (i.e., an instruction operation) using the electronic pen 50 to indicate one of the enabled groups G2 to G5 from the set of strokes G0 displayed on the touch panel display 41. This instruction operation may be, for example, a long press on the "eggs" (group G4) area in Figure 6B or Figure 7B.

[0061] In step S22, the user terminal 40 reads the applied ink stored in memory 47 and obtains the semantic attributes of group G4 as instructed in step S21. Specifically, the user terminal 40 identifies the group ID corresponding to the instruction position of the electronic pen 50 by analyzing the classification data of the applied ink. Then, the user terminal 40 identifies the semantic attributes corresponding to the group ID by analyzing the semantic data of the applied ink.

[0062] In step S23, the user terminal 40 receives the data signal transmitted from the electronic pen 50 using the sensor electrode 43, thereby obtaining the pen ID of the electronic pen 50.

[0063] In step S24, the user terminal 40 sends the semantic data acquired in step S22, linked to the pen ID acquired in step S23, to the content server 30. In other words, the user terminal 40 requests the content server 30 to search for content.

[0064] In step S25, the content server 30 receives the data sent from the user terminal 40 in step S24 and obtains the pen ID and semantic attributes, respectively.

[0065] In step S26, the content server 30 searches for content using the semantic attributes and pen ID obtained in step S25. Specifically, the content server 30 sequentially performs [1] a search for content using the pen ID and [2] a search within the content using the semantic attributes. Here, the pen ID functions as identification information (hereinafter referred to as the usage ID) for determining whether the content is permitted to be used. This usage ID may be the pen ID, the identification information of the user terminal 40 (hereinafter referred to as the terminal ID), the user's identification information (hereinafter referred to as the user ID), or a combination thereof.

[0066] Figure 9A shows an example of the data structure of the electronic pen information 62. The electronic pen information 62 shows the correspondence between the "pen ID," which is the identification information of the electronic pen 50, the "school" to which the user belongs, and the "grade / class" to which the user belongs. This pen ID corresponds to a user ID for managing the usage status of the content. Examples of "schools" include elementary schools, junior high schools, high schools, universities, and cram schools. Depending on the type of school, various grades or classes may be selected for the "grade / class."

[0067] Figure 9B shows an example of the data structure of content management information 64. Content management information 64 shows the correspondence between the "school" to which the user belongs, the "grade / class" corresponding to the user, and the "types of content" that can be used. "School" and "grade / class" are basically defined in the same way as electronic pen information 62. Examples of types of content include book titles (including textbooks, reference books, and workbooks), subject names, and publisher names.

[0068] The content server 30 reads the electronic pen information 62 and the content management information 64, and retrieves at least one type of content corresponding to the pen ID by matching it with "school" and "grade / class" as search keys. The content server 30 then uses various search methods to retrieve information (hereinafter referred to as "related content C1") from the search target content that matches or approximates the "semantic attribute (value)" used as the search key.

[0069] In step S27 of Figure 8, the content server 30 sends the related content C1, which is the search result from step 26, to the user terminal 40, with the pen ID associated with it.

[0070] In step S28, the user terminal 40 receives the data sent from the content server 30 in step S27 and obtains the associated content C1 and pen ID, respectively.

[0071] In step S29, the user terminal 40 temporarily stores the related content C1 acquired in step S28 in memory 47.

[0072] In step S30, the user terminal 40 simultaneously displays the related content C1 obtained through the search, along with the stroke collection G0, on the touch panel display 41.

[0073] Figures 10A and 10B are transition diagrams showing the third state change in the display of the user terminal 40. More specifically, Figure 10A shows the display state before the related content C1 appears, and Figure 10B shows the display state after the related content C1 appears. As can be seen from both figures, by performing an instruction operation on the "eggs" section, related content C1 is displayed around "eggs" (in this case, in the lower right). This related content C1 consists of "eggs" written in English and "Meaning: Eggs" written in Japanese.

[0074] The user can easily confirm that the Japanese translation of the English word "eggs" indicated using the electronic pen 50 is "tamago (plural)." In this way, the learning efficiency is further enhanced through the learning support service provided by the digital ink processing system 10.

[0075] Incidentally, administrators of electronic notebooks sometimes lend them to users within the school while prohibiting them from taking them outside the school. Therefore, by associating the content user ID with the pen ID, users can access the aforementioned learning support services even when they are at home.

[0076] Figure 11A shows a case where user terminal 40B is different from user terminal 40A, which performed handwriting input using the electronic pen 50. For example, a user can edit the digital ink using the same electronic pen 50 that they brought home from school. Alternatively, the digital ink processing system 10 may be configured to refer to the terminal ID included in the "device data" of the digital ink and restrict or prohibit editing of the digital ink using user terminal 40B whose value does not match.

[0077] On the other hand, as shown in Figure 11B, when a user points to a relevant area using the electronic pen 50, the user terminal 40B can display the relevant content C1 on the touch panel display 41, just as in the case of user terminal 40A. This allows users to receive learning support services in various locations by carrying their electronic pen 50, even in situations where it is difficult to freely move the user terminal 40A.

[0078] [Examples of improved operation] The following describes examples of improvements to the operation of the digital ink processing system 10, with reference to Figures 12A to 13B.

[0079] <First improvement example> As a user's learning progresses and their proficiency increases, it is expected that the information they want to know will gradually change. Therefore, the user terminal 40 may be configured to display different related content C1 and C2 according to the user's proficiency level. For example, in the sequence diagram of Figure 8, an additional step S31 may be provided between steps S23 and S24.

[0080] In step S31 of Figure 8, the user terminal 40 performs a search within the semantic data using the "value" (e.g., eggs) of the semantic attribute obtained in step S22 as a keyword. The target of this search is not limited to the digital ink being edited, but may also include the digital ink stored in its own memory 47 or the digital ink server 20. For example, the user terminal 40 counts the number of "values" that match or approximate the keyword and calculates the number of occurrences as the "frequency of occurrence". Subsequently, in steps S24 and S25, data including the frequency of occurrence is sent and received between the user terminal 40 and the content server 30.

[0081] In step S26, the content server 30 searches for content using the pen ID, semantic attributes, and frequency of occurrence obtained in step S25. Specifically, the content server 30 sequentially performs [1] a search for content using the pen ID, [2] selection of content using the frequency of occurrence, and [3] a search within the content using the semantic attributes.

[0082] Figure 12A shows an example of the data structure of the judgment table 66. The judgment table 66 shows the correspondence between "frequency of occurrence," which indicates the degree of occurrence, and the "proficiency level" of a typical user. In the example shown in this figure, the judgment table 66 describes the following criteria: [1] if the frequency of occurrence is less than 9, the proficiency level is low; [2] if the frequency of occurrence is 10 or more but less than 20, the proficiency level is moderate; and [3] if the frequency of occurrence is 20 or more, the proficiency level is high. In this figure, the frequency of occurrence is given as an example of the degree of occurrence, but the frequency of occurrence may be used in conjunction with or separately from this.

[0083] The content server 30 uses this determination table 66 to identify the user's proficiency level based on the frequency of appearance, and then selects content from among multiple types of content corresponding to the pen ID that is appropriate for that proficiency level. For example, suppose that a user with a higher proficiency level obtains related content C2, which is different from related content C1 shown in Figure 10B. Then, steps S27 to S29 are executed in the same manner as described above.

[0084] In step S30, the user terminal 40 displays the related content C2 obtained through the search on the touch panel display 41 together with the stroke set G0. As shown in Figure 12B, the related content C2 is displayed at a position around "eggs" where the instruction operation was performed. This related content C2 consists of the string "Birds lay eggs." which is an example sentence using the English word "eggs".

[0085] In this way, by displaying different related content C1 and C2 according to the frequency of occurrence of semantic attributes within the same user ID (in this case, the pen ID), it is possible to present information tailored to individual users with different levels of proficiency. Furthermore, since there is a high correlation between frequency of occurrence and the user's level of interest, displaying the content in the same way as described above for proficiency levels is also effective.

[0086] <Second improvement example> Since electronic notebooks are for personal use, it is undesirable for others to be able to edit one's electronic notebook without permission. Therefore, the user terminal 40 may be configured to allow editing permissions to be changed according to the pen IDs of the electronic pens 50A and 50B.

[0087] As shown in Figure 13A, the pen ID of the electronic pen 50B recognized by the user terminal 40 (for example, ID=002) is different from the pen ID of the electronic pen 50A used for handwriting input (for example, ID=001). In this case, the user terminal 40 displays a message 70 on the touch panel display 41 indicating that editing of the digital ink is restricted. When message 70 is displayed, the user terminal 40 prohibits or restricts editing of the digital ink using the electronic pen 50B. Here, "prohibit" means an action that does not accept editing operations. Also, "restrict" means an action that prevents the execution of some editing functions.

[0088] On the other hand, as shown in Figure 13B, when the user indicates the relevant area using the electronic pen 50B, the user terminal 40 displays the relevant content C1 on the touch panel display 41, similar to the case with the electronic pen 50A. This prevents unauthorized editing of the user's electronic notebook and allows others to access the content citation service.

[0089] [Billing flow for learning support services] Incidentally, the provision of the above-mentioned learning support services will involve the exchange of [1] usage fees for the semantic interpretation service and [2] usage fees for the content citation service. The following describes the billing flow between the digital ink server 20, the content server 30, the user terminal 40, and the service provider's server 80.

[0090] Here, the service provider server 80 is a server managed by the content provider (for example, a publisher). For the sake of explanation, the user terminal 40 is shown as being involved in the billing flow, but a school server (not shown) that manages each individual user terminal 40 may also be involved in this billing flow, either in conjunction with or separately from the user terminal 40.

[0091] <Example 1> Figure 14 is a schematic diagram showing the first example of the billing flow that occurs when providing learning support services. [T11] The user terminal 40 accesses the digital ink server 20 each time it uses the semantic interpretation service. [T12] The digital ink server 20 aggregates the usage status of the semantic interpretation service for each billing cycle / user and performs billing processing to charge the usage fee for the service in accordance with the prescribed billing rules. [T13] The digital ink server 20 notifies each user terminal 40 that it will bill them for the usage fee determined by the billing process. When the user who receives this notification makes the payment within the payment period, the user's payment of the usage fee is completed.

[0092] [T14] Meanwhile, the user terminal 40 accesses the content server 30 each time it uses the content citation service. [T15] The content server 30 aggregates the usage status of content related to the content citation service for each billing cycle and each user. [T16] The content server 30 notifies the service provider server 80 of the content usage status for each user. [T17] The service provider's server 80 performs billing processing to charge content usage fees according to predetermined billing rules based on the aggregated usage status. [T18] The service provider's server 80 notifies each user terminal 40 that it will bill them for the usage fee determined by the billing process. When the user receives this notification and makes the payment within the payment period, the user's payment of the usage fee is completed. [T19] The service provider's server 80 notifies the content server 30 that it will pay the service platform fee in accordance with the prescribed contract rules (for example, a usage-based system proportional to the usage fee of the content).

[0093] <Example 2> Figure 15 is a schematic diagram showing a second example of the billing flow that occurs when providing learning support services. [T21] The user terminal 40 accesses the digital ink server 20 each time it uses the semantic interpretation service. [T22] The digital ink server 20 aggregates the usage status of the semantic interpretation service for each billing cycle / user and performs billing processing to charge the service usage fee according to the prescribed billing rules. [T23] The digital ink server 20 notifies each user terminal 40 that it will bill them for the usage fee determined by the billing process. When the user who receives this notification makes the payment within the payment period, the user's payment of the usage fee is completed.

[0094] [T24] Meanwhile, the user terminal 40 accesses the content server 30 each time it uses the content citation service. [T25] The content server 30 aggregates the usage status of content associated with the content citation service for each billing cycle / user and performs billing processing to charge the usage fee for the content in accordance with the prescribed billing rules. [T26] The content server 30 notifies each user terminal 40 that it will bill them for the usage fee determined by the billing process. When the user receives this notification and makes the payment within the payment period, the user's payment of the usage fee is completed. [T27] The content server 30 notifies the service provider server 80 that it will pay the amount obtained by subtracting the service platform provision fee from the content usage fee.

[0095] [Differentiation] It should be noted that the present invention is not limited to the embodiments described above, and can be freely modified without departing from the spirit of the invention. Alternatively, each component may be combined in any way that does not create a technical inconsistency.

[0096] In the embodiment described above, the digital ink processing system 10 is configured to provide learning support services that assist in student education, but the type of service or target users is not limited to the example described above.

[0097] In the embodiment described above, the digital ink server 20 performs stroke classification and assigns semantic attributes, but the user terminal 40 may be configured to perform classification, or both classification and assignment, instead of the digital ink server 20. For example, the user terminal 40 may obtain semantic attributes by analyzing the digital ink itself, or it may obtain semantic attributes through manual input operations by the user.

[0098] In the embodiments described above, a digital ink with defined semantic data was used as an example, but a digital ink without defined semantic data may also be used. In this case, the same operation as in the embodiments described above can be achieved by linking and managing the digital ink and semantic data together.

[0099] [Summary of Embodiments] As described above, the digital ink processing system 10 includes an electronic pen 50 and a user terminal 40 configured to receive writing operations from the electronic pen 50 via a touch panel display 41 and generate digital ink that describes a collection of strokes G0. The user terminal 40 activates instruction operations of the electronic pen 50 for groups G2 to G5 of strokes that satisfy the criteria for classifying strokes from the collection G0 displayed on the touch panel display 41, while suspending the activation of instruction operations for group G1 of strokes that do not satisfy the criteria. Upon receiving an activated instruction operation, the user terminal 40 searches for content related to the semantic attributes of the specified groups G2 to G5 or requests such a search from an external source, and displays the related content C1 and C2 obtained from the search together with the collection G0 on the touch panel display 41.

[0100] Furthermore, according to the digital ink processing method and program, the user terminal 40 performs the following steps: (S13) activates the instruction operation of the electronic pen 50 to indicate groups G2 to G5 of strokes that satisfy the determination conditions for stroke classification from among the collection of strokes G0 displayed on the touch panel display 41, while suspending the activation of the instruction operation for group G1 of strokes that do not satisfy the determination conditions; and (S21) when the activated instruction operation is received, the user terminal 40 searches for content related to the semantic attributes assigned to groups G2 to G5 or requests such search from an external source (S24), and displays the related content C1 and C2 obtained by the search together with the collection of strokes G0 on the touch panel display 41 (S30).

[0101] In this way, only the instruction operation of the electronic pen 50 that indicates stroke groups G2 to G5 that satisfy the criteria for stroke classification is enabled. By appropriately setting the criteria, content searching is permitted only for groups G2 to G5 that have been classified in a way that increases the likelihood of them forming a meaningful whole. This makes it possible to present users with meaningful and highly relevant information in the search results when using digital ink.

[0102] Furthermore, the user terminal 40 may display groups G2 to G5, for which instruction operations have been enabled, with greater emphasis compared to before activation. This makes it easier for the user to visually recognize that instruction operations have been enabled and to recognize their location.

[0103] Furthermore, the determination criterion may also be that semantic attributes have been assigned to stroke groups G2 to G5. This allows the assigned semantic attributes to be used as search keys, resulting in more meaningful search results for the user.

[0104] Furthermore, the digital ink server 20 may assign semantic attributes to groups G2 to G5 by analyzing the digital ink transmitted from the user terminal 40, and the determination condition may be that semantic data indicating the semantic attributes has been obtained from the digital ink server 20. By having the digital ink server 20 assign semantic attributes on behalf of the user terminal 40, the analysis burden on the user terminal 40 is reduced.

[0105] Furthermore, the determination criterion may also be that the user has performed an action to assign a mark or annotation to groups G2 to G5. This makes it possible to specify a group of strokes that the user who performed the action found meaningful through the mark or annotation, making it easier to obtain more accurate semantic attributes.

[0106] Alternatively, the content server 30 may store content in association with user IDs, and the user terminal 40 may request a search by sending data including the user ID and semantic attributes to the content server 30, thereby obtaining related content C1 and C2 from the content server 30 that are associated with semantic attributes and whose use is permitted.

[0107] Furthermore, the user terminal 40 may acquire different related content C1 and C2 depending on the frequency of occurrence of semantic attributes within the same user ID. By considering the frequency of occurrence that has a high correlation with the user's proficiency or level of interest, it is possible to select and present information suitable for users with different levels of proficiency or interest.

[0108] Furthermore, the user ID may also be a pen ID used to identify the electronic pen 50. This makes it possible to associate available content with the electronic pen 50, and by using the same electronic pen 50, the same related content C1 and C2 will be presented regardless of whether the user terminals 40A and 40B are used.

[0109] Furthermore, if digital ink is generated using electronic pen 50A assigned a first pen ID, and electronic pen 50B assigned a second pen ID different from the first pen ID is used, the user terminal 40 may prohibit or restrict editing of the digital ink, while accepting instructions from electronic pen 50B. This prevents unauthorized editing of the digital ink generated by the user, and also ensures that the same related content C1 and C2 are presented to persons other than the user as they are to the user.

[0110] Furthermore, the content server 30 may charge content users and providers based on their usage of the content. This allows for centralized management of content and usage fees using a single device. [Explanation of Symbols]

[0111] 10 Digital ink processing system, 20 Digital ink server, 30 Content server, 40, 40A, 40B User terminal, 41 Touch panel display, 50, 50A, 50B Electronic pen, 80 Operator server, C1, C2 Related content, G0 Group, G1-G5 Group, Ink Digital ink

Claims

1. This is a processing device that generates strokes in response to input from an electronic pen. It has a processor, The aforementioned processor, Control the display to show the aforementioned stroke clusters, By interpreting the aforementioned stroke clusters using a machine learning-based classifier, content related to the output semantic attributes is searched for, or a search is requested from an external server. A processing device that controls the display to show different content on the display according to the frequency of occurrence calculated by counting the values ​​of the semantic attributes of the content obtained by the search.

2. The aforementioned semantic attribute is characterized by including the type of the stroke grouping, The apparatus according to claim 1.

3. The types of stroke clusters are characterized by including text, figures, mathematical formulas, chemical formulas, lists, and tables. The apparatus according to claim 2.

4. The aforementioned semantic attribute is characterized by including the value of the stroke grouping, The apparatus according to claim 1.

5. The value of the stroke group is characterized by including characters, strings of characters, and object names. The apparatus according to claim 4.

6. The aforementioned processor, The method is characterized by obtaining the semantic attributes output from a digital ink server that interprets the stroke groupings using the aforementioned machine learning classifier. The apparatus according to claim 1.

7. The aforementioned semantic attributes are stored in a data structure for the stroke group, The apparatus according to claim 1.

8. The aforementioned machine learning-based classifier is characterized by being a neural network. The apparatus according to claim 1.

9. This is a display method that displays strokes in response to input from an electronic pen. The display shows the grouping of the strokes, The processor interprets the group of strokes using a machine learning-based classifier, searches for content related to the output semantic attributes, or requests such search from an external server. A display method wherein the display shows different content based on the frequency of occurrence calculated by counting the values ​​of the semantic attributes, which are content obtained by the search.

10. The aforementioned semantic attribute is characterized by including the type of the stroke grouping, The display method according to claim 9.

11. The types of stroke clusters are characterized by including text, figures, mathematical formulas, chemical formulas, lists, and tables. The display method according to claim 10.

12. The aforementioned semantic attribute is characterized by including the value of the stroke grouping, The display method according to claim 9.

13. The value of the stroke group is characterized by including characters, strings of characters, and object names. The display method according to claim 12.

14. The aforementioned processor, The method is characterized by obtaining the semantic attributes output from a digital ink server that interprets the stroke groupings using the aforementioned machine learning classifier. The display method according to claim 9.

15. The aforementioned semantic attributes are stored in a data structure for the stroke group, The display method according to claim 9.

16. The aforementioned machine learning-based classifier is characterized by being a neural network. The display method according to claim 9.

Images