Content evaluation system, method, and program

Abstract

The present invention relates to a content evaluation system, a method, and a program. A content evaluation system (10) comprises: a first calculation unit (42) that calculates, from acquired content data, a plurality of types of time-series features (54) indicating the temporal changes of a plurality of types of features relating to the creation process of handwritten content (60, 100); a second calculation unit (44) that uses two or more of the plurality of types of time-series features (54) to calculate an integrated feature (56) relating to the creation process of handwritten content (60, 100); and a content evaluation unit (46) that evaluates the handwritten content (60, 100) on the basis of the integrated feature (56) and outputs evaluation result information (58) indicating the result of evaluating the handwritten content (60, 100).

Description

Content rating system, method, and program

[0001] The present invention relates to a content rating system, method, and program.

[0002] Techniques for evaluating the drawing content of handwritten content have been known for some time, and these techniques can be used to automatically grade, for example, written answers.

[0003] Patent document 1 discloses a text evaluation device that acquires a first logical relationship that holds between first events that make up a first sentence, extracts a second logical relationship that holds between second events that make up a second sentence from the second sentence that is the subject of evaluation, and evaluates the second sentence by comparing the first logical relationship with the second logical relationship.

[0004] Patent No. 6717387

[0005] However, while the device disclosed in Patent Document 1 can evaluate academic ability by calculating scores and correct answer rates using the content of the answer sheet after it has been completed, it has the problem of not being able to evaluate various items related to the process of creating the answer sheet.

[0006] The present invention has been made in consideration of such problems, and its purpose is to provide a content evaluation system, method, and program that can evaluate handwritten content more precisely than when evaluation is simply performed using the finished drawing content.

[0007] A content evaluation system according to a first aspect of the present invention includes an electronic pen; a data generation unit that generates content data representing handwritten content by receiving writing operations using the electronic pen; a data acquisition unit that acquires the content data generated by the data generation unit; a first calculation unit that calculates, from the content data acquired by the data acquisition unit, multiple types of time-series features that indicate changes over time in multiple types of features related to the process of creating the handwritten content; a second calculation unit that calculates an integrated feature related to the process of creating the handwritten content using two or more types of time-series features among the multiple types of time-series features calculated by the first calculation unit; and a content evaluation unit that evaluates the handwritten content based on the integrated feature calculated by the second calculation unit and outputs evaluation result information that indicates the evaluation result of the handwritten content.

[0008] In the content evaluation method of the second aspect of the present invention, one or more computers execute the following steps: an acquisition step of acquiring content data indicating handwritten content; a first calculation step of calculating, from the acquired content data, multiple types of time-series features indicating temporal changes in multiple types of features related to the creation process of the handwritten content; a second calculation step of calculating, using two or more types of time-series features among the calculated multiple types of time-series features, an integrated feature related to the creation process of the handwritten content; and an evaluation step of evaluating the handwritten content based on the calculated integrated feature and outputting evaluation result information indicating the evaluation result of the handwritten content.

[0009] The content evaluation program of the third aspect of the present invention causes one or more computers to execute the following steps: an acquisition step of acquiring content data indicating handwritten content; a first calculation step of calculating, from the acquired content data, multiple types of time-series features indicating temporal changes in multiple types of features related to the creation process of the handwritten content; a second calculation step of calculating, using two or more types of time-series features from the calculated multiple types of time-series features, an integrated feature related to the creation process of the handwritten content; and an evaluation step of evaluating the handwritten content based on the calculated integrated feature and outputting evaluation result information indicating the evaluation result of the handwritten content.

[0010] According to the present invention, handwritten content can be evaluated more precisely than when evaluation is simply performed using the content of the finished drawing.

[0011] 1 is an overall configuration diagram of a content evaluation system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the configuration of a server device in FIG. 1. FIG. 3 is a flowchart showing a first operation of the content evaluation system of FIG. 1. FIG. 4 is a diagram showing a first example of handwritten content created using the user device of FIG. 1. FIG. 5 is a diagram showing an example of a data structure of content data included in evaluation data. FIG. 6 is a diagram showing an example of time-series features calculated by a first calculation unit. FIG. 7 is a diagram showing a first example of a network structure of a learning device. FIG. 8 is a diagram showing an example of a method for determining motivation to learn. FIG. 9 is a diagram showing an example of evaluation result information displayed together with handwritten content. FIG. 10 is a diagram showing changes in the state of symbols according to a student's motivation to learn. FIG. 11 is a diagram showing a second example of content created using the user device of FIG. 1. FIG. 12 is a diagram showing a second example of a network structure of a learning device. FIG. 13 is a diagram showing an example of a method for classifying styles.

[0012] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. To facilitate understanding of the description, the same components in the drawings will be denoted by the same reference numerals as much as possible, and duplicated descriptions will be omitted.

[0013] [Configuration of Content Rating System 10] <Overall Configuration> Fig. 1 is a diagram showing the overall configuration of a content rating system 10 according to one embodiment of the present invention. The content rating system 10 is provided to provide a "content rating service" for rating handwritten digital content. Specifically, this content rating system 10 includes one or more user devices 12, one or more electronic pens 14, and a server device 16 (corresponding to a "content rating device").

[0014] The user device 12 is a computer owned by a user who uses the content evaluation service (e.g., [1] a student taking online lessons, or [2] a content creator), and is configured, for example, as a tablet, smartphone, personal computer, etc. Each user device 12 is configured to be able to exchange various data, including evaluation data D1 and display data D2, which will be described later, with the server device 16 via the network NT.

[0015] Specifically, the user device 12 includes a processor 21 (corresponding to a “data generation unit”), a memory 22, a communication unit 23, a display unit 24 (corresponding to a “display unit”), and a touch sensor 25.

[0016] The processor 21 is configured with an arithmetic processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an MPU (Micro-Processing Unit). The processor 21 reads out programs and data stored in the memory 22, and performs a generation process for generating digital ink by accepting writing operations using the electronic pen 14, and a rendering process for displaying handwritten content indicated by the digital ink.

[0017] The memory 22 stores programs and data necessary for the processor 21 to control each component. The memory 22 is composed of a non-transitory, computer-readable storage medium. Here, the computer-readable storage medium is composed of [1] a storage device such as a hard disk drive (HDD) or solid state drive (SSD) built into the computer system, or [2] a portable medium such as a magneto-optical disk, ROM, CD-ROM, or flash memory.

[0018] The communication unit 23 has a communication function for performing wired or wireless communication with an external device, which allows the user device 12 to exchange various data, including the evaluation data D1 or the display data D2, with the server device 16, for example.

[0019] The display unit 24 can visibly display content including images or videos, and is configured with, for example, a liquid crystal panel, an organic electroluminescence (EL) panel, or electronic paper. By making the display unit 24 flexible, the user can perform various writing operations while keeping the touch surface of the user device 12 curved or bent.

[0020] The touch sensor 25 is a capacitance type sensor having a plurality of sensor electrodes arranged in a plane. The touch sensor 25 includes, for example, a plurality of X-line electrodes for detecting the position on the X axis of a sensor coordinate system and a plurality of Y-line electrodes for detecting the position on the Y axis. Note that the touch sensor 25 may be a self-capacitance type sensor in which block-shaped electrodes are arranged in a two-dimensional lattice pattern, instead of the mutual capacitance type sensor described above.

[0021] The electronic pen 14 is a pen-shaped pointing device configured to be capable of one-way or two-way communication with the user device 12. The electronic pen 14 is, for example, an active electrostatic coupling (AES) or electromagnetic induction (EMR) stylus. A student as a user can write pictures, letters, etc. on the user device 12 by holding the electronic pen 14 and moving it while pressing the tip of the pen against the touch surface of the user device 12.

[0022] The server device 16 is a computer that performs overall control over the evaluation of handwritten content, and may be either a cloud-based or on-premise type. Although the server device 16 is illustrated as a single computer, the server device 16 may instead be a group of computers that form a distributed system.

[0023] <Block Diagram of Server Device 16> Fig. 2 is a block diagram showing an example of the configuration of the server device 16 in Fig. 1. Specifically, the server device 16 includes a communication unit 30, a control unit 32, and a storage unit 34.

[0024] The communication unit 30 is an interface for transmitting and receiving electrical signals to and from an external device, which allows the server device 16 to obtain the evaluation data D1 from the user device 12 and provide the display data D2 generated by the server device 16 to the user device 12.

[0025] The control unit 32 is configured with one or more processors (such as a CPU or a GPU). The control unit 32 reads and executes the programs and data stored in the storage unit 34, thereby functioning as a data acquisition unit 40, a first calculation unit 42, a second calculation unit 44, a content evaluation unit 46, and a display instruction unit 48.

[0026] The data acquisition unit 40 acquires various data (hereinafter referred to as "evaluation data D1") related to the handwritten content to be evaluated. The data acquisition unit 40 may acquire the evaluation data D1 from an external device via communication, or may acquire the evaluation data D1 by reading it from the storage unit 34. The acquisition timing may be after the creation of the handwritten content is completed, or may be during the creation of the handwritten content. In the latter case, the data acquisition unit 40 automatically acquires the evaluation data D1 in response to receiving an explicit instruction operation (e.g., tapping a button) or an implicit instruction operation (e.g., writing strokes) from the creator.

[0027] The first calculation unit 42 calculates time-series feature quantities 54 indicating time changes of multiple types of feature quantities related to the creation process of handwritten content from the evaluation data D1 acquired by the data acquisition unit 40. During this calculation, the first calculation unit 42 performs processes such as [1] a "removal process" that removes data unnecessary for calculating the time-series feature quantities 54, [2] a "correspondence process" that associates each feature quantity with time, [3] a "truncation process" that trims blank time as necessary, and [4] a "normalization process" that normalizes feature quantities and time.

[0028] For example, when the handwritten content is an answer 60 (FIG. 4) to a given question, the two or more types of time-series feature quantities 54 include a first time-series feature quantity indicating a change over time in the moving speed of the electronic pen 14 corresponding to the strokes, and a second time-series feature quantity indicating a change over time in the writing pressure of the electronic pen 14 corresponding to the strokes. Also, when the handwritten content is an artwork 100 (FIG. 11), the two or more types of time-series feature quantities 54 include a first time-series feature quantity indicating a change over time in the position corresponding to the strokes, and a second time-series feature quantity indicating a change over time in the writing pressure of the electronic pen 14 corresponding to the strokes.

[0029] The second calculation unit 44 calculates an integrated feature 56 related to the creation process of the handwritten content using two or more types of time-series feature 54 among the multiple types of time-series feature 54 calculated by the first calculation unit 42. The calculation of the integrated feature 56 uses one or more learning devices 50 that have undergone machine learning. The learning device 50 is, for example, a neural network computing device in which a set of learning parameters is preset. The set of learning parameters is a collection of learning parameters for specifying the computation rules of the learning device 50. The learning parameters may include, for example, a coefficient describing the activation function of the computation unit, a weighting coefficient corresponding to the strength of synaptic connection, the number of computation units constituting each layer, the number of intermediate layers, etc.

[0030] The content evaluation unit 46 evaluates the handwritten content based on the integrated feature 56 calculated by the second calculation unit 44, and outputs information on the evaluation result of the handwritten content (i.e., “evaluation result information 58”). The evaluation processing method differs depending on the type of handwritten content.

[0031] When evaluating the answer sheet 60 (FIG. 4), examples of evaluation items include [1] the degree of completion of the answer sheet 60, [2] the tendency of understanding, and [3] the degree of effort. Here, the "degree of completion" may be a continuous value such as a score, or a discrete value such as a rank. Examples of the "tendency of understanding" include strengths, weaknesses, and the pattern of creating the answer sheet 60. Examples of the "degree of effort" include the motivation to study and the degree of concentration in creating the answer sheet 60.

[0032] When evaluating artwork 100 ( FIG. 11 ), examples of evaluation items include: [1] the style of artwork 100; [2] the habits of the creator; [3] the creator's psychological state; and [4] the external environment at the time of creation. Here, "style" refers to the creator's personality or ideas as expressed in the handwritten content. Examples of "habits" include color use, stroke drawing tendencies, device usage tendencies, and the degree of operational errors. Examples of "psychological state" include emotions including joy, anger, sadness, and pleasure, as well as various states such as drowsiness, relaxation, and tension. Examples of "external environment" include the surrounding brightness, temperature, weather, and season.

[0033] The content evaluation unit 46 may also calculate a similarity between the integrated feature 56 corresponding to the content to be evaluated (i.e., the first integrated feature) and the integrated feature 56 corresponding to the authentic content (i.e., the second integrated feature), and determine the authenticity of the handwritten content based on this similarity. Various indices, including, for example, a norm and a correlation coefficient, are used to calculate this similarity.

[0034] The display instruction unit 48 instructs the display of the evaluation result information 58 output by the content evaluation unit 46. The manner of this "instruction" includes not only "display control" for displaying on an output device (not shown) provided in the server device 16, but also "transmission" to an external device such as the user device 12 in FIG. 1 .

[0035] The storage unit 34 stores programs and data necessary for the control unit 32 to control each component. The storage unit 34 is non-transitory and is configured from a computer-readable storage medium. Here, the computer-readable storage medium is configured from [1] a storage device such as an HDD or SSD built into the computer system, or [2] a portable medium such as a magneto-optical disk, ROM, CD-ROM, or flash memory.

[0036] In the example of FIG. 2, a database related to handwritten content (hereinafter referred to as a content DB 52) is constructed in the storage unit 34, and multiple types of time-series features 54, integrated features 56, and evaluation result information 58 are stored.

[0037] The evaluation data D1 includes "content data" which is a collection of content elements that make up the handwritten content, and "related data" which includes various information related to the creation of the handwritten content.

[0038] The content data may be, for example, ink data (or digital ink) that represents the process of creating handwritten content. Examples of "ink description languages" for describing digital ink include WILL (Wacom Ink Layer Language), InkML (Ink Markup Language), and ISF (Ink Serialized Format). The handwritten content may be, for example, [1] written works such as answers, reports, and memos, or [2] artwork such as paintings, illustrations, and characters.

[0039] Examples of related data include: (1) creator information, including the identification information and attributes of the creator of the handwritten content; (2) "device driver setting conditions," including the resolution, size, and type of the display unit 24, the detection performance and type of the touch sensor 25, and the shape of the pen pressure curve; (3) "drawing application setting conditions," including the type of handwritten content, color information for the color palette and brush, and visual effect settings; (4) "creator operation history," which is stored sequentially throughout the execution of the drawing application; (5) "biometric data," which indicates the creator's biometric signals at the time the handwritten content was created; or (6) "environmental data," which indicates the state of the external environment at the time the handwritten content was created.

[0040] The time-series feature 54 indicates changes over time in feature amounts relating to the creation process of handwritten content, and is stored in association with identification information of the handwritten content or the creator. Specifically, the time-series feature 54 is a collection of data pairs indicating the correspondence between multiple types of feature amounts and time. The time may be real time including date and time, elapsed time from the start of creation, or the order in which content elements were generated or edited (e.g., the order in which strokes were written).

[0041] Regarding the evaluation of the answer sheet 60, examples of feature quantities used to evaluate the "degree of completion" include the position and color of strokes. Examples of feature quantities used to evaluate the "tendency of understanding" include the position and color of strokes, the movement speed of the electronic pen 14, and rewriting patterns. Examples of feature quantities used to evaluate the "degree of effort" include the position and color of strokes, the writing pressure, tilt angle, and movement speed of the electronic pen 14, and the student's pulse rate, heart rate, and grip pressure.

[0042] Regarding the evaluation of the artwork 100, examples of feature quantities used to evaluate "style" include the position and color of strokes, the pressure, tilt angle, and movement speed of the electronic pen 14. Examples of feature quantities used to evaluate "habits" include the drafting method, the color application method, color selection tendency, redrawing pattern, the hover position of the electronic pen 14, and the type of tool used. Examples of feature quantities used to evaluate "mental state" include pulse rate or heart rate, grip pressure, etc. Examples of feature quantities used to evaluate "external environment" include location, external sound intensity, illuminance, temperature, humidity, etc.

[0043] The integrated feature 56 is a feature related to the process of creating handwritten content, and is calculated using two or more types of time-series feature 54. For example, by integrating the time-series feature 54 related to at least two of [1] the position corresponding to the stroke, [2] the movement speed of the electronic pen 14, and [3] the writing pressure of the electronic pen 14, an integrated feature 56 for evaluating the motivation or concentration level of the creator who created the answer sheet 60 can be obtained. Alternatively, by integrating the time-series feature 54 related to at least two of [1] the position corresponding to the stroke, [2] the color, and [3] the writing pressure of the electronic pen 14, an integrated feature 56 for evaluating the style or authenticity of the artwork 100 can be obtained.

[0044] The evaluation result information 58 includes the evaluation results of the handwritten content by the content evaluation unit 46. Examples of the evaluation results include [1] the results of individual evaluations including classification categories, scores, states, etc., and [2] the results of comparative evaluations including similarity and authenticity determinations. The evaluation result information 58 is information derived from the above-mentioned evaluation results, and may include, for example, visible information for providing awareness to the creator of the handwritten content (hereinafter referred to as "awareness information"). Examples of awareness information include alternative expressions that abstract or obscure the content of the evaluation items (for example, symbols indicating the strength of characteristics, words with high similarity, etc.).

[0045] [Operation of Content Evaluation System 10] The content evaluation system 10 in this embodiment is configured as described above. Next, the operation of evaluating handwritten content by the content evaluation system 10 will be described with reference to FIGS.

[0046] <First Example: Evaluation of Learning Motivation> Fig. 3 is a flowchart showing a first operation of the content evaluation system 10 of Fig. 1. This "first operation" refers to, for example, an operation for evaluating and presenting a student's "learning motivation" during online learning.

[0047] In step SP10 of FIG. 3, the data acquisition unit 40 of the server device 16 acquires evaluation data D1 used to evaluate handwritten content, specifically, at least content data, and, if necessary, related data.

[0048] 4 is a diagram showing a first example of handwritten content created using the user device 12 of FIG. 1. In this example, the handwritten content shows an answer 60 to a mathematical problem asking for the length of the hypotenuse of a right-angled triangle. The answer 60 includes a content element 62 showing a shape, a content element 64 showing a formula related to Pythagoras' theorem, and a content element 66 showing the calculation process.

[0049] 5 is a diagram showing an example of the data structure of content data included in the evaluation data D1. In this example, the content data is digital ink. The digital ink has a data structure in which [1] document metadata, [2] ink semantics, [3] device data, [4] stroke data, [5] group data, and [6] context data are sequentially arranged.

[0050] The stroke data 68 is data for describing individual strokes that make up handwritten content, and indicates the shape of the strokes and the order in which the strokes are written. As can be seen from FIG. 5 , one stroke is described by multiple point data items arranged sequentially within the <trace> tag. Each point data item consists of at least a designated position (X coordinate, Y coordinate) and is separated by a delimiter such as a comma. For convenience of illustration, only point data indicating the start and end points of the stroke are shown, and point data indicating multiple intermediate points are omitted. In addition to the designated positions described above, this point data may also include the order in which the strokes were created or edited, the pressure and posture of the electronic pen 14, and the like.

[0051] 3, the first calculation unit 42 performs a process of associating content elements with time using the evaluation data D1 acquired in step SP10. Content elements include, for example, strokes, shapes, text characters, and coloring. For example, if a timestamp is provided for each user operation, the first calculation unit 42 associates the content element corresponding to the user operation with the real time specified by the timestamp. Alternatively, if an index indicating the stroke order for each content element is assigned, the first calculation unit 42 associates the content element with the index.

[0052] In step SP14, the first calculation unit 42 calculates time-series feature quantities 54 indicating changes over time in feature quantities related to the content creation process, using the data sets associated in step SP12. Specifically, the first calculation unit 42 calculates the multiple types of time-series feature quantities 54 by aggregating the multiple types of feature quantities by time.

[0053] 6 is a diagram showing an example of the time-series feature 54 calculated by the first calculation unit 42. The horizontal axis of the graph represents normalized time (unit: dimensionless), and the vertical axis of the graph represents normalized feature (unit: dimensionless). Here, the normalized time is the normalized elapsed time from the point in time when creation of the handwritten content started (i.e., the start point), and corresponds to the degree of completion of the handwritten content.

[0054] Specifically, the normalized time is defined so that the value corresponding to the start time is "0" and the value corresponding to the time when the creation of the handwritten content is completed (i.e., the end time) is "1." In this way, by keeping the length of time required to create the handwritten content (i.e., the time range) constant, it becomes easier to capture the characteristics of the handwritten content more accurately. Specifically, when the feature is an 8-bit color value (0 to 255) in the CIE RGB color system, the normalized feature is a value normalized to the range of [0, 1] by dividing the R value, G value, and B value by the maximum value 255. In this way, by keeping the range of the feature for the content element constant, it becomes easier to capture the characteristics of the content more accurately.

[0055] 3, the second calculation unit 44 calculates a motivation feature 56A (one aspect of the integrated feature 56) for evaluating the student's motivation to learn, based on two or more types of time-series feature 54 among the multiple types of time-series feature 54 calculated in step SP14. A learning device 50A is used to quantify the motivation to learn.

[0056] FIG. 7 is a diagram showing a first example of a network structure of a learning device 50A. The learning device 50A is composed of an autoencoder for detecting anomalies in data. Specifically, this learning device 50A is a hierarchical neural network computing device composed of an input layer 81, an intermediate layer 82, and an output layer 83. The input layer 81 is composed of N computation units for inputting each value of a feature quantity. The intermediate layer 82 is composed of M computation units (here, 1≦M<N). The output layer 83 is composed of the same number of computation units as the input layer 81 (i.e., N). In this network configuration, the input layer 81 and intermediate layer 82 perform a "dimensionality reduction function," and the intermediate layer 82 and output layer 83 perform a "dimensionality restoration function."

[0057] The learning parameters constituting the learning device 50A are optimized by so-called "unsupervised learning." In the example shown in the figure, the data set used for machine learning is composed of the movement feature of the electronic pen 14, the writing pressure feature of the electronic pen 14, and normalized time. This data set is, for example, a collection of data collected when students (specific or unspecified number of students) who are highly motivated to learn actually create answer sheets 60.

[0058] Meanwhile, a subtractor 84 connected to the input layer 81 and the output layer 83 is provided outside the learning device 50A. The subtractor 84 calculates the error between the input feature set and the output feature set (hereinafter also referred to as the "input / output error"). This input / output error may be, for example, either an L1 error or an L2 error. Through this series of calculations, a motivation feature 58A indicating the degree of decline in learning motivation is calculated.

[0059] 3, the second calculation unit 44 supplies the motivation feature 56A calculated in step SP16 to the storage unit 34 (FIG. 2) together with creator information of the handwritten content (here, the answer sheet 60 in FIG. 4). As a result, the motivation feature 56A is stored in the content DB 52 in association with the creator.

[0060] In step SP20, the content evaluation unit 46 performs a desired evaluation process on the motivation feature 56A saved in step SP18, as necessary, and outputs evaluation result information 58 including the evaluation result of the handwritten content. In the first operation example, the content evaluation unit 46 evaluates the learning motivation of the student who created the corresponding answer sheet 60 by performing a judgment process on the motivation feature 56A.

[0061] FIG. 8 is a diagram showing an example of a method for determining learning motivation. The axis of the graph indicates the magnitude of the input / output error indicated by the motivation feature 56A. [Condition A] If the input / output error is equal to or greater than 0 and less than E1, the content evaluation unit 46 determines that the student's learning motivation is "high." [Condition B] If the input / output error is equal to or greater than E1 and less than E2, the content evaluation unit 46 determines that the student's learning motivation is "medium." [Condition C] If the input / output error is equal to or greater than E2, the content evaluation unit 46 determines that the student's learning motivation is "low."

[0062] 3, the display instruction unit 48 instructs the display of the evaluation result information 58 obtained in step SP20. Specifically, the display instruction unit 48 transmits communication data including the evaluation result information 58 to an external device (e.g., the user device 12). This allows the student to visually check the evaluation results of the answer sheet 60 that is being created or has been created via the display unit 24 (FIG. 1) of his or her own user device 12.

[0063] 9 is a diagram showing an example of evaluation result information 58 displayed together with handwritten content. The display unit 24 of the user device 12 is provided with a display area 90 for displaying the drawing contents of the answer sheet 60 as handwritten content. An information presentation field 92 for displaying the evaluation result information 58 is provided in the lower left corner of this display area 90. A "flame" symbol 94, which is one aspect of the evaluation result information 58, is arranged in the information presentation field 92.

[0064] FIG. 10 shows the state changes of symbols 94 to 96 according to the student's motivation to learn. Within the information presentation field 92, one of symbols 94, 95, or 96 representing a flame is arranged. The symbols 94 to 96 have different forms depending on the student's level of motivation to learn. For example, when the student's motivation to learn is "high," the symbol 94 representing a flame with a strong flame is selected. When the student's motivation to learn is "moderate," the symbol 95 representing a flame with a medium flame is selected. When the student's motivation to learn is "low," the symbol 96 representing a flame with a weak flame is selected. In this way, the user device 12 visualizes and displays the symbols 94 to 96, which are one aspect of the evaluation result information 58, along with the answer sheet 60, which is one aspect of handwritten content, thereby making the student, who created the answer sheet 60, aware of his or her motivation to learn.

[0065] In this way, the content evaluation system 10 completes the first operation shown in the flowchart of Fig. 3. For example, by starting execution of this flowchart when the creation of the answer 60 is completed, the content evaluation system 10 can perform evaluation throughout the entire process of creating the answer 60. On the other hand, by starting execution of this flowchart while the answer 60 is being created, the content evaluation system 10 can perform evaluation in real time.

[0066] <Second Operation: Evaluating Artwork Style> Next, the second operation of the content evaluation system 10 will be described with reference to Figures 11 to 13. This "second operation" refers to, for example, an operation for evaluating and presenting the "style" of handwritten content. The content evaluation system 10 basically performs the second operation in accordance with the flowchart shown in Figure 3, similar to the first operation. Below, the operations at steps SP16, SP18, and SP20 in Figure 3 will be described in detail.

[0067] 11 is a diagram showing a second example of handwritten content created using the user device 12 of FIG. 1. In this example, the handwritten content is artwork 100 depicting a beach scene. A creator (here, a "creator") completes the desired artwork 100 by making full use of the user device 12 and the electronic pen 14. Even when multiple creators draw the same scene, the creation process, including the drawing steps and color usage, differs for each creator.

[0068] 3, the second calculation unit 44 calculates a style feature 56B (one aspect of the integrated feature 56) based on two or more types of time-series feature 54 among the multiple types of time-series feature 54 calculated in step SP14. A learning device 50B is used to quantify the style.

[0069] FIG. 12 is a diagram showing a second example of the network structure of the learning device 50B. The learning device 50B is composed of an autoencoder for extracting features from data. Specifically, this learning device 50B is a hierarchical neural network computing device composed of an input layer 111, an intermediate layer 112, and an output layer 113. The input layer 111 is composed of N computation units for inputting each value of a feature quantity. The intermediate layer 112 is composed of M computation units (here, 1≦M<N). The output layer 113 is composed of the same number of computation units as the input layer 111 (i.e., N). In this network configuration, the input layer 111 and the intermediate layer 112 perform a "dimensionality compression function," and the intermediate layer 112 and the output layer 113 perform a "dimensionality restoration function."

[0070] The learning parameters constituting the learning device 50B are optimized by so-called "unsupervised learning." In the example shown in the figure, the data set used for machine learning is composed of color and position feature values ​​relating to the color and position of strokes, pen pressure feature values ​​of the electronic pen 14, and the degree of completion of the artwork 100. This data set is, for example, a collection of data collected when the creator to be evaluated created multiple artworks 100 in the past.

[0071] The learning device 50B generates drawing features for each stroke by inputting a feature set (N-dimensional vector) for each stroke to the input layer 111 and outputting compressed features (M-dimensional vector) from the intermediate layer 112. The second calculation unit 44 then calculates a representative value (e.g., a statistical value including the average, maximum, minimum, and mode) for each vector component of the population of drawing features. Through this series of calculations, a style feature 56B indicating the style of the handwritten content is calculated.

[0072] 3, the content evaluation unit 46 performs a desired evaluation process on the style feature values ​​56B saved in step SP18, as needed, and outputs evaluation result information 58 including the evaluation results of the handwritten content. In the second operation example, the content evaluation unit 46 evaluates the style of the corresponding artwork 100 by performing a classification process on the style feature values ​​56B.

[0073] 13 is a diagram showing an example of a method for classifying styles. Here, an M-dimensional feature coordinate space 120 is shown, with M-dimensional vector components, which are style features 56B, as the axes. For ease of illustration, only two components (i, j) of the M vector components are shown. A collection of feature points 122 corresponding to various artworks 100 is provided on this feature coordinate space 120.

[0074] A collection of feature points 122 forms a plurality of clusters (three in this example). Group G1 constituting the first style is divided by two boundary lines 124 and 125. Group G2 constituting the second style is divided by two boundary lines 124 and 126. Group G3 constituting the third style is divided by two boundary lines 125 and 126. Here, it is determined which of groups G1 to G3 the style feature amount 56B belongs to based on the relationship between the position indicated by the style feature amount 56B and the three boundary lines 124 to 126.

[0075] In this way, as in the first operation, the user device 12 visualizes and displays the evaluation result information 58 together with the artwork 100, which is one form of handwritten content, thereby making the creator of the artwork 100 aware of his or her own style. In this way, the content evaluation system 10 completes the second operation shown in the flowchart of FIG.

[0076] [Summary of the embodiment] As described above, the content evaluation system 10 in this embodiment includes: an electronic pen 14; a data generation unit (here, the processor 21) that generates content data indicating handwritten content (e.g., the answer sheet 60 or the artwork 100) by accepting writing operations using the electronic pen 14; a data acquisition unit 40 that acquires the content data generated by the processor 21; a first calculation unit 42 that calculates, from the acquired content data, multiple types of time-series feature quantities 54 that indicate changes over time in multiple types of feature quantities related to the process of creating the handwritten content; a second calculation unit 44 that calculates an integrated feature quantity 56 related to the process of creating the handwritten content using two or more types of time-series feature quantities 54 out of the calculated multiple types of time-series feature quantities 54; and a content evaluation unit 46 that evaluates the handwritten content based on the calculated integrated feature quantity 56 and outputs evaluation result information 58 that indicates the evaluation result of the handwritten content.

[0077] Furthermore, according to the content evaluation method or program of this embodiment, one or more computers (or processors) execute the following steps: an acquisition step (SP10) of acquiring content data indicating handwritten content; a first calculation step (SP14) of calculating, from the acquired content data, multiple types of time-series feature quantities 54 indicating temporal changes in multiple types of feature quantities related to the creation process of the handwritten content; a second calculation step (SP16) of calculating, using two or more types of time-series feature quantities 54 among the calculated multiple types of time-series feature quantities 54, an integrated feature quantity 56 related to the creation process of the handwritten content; and an evaluation step (SP20) of evaluating the handwritten content based on the calculated integrated feature quantity 56 and outputting evaluation result information 58 indicating the evaluation result of the handwritten content. Furthermore, the recording medium in this embodiment is a computer-readable, non-transitory recording medium that stores the above-mentioned program.

[0078] In this way, by calculating an integrated feature 56 relating to the creation process of the handwritten content using two or more types of time-series feature 54 that indicate the change in feature over time, and evaluating the handwritten content based on this integrated feature 56, it is possible to evaluate the handwritten content more precisely than when evaluating it simply using the drawing content of the finished product.

[0079] For example, if the handwritten content is an answer 60 (Figure 4) to a given question, [1] the content data may include stroke data 68 indicating a collection of strokes, [2] the two or more types of time series features 54 may include a first time series feature indicating a change in position corresponding to the stroke over time, and a second time series feature indicating a change in the movement speed of the electronic pen 14 corresponding to the stroke over time, and [3] the evaluation result information 58 may include the motivation or concentration level of the creator who created the answer 60.

[0080] For example, if the handwritten content is an answer 60 (Figure 4) to a given question, [1] the content data may include stroke data 68 indicating a collection of strokes, [2] the two or more types of time series features 54 may include a first time series feature indicating a change over time in the movement speed of the electronic pen 14 corresponding to the strokes, and a second time series feature indicating a change over time in the writing pressure of the electronic pen 14 corresponding to the strokes, and [3] the evaluation result information 58 may include the motivation or concentration level of the creator who created the answer 60.

[0081] For example, if the handwritten content is artwork 100 (Figure 11), then [1] the content data includes stroke data 68 indicating a collection of strokes, [2] the two or more types of time series features 54 include a first time series feature indicating a change in position corresponding to the stroke over time, and a second time series feature indicating a change in pressure of the electronic pen 14 corresponding to the stroke over time, and [3] the evaluation result information 58 may include the style or authenticity of the artwork 100.

[0082] For example, if the handwritten content is artwork 100 (Figure 11), [1] the content data may include stroke data 68 indicating a collection of strokes, [2] the two or more types of time series features 54 may include a first time series feature indicating a change over time in the position corresponding to the strokes, and a second time series feature indicating a change over time in the color corresponding to the strokes, and [3] the evaluation result information 58 may include the style or authenticity of artwork 100.

[0083] The content evaluation system 10 may further include a display unit (here, the display unit 24) that displays the evaluation result information 58 output from the content evaluation unit 46 together with the handwritten content. This allows a user who views the evaluation result information 58 to be aware of the process of creating the handwritten content.

[0084] Alternatively, the user device 12 may be provided with the processor 21 and the display unit 24, and the server device 16 configured to be able to communicate with the user device 12 may be provided with the data acquisition unit 40, the first calculation unit 42, the second calculation unit 44, and the content evaluation unit 46. The user device 12 may sequentially transmit content data generated by the processor 21, receive evaluation result information 58 from the server device 16, and display the evaluation result information 58 on the display unit 24 each time it receives a writing operation using the electronic pen 14. This allows the evaluation results to be provided in real time while handwritten content is being created, allowing the creator to reflect the evaluation results in subsequent creations.

[0085] Furthermore, the second calculation unit 44 may calculate the integrated feature 56 using a neural network calculator (here, the learning device 50) that receives the time-series feature 54 as input. This makes it possible to obtain the integrated feature 56 that reflects the complex relationships between multiple types of feature.

[0086] [Modifications] The present invention is not limited to the above-described embodiment, and can be freely modified without departing from the spirit of the present invention. Alternatively, the respective configurations may be arbitrarily combined within the scope of no technical contradiction. Alternatively, the execution order of each step constituting the flowchart may be changed within the scope of no technical contradiction.

[0087] In the above-described embodiment, an example has been described in which the integrated feature 56 is calculated using an autoencoder (FIGS. 7 and 12), but the network structure of the learning device 50 is not limited to this. For example, the learning device 50 may be configured as either a "regression-type" neural network computing device that receives an input of a collection of multiple types of time-series feature 54 and outputs [1] a quantitative value indicating the creator's level of motivation or concentration, or a "classification-type" neural network computing device that outputs [2] a label value indicating a style classification.

[0088] [Explanation of symbols] 10... content evaluation system, 12... user device, 14... electronic pen, 16... server device, 21... processor (data generation unit), 24... display unit (display unit), 40... data acquisition unit, 42... first calculation unit, 44... second calculation unit, 46... content evaluation unit, 50, 50A, 50B... learning device, 54... time-series feature amount, 56... integrated feature amount, 58... evaluation result information, 60... answer sheet (handwritten content), D1... evaluation data, D2... display data

Claims

1. An electronic pen, a data generating unit that generates content data representing handwritten content by receiving a writing operation using the electronic pen; a data acquisition unit that acquires the content data generated by the data generation unit; a first calculation unit that calculates, from the content data acquired by the data acquisition unit, a plurality of types of time-series feature quantities indicating time changes of a plurality of types of feature quantities related to a creation process of the handwritten content; a second calculation unit that calculates an integrated feature amount related to a creation process of the handwritten content by using two or more types of time-series feature amounts among the plurality of types of time-series feature amounts calculated by the first calculation unit; a content evaluation unit that evaluates the handwritten content based on the integrated feature calculated by the second calculation unit and outputs evaluation result information indicating an evaluation result of the handwritten content; A content rating system comprising:

2. the handwritten content being an answer to a given problem; The content rating system of claim 1 .

3. the content data includes stroke data indicating a collection of strokes; the two or more types of time-series feature amounts include a first time-series feature amount indicating a time change in a position corresponding to the stroke, and a second time-series feature amount indicating a time change in a moving speed of the electronic pen corresponding to the stroke, The evaluation result information includes a degree of motivation or concentration of the person who created the answer sheet. The content rating system of claim 2 .

4. the content data includes stroke data indicating a collection of strokes; the two or more types of time-series feature amounts include a first time-series feature amount indicating a time change in a moving speed of the electronic pen corresponding to the stroke, and a second time-series feature amount indicating a time change in a writing pressure of the electronic pen corresponding to the stroke, The evaluation result information includes a degree of motivation or concentration of the person who created the answer sheet. The content rating system of claim 2 .

5. the handwritten content is artwork; The content rating system of claim 1 .

6. the content data includes stroke data indicating a collection of strokes; the two or more types of time-series feature amounts include a first time-series feature amount indicating a time change in a position corresponding to the stroke, and a second time-series feature amount indicating a time change in a writing pressure of the electronic pen corresponding to the stroke, The evaluation result information includes the style or authenticity of the artwork. The content rating system of claim 5 .

7. the content data includes stroke data indicating a collection of strokes; the two or more types of time-series feature amounts include a first time-series feature amount indicating a change over time of a position corresponding to the stroke, and a second time-series feature amount indicating a change over time of a color corresponding to the stroke, The evaluation result information includes the style or authenticity of the artwork. The content rating system of claim 5 .

8. a display unit that displays the evaluation result information output from the content evaluation unit together with the handwritten content, The content rating system of claim 1 .

9. the data generating unit and the display unit are provided in a user device, the data acquisition unit, the first calculation unit, the second calculation unit, and the content evaluation unit are provided in a server device configured to be able to communicate with the user device, the user device sequentially transmits the content data generated by the data generation unit, receives the evaluation result information from the server device, and displays the evaluation result information on the display unit, every time the user device receives a writing operation using the electronic pen. The content rating system of claim 8 .

10. the second calculation unit calculates the integrated feature using a neural network calculator that receives the time-series feature as an input; The content rating system of claim 1 .

11. One or more computers An acquisition step of acquiring content data indicative of handwritten content; a first calculation step of calculating, from the acquired content data, a plurality of types of time-series feature quantities indicating time changes of a plurality of types of feature quantities related to a creation process of the handwritten content; a second calculation step of calculating an integrated feature amount related to a creation process of the handwritten content by using two or more types of time-series feature amounts among the plurality of types of time-series feature amounts calculated; an evaluation step of evaluating the handwritten content based on the calculated integrated feature amount and outputting evaluation result information indicating an evaluation result of the handwritten content; Run your content evaluation method.

12. On one or more computers, An acquisition step of acquiring content data indicative of handwritten content; a first calculation step of calculating, from the acquired content data, a plurality of types of time-series feature quantities indicating time changes of a plurality of types of feature quantities related to a creation process of the handwritten content; a second calculation step of calculating an integrated feature amount related to a creation process of the handwritten content by using two or more types of time-series feature amounts among the plurality of types of time-series feature amounts calculated; an evaluation step of evaluating the handwritten content based on the calculated integrated feature amount and outputting evaluation result information indicating an evaluation result of the handwritten content; A content evaluation program that allows users to