Information processing equipment, methods, programs and systems

Abstract

The present invention relates to an information processing device, a method, a program, and a system. An information processing device (12) comprises: an operation reception unit (64) that receives a cancellation operation for cancelling a first handwriting operation by an input device (14) used for handwriting input; an information acquisition unit (66) that acquires learning information (46) indicating a learning result regarding a relationship between the contents of the first handwriting operation and the contents of a second handwriting operation by the input device (14) made after the cancellation operation; and an information correction unit (70) that corrects handwriting information (72) indicating a handwriting result corresponding to a third operation by the input device (14) or handwriting configuration information (74) regarding a configuration at the time of handwriting in the input device (14) on the basis of the learning information (46).

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, method, program, and system.

Background Art

[0002] Conventionally, an information processing apparatus for a user to input content such as characters and pictures by handwriting has been known. For example, Japanese Patent Application Laid-Open No. 2020-024756 discloses a method of changing handwritten data once stored in a file while the user checks the display state.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the method disclosed in Japanese Patent Application Laid-Open No. 2020-024756, operations for checking and changing data are required, which increases the burden on the user accordingly.

[0005] The present invention has been made in view of such problems, and an object thereof is to provide an information processing apparatus, method, program, and system capable of automatically and timely correcting various information obtained through a writing operation using an input device.

Means for Solving the Problems

[0006] The first information processing device of the present invention includes: an operation receiving unit that receives a cancellation operation to cancel a first operation performed by an input device used for handwriting input; an information acquisition unit that acquires learning information indicating the relationship between the content of the first operation and the content of a second operation performed by the input device after the cancellation operation received by the operation receiving unit; and an information modification unit that modifies handwriting information indicating the handwriting result corresponding to a third operation performed by the input device or handwriting setting information relating to the handwriting settings corresponding to the third operation, based on the learning information acquired by the information acquisition unit.

[0007] A second information processing method according to the present invention is performed by one or more computers, comprising: a reception step of receiving a cancellation operation to cancel a first operation performed by an input device used for handwriting input; an acquisition step of acquiring learning information that shows the learning result regarding the relationship between the content of the first operation and the content of a second operation performed by the input device after the cancellation operation received by the operation reception unit; and a modification step of modifying handwriting information that shows the handwriting result corresponding to a third operation performed by the input device or handwriting setting information that relates to the handwriting settings corresponding to the third operation, based on the acquired learning information.

[0008] A third information processing program in the present invention causes one or more computers to execute: a reception step of receiving a cancellation operation to cancel a first operation performed by an input device used for handwriting input; an acquisition step of acquiring learning information that shows the learning result regarding the relationship between the content of the first operation and the content of a second operation performed by the input device after the cancellation operation received by the operation reception unit; and a modification step of modifying handwriting information that shows the handwriting result corresponding to a third operation performed by the input device or handwriting setting information that relates to the handwriting settings corresponding to the third operation, based on the acquired learning information.

[0009] The fourth information processing system of the present invention comprises an information processing device according to the first invention and a server device that transmits the learning information to the information processing device in response to a request from the information processing device. [Effects of the Invention]

[0010] According to the present invention, various types of information obtained through writing operations using an input device can be automatically and promptly corrected. [Brief explanation of the drawing]

[0011] [Figure 1] This is an overall configuration diagram of an information processing system in one embodiment of the present invention. [Figure 2] This block diagram shows an example of the configuration of the user device in Figure 1. [Figure 3] Figure 1 is a flowchart illustrating an example of how the server device generates training data. [Figure 4] Figure 1 is a flowchart illustrating an example of how the server device performs learning for the second classifier. [Figure 5] This figure shows an example of the network structure of the first discriminator. [Figure 6] This figure shows an example of the network structure of the second discriminator. [Figure 7] Figures 1 and 2 are flowcharts illustrating an example of how the user device modifies various information. [Figure 8] This figure shows the first specific example of a corrective action. [Figure 9] This figure shows a second specific example of corrective actions. [Figure 10] This figure shows a third specific example of corrective actions. [Figure 11] This figure shows a fourth specific example of corrective actions. [Modes for carrying out the invention]

[0012] Embodiments of the present invention will be described below with reference to the attached drawings. To facilitate understanding of the description, the same reference numerals are used for identical components in each drawing whenever possible, and redundant explanations are omitted.

[0013] [Overall Configuration of Information Processing System 10] FIG. 1 is an overall configuration diagram of an information processing system 10 according to an embodiment of the present invention. The information processing system 10 is provided to provide an "input support service" for assisting a user's handwriting input. Specifically, the information processing system 10 includes one or more user devices 12 (corresponding to "information processing devices"), one or more electronic pens 14 (corresponding to "input devices"), and a server device 16.

[0014] The user device 12 is a computer owned by a user who uses the input support service, and has a function of detecting an instruction position by the electronic pen 14. The user device 12 is composed of, for example, a tablet, a smartphone, a personal computer, etc.

[0015] The electronic pen 14 is a pen-type pointing device, and is configured to be communicable in one direction or two directions with the user device 12. This electronic pen 14 is, for example, a stylus of an active electrostatic coupling method (AES) or an electromagnetic induction method (EMR). The user can hold the electronic pen 14 and move it while pressing the pen tip against the touch surface of the user device 12 to write pictures and characters on the user device 12.

[0016] The server device 16 is a computer that performs overall control related to the support of handwriting input, and may be either a cloud type or an on-premises type. Here, although the server device 16 is illustrated as a single computer, the server device 16 may be a group of computers that constructs a distributed system instead. Specifically, the server device 16 includes a server-side communication unit 20, a server-side control unit 22, and a server-side storage unit 24.

[0017] The server-side communication unit 20 is an interface for transmitting and receiving electrical signals to and from an external device. Thereby, the server device 16 can acquire the digital ink 76 (FIG. 2) from the user device 12 and supply the second parameter group 46 generated by itself to the user device 12.

[0018] The server-side control unit 22 is composed of a processor including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The server-side control unit 22 functions as a data processing unit 26, a learning processing unit 28, and an output processing unit 30 by reading and executing programs and data stored in the server-side storage unit 24.

[0019] The data processing unit 26 processes the digital ink 76 (FIG. 2) generated by the user device 12 and generates learning data 42, which will be described later. The operation of the data processing unit 26 will be described in detail later while referring to the flowchart of FIG. 3.

[0020] The learning processing unit 28 performs learning processing on the first discriminator 32 and the second discriminator 34 using the learning data 42 generated by the data processing unit 26. The operation of the learning processing unit 28 will be described in detail later while referring to the flowchart of FIG. 4.

[0021] The output processing unit 30 outputs or transmits the second parameter group 46 (FIG. 2) obtained by the learning processing performed by the learning processing unit 28 toward the user device 12. Thereby, the user device 12 can utilize the second discriminator 34 through the second parameter group 46 customized according to the user.

[0022] The server-side storage unit 24 stores programs and data necessary for the server-side control unit 22 to control each component. The server-side storage unit 24 is composed of a non-transient and computer-readable storage medium. Here, the computer-readable storage medium consists of [1] storage devices such as hard disk drives (HDDs) and solid state drives (SSDs) built into the computer system, and [2] portable media such as magneto-optical disks, ROMs, CD-ROMs, and flash memory.

[0023] Furthermore, the server-side storage unit 24 has a database (hereinafter referred to as the Digital Ink DB 40) relating to the Digital Ink 76 (Figure 2), which will be described later, and also stores training data 42, one or more first parameter groups 44, and one or more second parameter groups 46 (corresponding to "training information").

[0024] The training data 42 is a collection of data used for machine learning performed by the learning processing unit 28. The training data 42 includes [1] a feature set for each writing operation, [2] a state vector representing the state space when performing a writing operation, and [3] the type of event.

[0025] Here, operations are classified into first, second, and third operations. All three operations share the common characteristic of being operations performed by an input device used for handwriting input. The first operation corresponds to an operation that has been canceled by a cancel operation. The second operation corresponds to an operation performed after a cancel operation that was not subsequently canceled. The third operation is an operation that is identical or similar to the first operation and therefore may be canceled. For example, if the input device is an electronic pen 14, the unit of operation is a series of operations from the time the pen is down to the time the pen is up.

[0026] Examples of feature sets include: [1] "Stroke features" including coordinate values, pressure values, and tilt angle when drawing a stroke; [2] "Device driver settings" including input device identification information (e.g., pen ID), type, user device 12 type, and pressure curve shape; and [3] "Drawing application settings" including content type, color palette and brush color information, and visual effect settings. The state vector also includes feature sets related to one or more past writing operations.

[0027] The first parameter group 44 is a set of learning parameters for identifying the operation rules of the first classifier 32. The second parameter group 46 is a set of learning parameters that can identify the operation rules of the second classifier 34. The learning parameters may include, for example, coefficients that describe the activation function of the operation unit, weighting coefficients corresponding to the strength of synaptic connections, the number of operation units constituting each layer, and the number of hidden layers.

[0028] The first parameter group 44 or the second parameter group 46 may be uniform regardless of the user, or they may be customized for each user. In the latter case, it is desirable that the first parameter group 44 or the second parameter group 46 are pre-stored in association with user-identifiable information and read out in a timely manner using the identification information contained in the digital ink 76 (Figure 2). As identification information, for example, a user account for the input support service, or various IDs related to the electronic pen 14 or user device 12 may be used.

[0029] Figure 2 is a block diagram showing an example of the configuration of the user device 12 in Figure 1. Specifically, this user device 12 consists of a touch panel display 50, a display driver IC (Integrated Circuit) 54, a touch IC 56, a communication module 58, a host processor 60 (corresponding to "processor"), and memory 62 (corresponding to "storage medium").

[0030] The touch panel display 50 comprises a display panel 51 capable of visibly displaying content, and a planar touch sensor 52 positioned to overlap the display panel 51 in a planar view. In the example shown in this figure, the touch sensor 52 is an "external" sensor attached to the display panel 51 from the outside, but it may be replaced with an "internal" sensor (further classified as on-cell or in-cell) that is integrated with the display panel 51.

[0031] The display panel 51 is capable of displaying monochrome or color images and is composed of, for example, a liquid crystal panel, an organic EL (Electro-Luminescence) panel, or electronic paper. By making the display panel 51 flexible, the user can perform various writing operations while the touch surface of the user device 12 is curved or bent.

[0032] The touch sensor 52 is a capacitive sensor comprising multiple sensor electrodes arranged in a planar configuration. This touch sensor 52 is composed of, for example, multiple X-line electrodes for detecting the position along the X axis of the sensor coordinate system and multiple Y-line electrodes for detecting the position along the Y axis. In this case, the X-line electrodes are provided extending in the Y-axis direction and are arranged at equal intervals along the X-axis direction. The Y-line electrodes are provided extending in the X-axis direction and are arranged at equal intervals along the Y-axis direction. Note that the touch sensor 52 may be a self-capacitive sensor in which block-shaped electrodes are arranged in a two-dimensional grid, instead of the mutual-capacitive sensor described above.

[0033] The display driver IC 54 is an integrated circuit that is electrically connected to the display panel 51 and controls the driving of the display panel 51. The display driver IC 54 drives the display panel 51 based on display signals supplied from the host processor 60. As a result, the content indicated by the digital ink 76 is displayed on the display panel 51.

[0034] The touch IC 56 is an integrated circuit that is electrically connected to the touch sensor 52 and controls the operation of the touch sensor 52. The touch IC 56 drives the touch sensor 52 based on control signals supplied from the host processor 60. As a result, the touch IC 56 performs a "pen detection function" to detect the state of the electronic pen 14 and a "touch detection function" to detect touches made by the user's finger or the like.

[0035] This pen detection function includes, for example, a scanning function for the touch sensor 52, a function for receiving and analyzing downlink signals, a function for estimating the state of the electronic pen 14 (e.g., position, orientation, pressure), and a function for generating and transmitting uplink signals that include commands for the electronic pen 14. The touch detection function also includes, for example, a two-dimensional scanning function for the touch sensor 52, a function for acquiring a detection map on the touch sensor 52, and a function for classifying areas on the detection map (e.g., classification of fingers, palms, etc.).

[0036] In this way, a graphical user interface (GUI) is constructed by combining the input functions of the electronic pen 14 and touch sensor 52 with the output functions of the display panel 51.

[0037] The communication module 58 has a communication function that enables wired or wireless communication with external devices. As a result, the user device 12 can transmit the digital ink 76 to the server device 16 and receive the second parameter group 46 from the server device 16.

[0038] The host processor 60 is composed of a processing unit including a CPU, GPU, or MPU (Micro-Processing Unit). The host processor 60 functions as an operation reception unit 64, an information acquisition unit 66, an event detection unit 68, and an information correction unit 70 by reading and executing programs and data stored in memory 62.

[0039] The operation reception unit 64 receives user operations from an input device used for writing input (e.g., an electronic pen 14). These operations execute various drawing functions, such as [1] writing functions including stroke generation, [2] setting functions related to stroke color, thickness, decoration, etc., and [3] editing functions including canceling operations, duplicating objects, changing layouts, and saving data. Operations to undo past operations (hereinafter referred to as "undo operations") include "UNDO operations" which undo the most recent operation, or "ERASE operations" which erase at least a part of a drawing object. Furthermore, an undo operation may undo a single operation, or it may undo some or all operations across multiple operations.

[0040] User operations include both (1) operations on the electronic pen 14 and (2) operations on the user device 12 performed using the electronic pen 14. Examples of the former include pen-down, pen-move, pen-up, and erase operations using the electronic pen 14, as well as pressing switches provided on the electronic pen 14. Examples of the latter include operations on user controls (windows, buttons, palettes, etc.) displayed on the touch panel display 50.

[0041] The information acquisition unit 66 acquires information (hereinafter referred to as "learning information") that shows the learning result regarding the relationship between the content of the first operation performed by the input device and the content of the second operation performed by the input device after the cancellation operation received by the operation reception unit 64. The information acquisition unit 66 may acquire the learning information by reading it from the memory 62, or it may acquire the learning information by downloading it from an external device including the server device 16. In addition, the information acquisition unit 66 may acquire learning information corresponding to the identification information of the electronic pen 14 (i.e., the pen ID).

[0042] The event detection unit 68 detects the presence and type of events that may be subject to a cancel operation from the current writing state of the input device. Here, event detection is performed by a second classifier 34 constructed according to the learned information acquired by the information acquisition unit 66. Event types are classified into [1] events caused by operational errors and [2] events caused by user habits, illusions, or misperceptions. Examples of the former include writing errors such as straight lines, curves, shapes, and symbols, as well as fluctuations in stroke position and pen pressure. Examples of the latter include timing discrepancies in switch presses, incorrect color selection in the color palette, and unnecessary operations performed habitually.

[0043] When the event detection unit 68 detects a predetermined event, the information correction unit 70 corrects the writing information 72 or writing setting information 74 corresponding to the third operation, depending on the type of event. In other words, if a third operation identical or similar to a first operation that may be subject to cancellation is performed, the writing information 72 or writing setting information 74 is automatically corrected as if an operation similar to a second operation had been performed. Specific examples of the correction operation will be described in detail later with reference to Figures 8 to 11.

[0044] Memory 62, like the server-side storage unit 24 (Figure 1), is composed of a non-transient and computer-readable storage medium. In the example shown in this figure, memory 62 stores writing information 72, writing setting information 74, digital ink 76, and the second parameter group 46 described above.

[0045] The writing information 72 is information that indicates the writing result corresponding to the operation by the input device. An example of the writing information 72 is stroke data, which includes coordinate values, pen pressure values, and tilt angle when drawing a stroke.

[0046] The writing setting information 74 is information regarding the settings used when writing, corresponding to operations performed by the input device. Examples of writing setting information 74 include "device driver-side setting conditions" which include the device ID and type, the type of user device 12, and the shape of the pen pressure curve, and "drawing application-side setting conditions" which include the type of content, color palette and brush color information, and visual effect settings.

[0047] Digital Ink 76 is ink data for representing handwritten content. Examples of Digital Ink 76 data formats, or so-called "ink description languages," include WILL (Wacom Ink Layer Language), InkML (Ink Markup Language), and ISF (Ink Serialized Format). Digital Ink 76 has a data structure consisting of a sequential arrangement of [1] document metadata, [2] ink semantics, [3] devices, [4] strokes, [5] groups, and [6] context data.

[0048] Stroke data is data used to describe the individual strokes that make up the content. A single stroke is, <trace>It is described by multiple point data arranged sequentially within a tag. Each point data consists of at least an indicated position (X coordinate, Y coordinate) and is separated by a delimiter such as a comma. For the sake of illustration, only the point data indicating the start and end points of a stroke are shown, and point data indicating multiple intermediate points are omitted. In addition to the indicated positions mentioned above, this point data may also include the writing order, the pressure and posture of the electronic pen 14, etc.

[0049] [Operation of Information Processing System 10] The information processing system 10 in this embodiment is configured as described above. Next, the operation of the information processing system 10 will be explained with reference to Figures 3 to 7.

[0050] <1. Generating Digital Ink 76> First, the operation by which the user device 12 in Figure 1 generates digital ink 76 will be explained with reference to Figures 1 and 2.

[0051] The user uses their own electronic pen 14 to input writing to the user device 12. The operation reception unit 64 (Figure 2) of the user device 12 receives the user's operations sequentially, generating writing information 72 and writing setting information 74. When the operation reception unit 64 receives a data saving operation, the host processor 60 generates digital ink 76 containing the writing information 72 and writing setting information 74, and stores the digital ink 76 in the memory 62.

[0052] Subsequently, the user device 12 transmits the digital ink 76 it generated to the server device 16. The server-side control unit 22 then stores the digital ink 76 received via the server-side communication unit 20 in the server-side storage unit 24. As a result, the new digital ink 76 is added and registered in the digital ink DB 40.

[0053] <2. Generation of training data 42> Next, the operation by which the server device 16 in Figure 1 generates training data 42 will be explained with reference to the flowchart in Figure 3 and Figure 1. Each step shown in Figure 3 is executed by the data processing unit 26 of the server device 16.

[0054] In step SP10 of Figure 3, the data processing unit 26 of the server device 16 refers to the digital ink DB 40 and obtains the digital ink 76 to be analyzed.

[0055] In step SP12, the data processing unit 26 analyzes the digital ink 76 acquired in step SP10 and searches for an instruction code indicating an UNDO operation.

[0056] In step SP14, the data processing unit 26 checks whether an UNDO operation was detected by the search in step SP12. If no UNDO operation is detected (step SP14: NO), the data processing unit 26 skips the execution of step SP16. On the other hand, if an UNDO operation is detected (step SP14: YES), the data processing unit 26 proceeds to the next step SP16.

[0057] In step SP16, the data processing unit 26 acquires and processes peripheral data related to the UNDO operation detected in step SP14, and generates training data 42.

[0058] In step SP18, the data processing unit 26 checks whether all data searches within the digital ink 76 have been completed. If the data searches are not yet complete (step SP18: NO), the data processing unit 26 returns to step SP12 and repeats steps SP12 to SP18 sequentially until all searches are completed. On the other hand, if all data searches are completed (step SP18: YES), the data processing unit 26 terminates the process shown in the flowchart in Figure 3.

[0059] <3. Learning of the second classifier 34> Next, the operation by which the server device 16 in Figure 1 performs learning for the second discriminator 34 will be explained with reference to the flowchart in Figure 4, Figures 5 and 6.

[0060] In step SP20 of Figure 4, the server-side control unit 22 (Figure 1) of the server device 16 checks whether or not it has received an update request from a specific user. If the first classifier 32 or the second classifier 34 is customized for each user, the learning processing unit 28 acquires learning data 42 corresponding to the identification information (e.g., pen ID) that identifies the user.

[0061] In step SP22, the learning processing unit 28 of the server device 16 uses the learning data 42 acquired in step SP20 and the first classifier 32 to determine the relationship between the first operation and the second operation.

[0062] Figure 5 shows an example of the network structure of the first classifier 32. The first classifier 32 is composed of, for example, a hierarchical neural network 80, and includes an input layer 82, an intermediate layer 84, and an output layer 86. The operation rules of the first classifier 32 are determined by the values ​​of the first parameter group 44 (Figure 1), which is a set of learning parameters.

[0063] The input layer 82 is a layer that receives feature sets related to the first and second operations, and consists of multiple (e.g., 2·m) computation units.

[0064] The hidden layer 84 consists of one or more layers and has a dimensionality reduction function that reduces the number of dimensions of the input vector. In other words, it is desirable that the number of arithmetic units constituting the hidden layer 84 be less than 2·m.

[0065] The output layer 86 is a layer that outputs a set of event labels, and in the example shown in this figure, it consists of (N+1) operation units. The "event n" label (n=1, ..., N) indicates the state where the nth event has occurred, and the "normal" label indicates the state where no events have occurred. When the activation function of the operation unit is the softmax function, this set of labels is an output vector consisting of (N+1) components that indicate the probability of an event occurring. In other words, if there is a label with the maximum label value (hereinafter referred to as a specific label), the first operation identifies that the event corresponding to the specific label has occurred.

[0066] In step SP24 of Figure 4, the learning processing unit 28 performs a learning process on the second classifier 34 using the training data 42 acquired in step SP20 and the event types identified in step SP22. This learning process is performed using various techniques commonly applied to machine learning. Through this learning process, a second set of parameters 46 for each user is calculated.

[0067] Figure 6 shows an example of the network structure of the second classifier 34. The second classifier 34 is composed of, for example, a hierarchical neural network 90, and includes an input layer 92, an intermediate layer 94, and an output layer 96. The operation rules of the second classifier 34 are determined by the values ​​of the second parameter group 46 (Figure 1), which is a set of learning parameters.

[0068] The input layer 92 is a layer that receives the state vector during the operation and a set of features related to the operation, and is composed of multiple {for example, (M+m)} operation units.

[0069] The hidden layer 94 consists of one or more layers and has a dimensionality reduction function that reduces the number of dimensions of the input vector. In other words, it is desirable that the number of arithmetic units constituting the hidden layer 94 is less than (M+m).

[0070] The output layer 96 is a layer that outputs a set of event labels, and in the example shown in this figure, it consists of (N+1) processing units. As the correct training data, a one-hot vector is used in which the vector component of the event identified in step SP22 is set to "1" and the remaining vector components are set to "0".

[0071] In step SP26 of Figure 4, the server-side communication unit 20 of the server device 16 transmits the second parameter group 46, calculated through the learning process in step SP24, to the user device 12 that made the update request. The user device 12 stores the received second parameter group 46 in memory 62, thereby enabling it to construct the second classifier 34 in its own device environment. In this way, the server device 16 completes the processing shown in the flowchart of Figure 4.

[0072] <4. Automatic correction of operations> Next, the operation of the user device 12, which constitutes part of the information processing system 10, specifically the operation modification operation, will be explained with reference to the flowchart in Figure 7.

[0073] In step SP30 of Figure 7, the event detection unit 68 checks whether the timing for detecting an event has arrived. The detection timing may be, for example, when a writing operation is completed. If the detection timing has not yet arrived (step SP30: NO), the unit remains in step SP30 until the detection timing arrives. On the other hand, if the detection timing has arrived (step SP30: YES), the unit proceeds to the next step SP32.

[0074] In step SP32, the event detection unit 68 acquires various information for identifying the event (in this case, writing information 72 and writing setting information 74).

[0075] In step SP34, the event detection unit 68 identifies the presence and type of an event from the handwriting information 72 and handwriting setting information 74 acquired in step SP32. This identification process is performed through a second classifier 34 that has been trained for the user.

[0076] In step SP36, the information correction unit 70 checks whether an event has been detected by referring to the identification result from step SP34. If no event is detected (step SP36: NO), the information correction unit 70 returns to step SP30 without correcting the writing information 72 and writing setting information 74. On the other hand, if an event is detected (step SP36: YES), the process proceeds to the next step SP38.

[0077] In step SP38, the information correction unit 70 corrects the handwriting information 72 or handwriting setting information 74 according to the type of event detected in step SP34. Specific examples of the correction will be explained in detail later with reference to Figures 8 to 11.

[0078] Subsequently, returning to step SP30, the host processor 60 repeats steps SP30 to SP38 while the user is performing the writing operation, correcting various information as needed. In this way, the processing shown in the flowchart in Figure 7 is executed sequentially.

[0079] [Specific examples of corrective actions] <Example 1> If the first to third operations are switch operations that involve pressing a switch 14s to change the settings during writing, the information modification unit 70 may modify the writing setting information 74 so that the start time of the events associated with the switch operation is earlier or later than the time the switch operation is executed.

[0080] Figure 8 shows a first example of a corrective action. This first example relates to the operation of a switch 14s located on the side of the electronic pen 14. For example, consider a case where a user has a habit of pressing the switch 14s too early. The user tends to press the switch 14s at a position P1 that is closer than the desired position while moving the electronic pen 14 in the direction of arrow M. Alternatively, in the touch panel display 50 (Figure 2) of the user device 12, the height position of the touch surface and the height position of the touch sensor 52 are slightly different, which can cause a timing discrepancy in the operation due to the user's parallax.

[0081] Therefore, after the learning process regarding the user's habits and illusions described above has been completed, if the user performs the same operation in the situation shown in the figure above, the event detection unit 68 detects an event called "timing discrepancy of press". As a result, the information correction unit 70 corrects the writing setting information 74 so that even if the switch 14s is pressed at position P1, the press operation of the switch 14s becomes effective at position P2, which is downstream of position P1 in the direction of arrow M. This allows the user to proceed to the next writing operation without performing an UNDO operation.

[0082] <Example 2> If the first to third operations are stroke operations that draw a stroke, and the writing information 72 includes a set of coordinate values ​​that indicate the time change of the position indicated by the electronic pen 14, the information correction unit 70 may modify the set of coordinate values ​​to suppress position fluctuations in the same stroke.

[0083] Figure 9 shows a second example of a correction operation. This second example relates to the operation of drawing multiple strokes sequentially using an electronic pen 14. For example, consider a case where a user has difficulty drawing an object 100 consisting of multiple parallel lines. When the user moves the electronic pen 14 and draws strokes 101, 102, and 103 in order from top to bottom, they tend to bend the end 104 of the lowest stroke downwards.

[0084] Therefore, after the learning process regarding the user's operational errors described above has been completed, if the user performs the same operation in the situation shown in the figure above, the event detection unit 68 detects an event called "stroke position fluctuation." As a result, when the information correction unit 70 draws strokes 111, 112, and 113 that constitute the object 110, it corrects the writing information 72 so that even if the end portion 114 of stroke 113 is curved downwards, the end portion 115 of stroke 113 becomes a straight line. This allows the user to proceed to the next writing operation without performing an UNDO operation.

[0085] In the second example described above, we assume that the position fluctuations of the stroke are suppressed by changing the trajectory, but there may also be cases where the trajectory is not changed. For example, when converting discrete coordinate data to continuous data by interpolation, the information correction unit 70 may suppress the position fluctuations of the stroke by obtaining continuous data using a smoother interpolation function. Furthermore, the stroke smoothing method is not limited to interpolation functions, and methods such as iterative averaging or decimation of coordinate values ​​may also be used.

[0086] <Example 3> If the first to third operations are stroke operations that draw strokes, and the writing information 72 includes a set of pressure values ​​that indicate the time change of pressure acting on the electronic pen 14, the information modification unit 70 may modify the coordinate value set to suppress fluctuations in the pressure level between multiple strokes.

[0087] Figure 10 shows a third example of a correction operation. This third example relates to the operation of drawing multiple strokes sequentially using an electronic pen 14. For example, consider a case where a user's pen pressure becomes unstable when quickly writing the character 120. When the user draws strokes 121, 122, and 123 while moving the electronic pen 14, the pen pressure of the last stroke, 123, tends to be lower overall, resulting in a narrower line width.

[0088] Therefore, after the learning process regarding the user's habits described above, if the user performs the same operation in the situation shown in the figure above, the event detection unit 68 detects an event called "stroke pressure fluctuation." As a result, the information correction unit 70 corrects the writing information 72 so that even if the pen pressure of the last stroke 133 becomes generally low when drawing strokes 131, 132, and 133 that make up the character 130, the pen pressure corresponding to stroke 133 becomes generally high. This allows the user to proceed to the next writing operation without performing an UNDO operation.

[0089] In the third example described above, we assume a case where fluctuations in pen pressure levels occur between multiple strokes, but this can also be applied to fluctuations in pen pressure levels within the same stroke. For example, when converting discrete data of pen pressure values ​​into continuous data by interpolation, the information correction unit 70 may suppress fluctuations in pen pressure levels by obtaining continuous data using a smoother interpolation function. Also, if the range of pen pressure used by the user is relatively narrow, the information correction unit 70 may change the settings of the pen pressure curve to widen the dynamic range of the pen pressure values.

[0090] <Example 4> If the first to third operations are palette operations to select a color corresponding to an indicated position in the color palette 140, and the writing setting information 74 includes color information in the color palette 140, the information modification unit 70 may modify the color information so that a color at a position different from the indicated position by the palette operation is selected.

[0091] Figure 11 shows a fourth example of a correction operation. This fourth example concerns the operation of selecting a color to be used for drawing from the color palette 140. For example, consider a case where a user prefers to use a particular color. When the user selects their preferred color while moving the electronic pen 14, they tend to mistakenly select a nearby color patch 142.

[0092] Therefore, after the user's habits described above have been learned, if the user performs the same operation in the situation shown in the figure above, the event detection unit 68 detects an event called "color selection error". As a result, the information correction unit 70 corrects the writing setting information 74 so that even if the user mistakenly selects a different color patch 142 when selecting a specific color in the color palette 140, the color patch 144 containing the specific color will be selected. This allows the user to proceed to the next writing operation without performing an UNDO operation.

[0093] In the fourth example described above, it is assumed that one color is selected from the color palette 140, but the number of colors selected may be two or more. In this case, the information modification unit 70 modifies the color combination used for drawing by changing at least one color from among the multiple colors. Alternatively, the information modification unit 70 may modify the writing setting information 74 to add a color that is likely to be used next, triggered by the color selection, based on the learning information that shows the learning results regarding the color combination before and after cancellation.

[0094] <Example 5> If the first to third operations are stroke operations that draw strokes, and the writing information 72 includes a set of coordinate values ​​indicating the time change of the position indicated by the electronic pen 14, the information modification unit 70 may modify the set of coordinate values ​​to delete the strokes used for the draft after part or all of the content is completed through the stroke operations. Alternatively, the information modification unit 70 may modify the writing information 72 to apply a drawing effect equivalent to "erasing" to the strokes after part or all of the content is completed through the stroke operations.

[0095] Furthermore, while the fifth example described above assumes the removal of part or all of a stroke, the object to be removed is not limited to strokes. For example, the information modification unit 70 may modify the writing information 72 to remove parts that are not drawing objects, such as the user's fingers or palm.

[0096] <Example 6> In the first to fifth examples described above, it is assumed that the information modification unit 70 always modifies the writing information 72 or writing setting information 74 when an event is detected. However, instead, the information modification unit 70 may modify the information only when certain activation conditions are met. Examples of activation conditions include [1] the "on" setting has been made through user operation, and [2] it is estimated that the user is irritated. The degree of irritation can be estimated, for example, by the movement speed and gripping state of the electronic pen 14.

[0097] [Effects of the Embodiment] As described above, the information processing system 10 in this embodiment comprises a user device 12 and a server device 16 that transmits learning information (in this case, a second parameter group 46) to the user device 12 in response to a request from the user device 12.

[0098] Furthermore, the user device 12 as an information processing device in this embodiment includes: an operation reception unit 64 that receives a cancellation operation to cancel a first operation performed by an input device (here, an electronic pen 14) used for handwriting input; an information acquisition unit 66 that acquires learning information (here, a second parameter group 46) that shows the learning result regarding the relationship between the content of the first operation and the content of a second operation performed by the electronic pen 14 after the cancellation operation received by the operation reception unit 64; and an information modification unit 70 that modifies handwriting information 72 that shows the handwriting result corresponding to a third operation by the electronic pen 14 or handwriting setting information 74 related to the handwriting settings associated with the third operation, based on the second parameter group 46 acquired by the information acquisition unit 66.

[0099] Furthermore, in the information processing method and information processing program in this embodiment, one or more computers (or processors) perform the following steps: a reception step of receiving a cancellation operation to cancel a first operation performed by an input device (i.e., an electronic pen 14) used for handwriting input; an acquisition step of acquiring learning information (i.e., a second parameter group 46) that shows the learning result regarding the relationship between the content of the first operation and the content of the second operation performed by the electronic pen 14 after the received cancellation operation; and a modification step (SP38 in Figure 7) of modifying writing information 72 that shows the writing result corresponding to the third operation by the electronic pen 14 or writing setting information 74 related to the settings at the time of writing associated with the third operation, based on the acquired second parameter group 46. Furthermore, the recording medium (or storage medium) in this embodiment is a computer-readable non-transient medium that stores the above-described program.

[0100] In this way, the handwritten information 72 or handwritten setting information 74 related to the third operation is modified based on the learning results showing the relationship between the content of the first operation and the content of the second operation performed by the input device after the accepted cancel operation, so that modifications are made that reflect previously performed cancel operations. As a result, various information obtained through handwritten operations using the input device can be automatically and promptly modified.

[0101] Furthermore, if the first to third operations are switch operations involving pressing switch 14s to change the settings during writing, the information modification unit 70 may modify the writing setting information 74 so that the start time of the event associated with the switch operation is earlier or later than the time the switch operation is executed. This eliminates the need to perform a cancel operation even if the timing of pressing switch 14s is off.

[0102] Furthermore, if the first to third operations are stroke operations that draw a stroke, and the writing information 72 includes a set of coordinate values ​​indicating the time change of the position indicated by the input device, the information correction unit 70 may modify the set of coordinate values ​​to suppress position fluctuations in the same stroke. This eliminates the need to perform an undo operation even if position fluctuations occur when drawing a stroke.

[0103] Furthermore, if the first to third operations are stroke operations that draw a stroke, and the writing information 72 includes a set of pressure values ​​that indicate the time change in the pressure acting on the input device, the information correction unit 70 may modify the set of pressure values ​​to suppress fluctuations in the pressure level between multiple strokes. This eliminates the need to perform an undo operation even if pressure fluctuations occur when drawing a stroke.

[0104] Furthermore, if the first to third operations are palette operations to select a color corresponding to an indicated position in the color palette 140, and the writing setting information 74 includes color information within the color palette 140, the information modification unit 70 may modify the color information so that a color at a different position from the indicated position selected by the palette operation is selected. This eliminates the need to perform a cancel operation even if a discrepancy in the indicated position occurs when selecting a color within the color palette 140.

[0105] Furthermore, if the event detection unit 68 detects the presence and type of an event that may be subject to a cancellation operation based on the current writing state of the input device, the information modification unit 70 may modify the writing information 72 or writing setting information 74 according to the type of event detected by the event detection unit 68. Also, if the input device is an electronic pen 14, the first to third operations may be a series of operations performed on the electronic pen 14 from the time of pen-down to the time of pen-up.

[0106] Furthermore, the information acquisition unit 66 may acquire learning information (i.e., the second parameter group 46) corresponding to the identification information of the electronic pen 14. This allows for modifications to be made to suit the user of the electronic pen 14.

[0107] Furthermore, the undo operation may be either an UNDO operation that cancels the previous operation or an erase operation that erases at least a portion of the drawing object. This makes it easier to accurately reflect the user's intention to undo an operation.

[0108] [Differentiation] It should be noted that the present invention is not limited to the embodiments described above, and can be freely modified without departing from the spirit of the invention. Alternatively, the various components may be combined in any way that does not create a technical inconsistency. Furthermore, the execution order of each step constituting the flowchart may be changed as appropriate, as long as it does not create a technical inconsistency.

[0109] [Explanation of symbols] 10... Information processing system, 12... User device (information processing device), 14... Electronic pen (input device), 16... Server device, 34... Second discriminator, 46... Second parameter group (learning information), 64... Operation reception unit, 66... ​​Information acquisition unit, 68... Event detection unit, 70... Information correction unit, 72... Handwriting information, 74... Handwriting setting information< / trace>

Claims

1. An operation reception unit that receives a cancel operation to cancel a first operation performed by an input device used for handwriting input, An information acquisition unit acquires learning information that shows the learning result regarding the relationship between the content of the first operation and the content of the second operation performed by the input device after the cancellation operation received by the operation reception unit, An information modification unit modifies writing information indicating the writing result corresponding to the third operation by the input device or writing setting information relating to the settings during writing corresponding to the third operation, based on the learning information acquired by the information acquisition unit. Equipped with, The first, second, and third operations are switch operations, which involve pressing a switch to change the settings during writing. The information modification unit modifies the writing setting information so that the start time of the event associated with the switch operation is earlier or later than the execution time of the switch operation. Information processing device.

2. A reception step that accepts a cancellation operation to cancel a first operation performed by an input device used for handwriting input, An acquisition step to acquire learning information showing the learning result regarding the relationship between the content of the first operation and the content of the second operation performed by the input device after the accepted cancellation operation, A correction step of correcting writing information indicating the writing result corresponding to the third operation by the input device or writing setting information relating to the settings during writing corresponding to the third operation, based on the acquired learning information, One or more computers run this, The first, second, and third operations are switch operations, which involve pressing a switch to change the settings during writing. In the modification step, the writing setting information is modified so that the start time of the event associated with the switch operation is earlier or later than the time the switch operation is executed. Information processing methods.

3. A reception step that accepts a cancellation operation to cancel a first operation performed by an input device used for handwriting input, An acquisition step to acquire learning information showing the learning result regarding the relationship between the content of the first operation and the content of the second operation performed by the input device after the accepted cancellation operation, A correction step of correcting writing information indicating the writing result corresponding to the third operation by the input device or writing setting information relating to the settings during writing corresponding to the third operation, based on the acquired learning information, Run this on one or more computers. The first, second, and third operations are switch operations, which involve pressing a switch to change the settings during writing. In the modification step, the writing setting information is modified so that the start time of the event associated with the switch operation is earlier or later than the time the switch operation is executed. Information processing program.

4. The information processing apparatus according to claim 1, A server device that transmits the learning information to the information processing device in response to a request from the information processing device, An information processing system equipped with the following features.

Images