Information processing apparatus and control method

Abstract

To reduce a delay in displaying handwritten characters.SOLUTION: An information processing apparatus includes: an input section capable of detecting handwritten input; a display section capable of displaying a trajectory of the handwritten input; a text recognition section which recognizes text on the basis of the handwritten input detected by the input section; a prediction processing section which predicts text candidates following the text on the basis of the text recognized by the text recognition section; a handwriting synthesis section which synthesizes the text candidates predicted by the prediction processing section into handwritten characters by combining first generation processing to generate handwritten characters on the basis of tokens, each of which is a set of one or more handwriting strokes, collected from the handwritten input by a user, and second generation processing to generate handwritten characters on the basis of a conversion model prepared in advance for converting text characters to handwritten characters; and a display processing section which displays, on the display section, the handwritten characters corresponding to the text candidates and synthesized by the handwriting synthesis section.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device and a control method. [Background technology]

[0002] In recent years, information processing devices such as notebook personal computers (hereinafter referred to as notebook PCs) and tablet terminals are known that allow handwritten input using a pen or a finger (see, for example, Patent Document 1). With such information processing devices, handwritten input can provide a user-friendly user interface, for example, allowing users to add handwritten notes to a text document to deepen their understanding of the document. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2019-507915 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional information processing devices, when input prediction is performed to improve productivity in character input and handwritten characters are output, handwritten characters are generated for each character, which takes time to process and causes delays before the characters are displayed.

[0005] The present invention has been made to solve the above problem, and an object of the present invention is to provide an information processing device and a control method that can reduce the delay in displaying handwritten characters. [Means for solving the problem]

[0006] In order to solve the above problem, one aspect of the present invention is an information processing device comprising: an input unit capable of detecting handwritten input; a display unit capable of displaying a trajectory of the handwritten input; a text recognition unit that recognizes text based on the handwritten input detected by the input unit; a prediction processing unit that predicts a text candidate that follows the text based on the text recognized by the text recognition unit; a handwriting synthesis unit that combines a first generation process that generates handwritten characters based on tokens collected by a user's handwritten input, the tokens being a set of one or more handwritings, and a second generation process that generates handwritten characters based on a pre-prepared conversion model that converts text characters into handwritten characters, and synthesizes the text candidates predicted by the prediction processing unit into handwritten characters; and a display processing unit that displays the handwritten characters synthesized by the handwriting synthesis unit and corresponding to the text candidates on the display unit.

[0007] In addition, one aspect of the present invention is the information processing device described above, further comprising a reference table storage unit that stores a reference table that associates token information indicating the token with text strings, and the handwriting synthesis unit generates the handwritten characters by the first generation process for character strings among the text candidates that match the text strings stored in the reference table storage unit, and generates the handwritten characters by the second generation process for characters among the text candidates that do not match the text strings stored in the reference table storage unit.

[0008] In another aspect of the present invention, in the above-mentioned information processing device, the handwriting synthesis unit may perform the following steps in the first generation process: extracting a string of N characters from the text candidate; and, if the extracted string matches the text string stored in the reference table storage unit, converting the text string into the handwritten character based on the token information corresponding to the matching text string (where N is an integer greater than or equal to 1); after the first process, subtracting 1 from N and repeating the first process until the number of N characters becomes 1 character; and after the second process, performing the third process to generate the handwritten character using the second generation process for the characters that did not match.

[0009] In addition, one aspect of the present invention may further include a token generation unit in the information processing device described above that collects the tokens based on a trajectory of the handwritten input, associates the token information with the text string corresponding to the token, and stores the associated information in the reference table storage unit.

[0010] In addition, one aspect of the present invention is that in the above-mentioned information processing device, the display processing unit displays display information indicating the text candidate predicted by the prediction processing unit on the display unit, and when the input unit receives an acceptance operation to accept the text candidate, the handwritten character corresponding to the text candidate synthesized by the handwriting synthesis unit may be displayed on the display unit.

[0011] In addition, one aspect of the present invention is that in the above-mentioned information processing device, the display processing unit displays the text candidate on the display unit in a specific font as display information indicating the text candidate, and when the input unit accepts the consent operation, the text candidate displayed in the specific font may be changed to the handwritten character synthesized by the handwriting synthesis unit.

[0012] Another aspect of the present invention is a control method for an information processing device having an input unit capable of detecting handwritten input and a display unit capable of displaying a trajectory of the handwritten input, the control method including: a step in which a text recognition unit recognizes text based on the handwritten input detected by the input unit; a step in which a prediction processing unit predicts a text candidate that follows the text recognized by the text recognition unit based on the text; a step in which the handwriting synthesis unit combines a first generation process in which handwritten characters are generated based on tokens collected by a user's handwritten input, the tokens being a set of one or more handwritings, and a second generation process in which handwritten characters are generated based on a pre-prepared conversion model that converts text characters into handwritten characters, thereby synthesizing the text candidate predicted by the prediction processing unit into handwritten characters; and a step in which a display processing unit displays the handwritten characters synthesized by the handwriting synthesis unit and corresponding to the text candidate on the display unit. [Effects of the Invention]

[0013] According to the present invention, it is possible to reduce the delay in displaying handwritten characters. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is an external view showing an example of a tablet terminal according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of a main hardware configuration of a tablet terminal according to the present embodiment. [Figure 3] FIG. 2 is a block diagram showing an example of the functional configuration of a tablet terminal according to the present embodiment. [Figure 4] 5A and 5B are diagrams illustrating an example of data stored in a reference table storage unit according to the present embodiment. [Figure 5] 10A and 10B are diagrams illustrating an example of handwritten character conversion using tokens according to the present embodiment. [Figure 6] 10A and 10B are diagrams illustrating an example of handwritten character conversion using a conversion model according to the present embodiment. [Figure 7]10A and 10B are diagrams illustrating an example of handwriting synthesis processing on a tablet terminal according to the present embodiment. [Figure 8] 10A and 10B are diagrams illustrating an example of a process for generating tokens and a reference table according to the present embodiment. [Figure 9] 10 is a flowchart illustrating an example of the operation of the tablet terminal according to the present embodiment. [Figure 10] FIG. 1 is a first diagram showing an example of the operation of the tablet terminal according to the present embodiment. [Figure 11] FIG. 2 is a second diagram showing an example of the operation of the tablet terminal according to the present embodiment. [Figure 12] 10 is a flowchart illustrating an example of handwriting synthesis processing on a tablet terminal according to the present embodiment. [Figure 13] 10A and 10B are diagrams illustrating examples of display in handwriting synthesis processing on a tablet terminal according to the present embodiment. [Figure 14] 10 is a flowchart illustrating an example of handwritten character conversion processing using tokens on a tablet terminal according to the present embodiment. [Figure 15] 10 is a flowchart illustrating an example of a token generation process of a tablet terminal according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] An information processing apparatus and a control method according to an embodiment of the present invention will be described below with reference to the drawings.

[0016] 1 is an external view showing an example of a tablet terminal 1 according to this embodiment. In this embodiment, the tablet terminal 1 will be described as an example of an information processing device.

[0017] As shown in FIG. 1, the tablet terminal 1 has a touch screen 20 installed on one main surface of a housing CS1, and uses a pen 30 to execute various processes, such as application programs that allow handwritten input.

[0018] The touch screen 20 includes a display unit 21 and a touch sensor unit 22, and the display unit 21 displays various types of information on a display screen DF. The touch sensor unit 22 is disposed over the display unit 21, and detects when a pen 30 (an example of an operation medium) comes into contact with the display screen DF of the display unit 21, and also detects the contact position of the pen 30.

[0019] The touch screen 20, the display unit 21, and the touch sensor unit 22 will be described in detail later.

[0020] Next, the main hardware configuration of the tablet terminal 1 will be described with reference to FIG. FIG. 2 is a diagram showing an example of the main hardware configuration of the tablet terminal 1 according to this embodiment.

[0021] As shown in FIG. 2, the tablet terminal 1 includes a processor 11, a main memory 12, a flash memory 13, a touch screen 20, a peripheral device 23, an audio system 24, a microphone 25, a speaker 26, a baseband chip 27, and a wireless unit 28.

[0022] The processor 11 is, for example, an application processor including a CPU (Central Processing Unit) 11. The processor 11 controls the tablet terminal 1 as a whole.

[0023] The main memory 12 is a writable memory used as a read area for the execution program of the processor 11 or as a work area for writing processing data for the execution program. The main memory 12 is composed of, for example, multiple DRAM (Dynamic Random Access Memory) chips. The execution program includes an OS (Operating System), various device drivers for operating peripheral devices, various services / utilities, application programs (application software), etc.

[0024] The flash memory 13 is, for example, a flash EEPROM (Electrically Erasable Programmable Read Only Memory), and stores an OS, various drivers, various services / utilities, application programs (hereinafter sometimes referred to as applications), and various data.

[0025] The display unit 21 is, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and performs display based on drawing data (display data) output from the processor 11.

[0026] The touch sensor unit 22 detects contact and contact position of the pen 30 (operation medium) on the screen (on the display screen DF) of the display unit 21. The touch sensor unit 22 can detect the position of the pen 30 on the screen, for example, by a capacitance method, an electromagnetic induction method, or the like. The touch sensor unit 22 detects detected position data (raw data) of the pen 30 at a predetermined detection interval (sampling period). The touch sensor unit 22 is an example of an input unit that can detect handwritten input by the pen 30.

[0027] In addition, the touch sensor unit 22 has a CPU (not shown) and memory units such as RAM and ROM inside, and the CPU executes firmware stored in the memory unit, thereby enabling various processes to be performed, such as removing noise from the detected position data of the pen 30.

[0028] The audio system 24 is, for example, an audio IC (Integrated Circuit), and inputs, records, plays back, and outputs sound data. For example, a microphone 25 and a speaker 26 are connected to the audio system 24. For example, the audio system 24 outputs sound data picked up by the microphone 25 to the processor 11 or the baseband chip 27. Furthermore, the audio system 24 converts sound data acquired from the processor 11 or the baseband chip 27 into a sound signal and outputs the sound signal to the speaker 26.

[0029] The microphone 25 picks up sounds around the tablet terminal 1. For example, the microphone 25 picks up sounds such as the user's voice when audio is integrated with other terminals. The speaker 26 outputs various sounds to the outside of the tablet terminal 1. For example, when audio integration with another terminal is performed, the speaker 26 outputs (emits) sounds received from the other terminal.

[0030] The baseband chip 27 is a dedicated IC that controls wireless communications such as 4G (fourth generation mobile communications system) and 5G (fifth generation mobile communications system). The baseband chip 27, for example, receives audio data using a wireless unit 28 and outputs it to a speaker 26 via an audio system 24. The baseband chip 27 also acquires, for example, audio data collected from a microphone 25 via the audio system 24 and outputs it via the mobile communications system using the wireless unit 28. The baseband chip 27 also communicates input and output data for data communications via the mobile communications system with the processor 11.

[0031] The radio unit 28 is a wireless communication device including an antenna for performing wireless communication via a mobile communication system.

[0032] The pen 30 is a pen-shaped operating medium, such as a touch pen, a stylus pen, etc. The pen 30 may be configured to include, for example, a resonant circuit, be supplied with power by electromagnetic induction to a coil of the resonant circuit, and be capable of detecting the position and angle of the pen 30 on the screen of the display unit 21 using the resonant circuit.

[0033] Next, the functional configuration of the tablet terminal 1 according to this embodiment will be described with reference to FIG. FIG. 3 is a block diagram showing an example of the functional configuration of the tablet terminal 1 according to this embodiment.

[0034] As shown in FIG. 3, the tablet terminal 1 includes a main control unit 10, a touch screen 20, a pen 30, and a storage unit 40. The touch screen 20 includes a display unit 21 and a touch sensor unit 22 .

[0035] The storage unit 40 is realized by, for example, the main memory 12 or the flash memory 13. The storage unit 40 includes a handwritten input storage unit 41, a predictive learning result storage unit 42, a synthesis information storage unit 43, a text history storage unit 44, and a reference table storage unit 45.

[0036] The handwriting input storage unit 41 is, for example, a storage unit realized by the main memory 12, and stores handwriting input data. The handwriting input data is, for example, pen input data that is the movement trajectory of the pen 30 based on detection position data of the pen 30 detected by the touch sensor unit 22.

[0037] The predictive learning result storage unit 42 is a storage unit realized by, for example, the flash memory 13, and stores the predictive learning result (next text prediction model). Note that the predictive learning result is a prediction model for predicting text candidates that follow input text (for example, a word, part of a sentence, etc.), and is the result of machine learning based on, for example, the user's past handwritten input or text input, fixed phrases, etc.

[0038] The synthesis information storage unit 43 is a storage unit realized by, for example, the flash memory 13, and stores a conversion model (handwriting synthesis model) that converts handwritten characters into other handwritten characters. The conversion model is a synthesis model for synthesizing input text characters with characters handwritten by the user, and is, for example, the result of machine learning based on the user's past handwritten inputs.

[0039] The text history storage unit 44 is a storage unit realized by, for example, the flash memory 13, and stores history information of past text inputs. The history information of text inputs stored in the text history storage unit 44 is used for learning the above-mentioned predictive learning result (next text prediction model).

[0040] The reference table storage unit 45 is a storage unit realized by, for example, the flash memory 13, and stores a reference table (lookup table) used when converting handwritten characters using tokens. Here, a token is collected by a user's handwritten input, and is a set of one or more handwritings (strokes), representing a complete character having one or more meanings. The reference table storage unit 45 stores, for example, a reference table that associates text strings of tokens with token information, as shown in FIG. 4.

[0041] FIG. 4 is a diagram showing an example of data stored in the reference table storage unit 45 in this embodiment. 4, the reference table storage unit 45 stores the text string of a token in association with token information. Here, the token information is information indicating the handwritten characters of the token, and includes, for example, encoded data that can be converted into the handwritten characters of the token by a decoder process described later. The token information may also include image data of the handwritten characters of the token as they are.

[0042] 4, the reference table storage unit 45 stores, for example, a token "to be or" (text string) in association with a corresponding handwritten character image and token information that is encoded data. Similarly, the reference table storage unit 45 stores, for the tokens "to," "that," and "the," in association with the text string and token information.

[0043] 3, the main control unit 10 is a functional unit realized, for example, by the processor 11 executing a program stored in the main memory 12 or the flash memory 13, and executes various processes based on an OS (for example, Android (registered trademark)). The main control unit 10 includes a handwriting input processing unit 101, a text recognition unit 102, a prediction processing unit 103, a handwriting synthesis unit 104, a display processing unit 105, a prediction learning unit 106, and a token generation unit 107.

[0044] The handwriting input processing unit 101 is a functional unit realized, for example, by the processor 11 executing a program stored in the main memory 12 or the flash memory 13, and executes handwriting input processing. The handwriting input processing unit 101 acquires detected position data of an operation medium such as the pen 30 from the touch sensor unit 22 of the touch screen 20, and displays a movement trajectory (handwritten input) of the pen 30 based on the acquired detected position data on the display unit 21. The handwriting input processing unit 101 causes the display processing unit 105, which will be described later, to display the movement trajectory (handwritten input) of the pen 30. The handwriting input processing unit 101 also stores the handwriting input data in the handwriting input storage unit 41.

[0045] The text recognition unit 102 is a functional unit that is realized, for example, by the processor 11 executing a program stored in the main memory 12 or the flash memory 13, and recognizes text based on handwritten input detected by the touch sensor unit 22. The text recognition unit 102 converts the handwritten input data into text data using, for example, a known handwritten character recognition technique.

[0046] Furthermore, the text recognition unit 102 stores the converted text data in the text history storage unit 44 as history information of text input.

[0047] The prediction processing unit 103 is a functional unit realized, for example, by the processor 11 executing a program stored in the main memory 12 or the flash memory 13, and predicts a text candidate that follows text recognized by the text recognition unit 102 based on the text. The prediction processing unit 103 predicts a text candidate that will be input next based on the text recognized by the text recognition unit 102, for example, using the predictive learning result (next text prediction model) stored in the predictive learning result storage unit 42.

[0048] That is, the prediction processing unit 103 predicts text candidates based on the input history of handwritten inputs made by the user in the past. The prediction processing unit 103 may predict text candidates on a word-by-word basis, or on a sentence-by-sentence basis. The prediction processing unit 103 may also predict text candidates on a character-by-character basis, a phrase-by-phrase basis, a numeric string basis (e.g., a telephone number), etc. In other words, the text candidates may be other content including numbers, symbols, characters, phrases, and sentences. Furthermore, the prediction processing unit 103 may predict, for example, a plurality of text candidates.

[0049] The handwriting synthesis unit 104 is a functional unit realized, for example, by the processor 11 executing a program stored in the main memory 12 or the flash memory 13, and synthesizes a text candidate (content of the text candidate) predicted by the prediction processing unit 103 into a handwritten character. The handwriting synthesis unit 104 combines a token-based process (first generation process) that generates handwritten characters based on tokens and a model-based process (second generation process) that generates handwritten characters based on a prepared conversion model that converts text characters into handwritten characters, to synthesize the text candidate predicted by the prediction processing unit 103 into a handwritten character.

[0050] The handwriting synthesis process that combines this token-based process (first generation process) and model-based process (second generation process) is called hybrid process. Tokens are collected from handwritten input by a user, and are a set of one or more handwritings as explained above with reference to FIG. 4. Model-based process is a conversion process of handwritten characters using an existing conversion model, and converts characters into handwritten characters, for example, character by character.

[0051] The handwriting synthesis unit 104 generates handwritten characters by token-based processing for character strings among the text candidates that match text strings stored in the lookup table storage unit 45. The handwriting synthesis unit 104 also generates handwritten characters by model-based processing for characters among the text candidates that do not match text strings stored in the lookup table storage unit 45. In the model-based processing, the handwriting synthesis unit 104 generates handwritten characters for each character using a conversion model stored in the synthesis information storage unit 43.

[0052] Here, handwritten character conversion by token-based processing in the handwriting synthesis unit 104 will be described with reference to FIG. FIG. 5 is a diagram illustrating an example of handwritten character conversion using tokens in this embodiment.

[0053] The example shown in Figure 5 illustrates the process in which the handwriting synthesis unit 104 converts the text string Tx1 ("be"), which is a text candidate, after the handwritten input handwritten character HW1 ("Legally") into the handwritten character HW2 using a token.

[0054] The handwriting synthesis unit 104 acquires token information TD1 (encoded data) corresponding to the text string Tx1 ("be") using the reference table LT1 stored in the reference table storage unit 45. The handwriting synthesis unit 104 decodes the token information TD1 (encoded data) and converts it into handwritten characters HW2 corresponding to "be". The display processing unit 105, which will be described later, displays the handwritten characters HW1 ("Legally") on the display screen DF of the display unit 21, and then displays the handwritten characters HW2 converted by the handwriting synthesis unit 104 through token-based processing.

[0055] Next, handwritten character conversion by the model-based processing of the handwriting synthesis unit 104 will be described with reference to FIG. FIG. 6 is a diagram illustrating an example of handwritten character conversion using a conversion model according to this embodiment.

[0056] The example shown in FIG. 6 illustrates a process (handwriting conversion process) in which the handwriting synthesis unit 104 converts a text string Tx2 ("the") into handwritten characters HW3 using a conversion model.

[0057] The handwriting synthesis unit 104 separates the text string Tx2 ("the") into text characters Tx21 ("t"), text characters Tx22 ("h"), and text characters Tx23 ("e"). The handwriting synthesis unit 104 converts the text characters Tx21 ("t"), text characters Tx22 ("h"), and text characters Tx23 ("e") character by character into handwritten characters HW31, handwritten characters HW32, and handwritten characters HW33. The handwriting synthesis unit 104 synthesizes the handwritten characters HW31, handwritten characters HW32, and handwritten characters HW33 to generate the handwritten character HW3.

[0058] Next, handwriting synthesis by hybrid processing in handwriting synthesis unit 104 will be described with reference to FIG. FIG. 7 is a diagram showing an example of handwriting synthesis processing of the tablet terminal 1 according to this embodiment.

[0059] The example shown in FIG. 7 shows an example in which the handwriting synthesis unit 104 converts the text string "juice" into handwritten characters HW4 by hybrid processing. 7, the handwriting synthesis unit 104 generates a four-character string (four-character string CG2) of "juic" and "uice" from the text string "juice," which is a five-character string (five-character string CG1), and determines whether either of the four character strings, "juic" or "uice," exists in the lookup table stored in the lookup table storage unit 45. If either of the four character strings exists in the lookup table, the handwriting synthesis unit 104 converts it into handwritten characters (four handwritten characters) using token information corresponding to the existing four character strings.

[0060] Furthermore, when there is a character example that has not been converted by the above-described token-based processing, the handwriting synthesis unit 104 generates a three-character string of "jui", "uic", and "ice" (a three-character string CG3) from a four-character string of "juic" and "uice" (a four-character string CG2). The handwriting synthesis unit 104 determines whether any of the three character strings "jui", "uic", and "ice" exists in the lookup table stored in the lookup table storage unit 45. If the three character strings exist in the lookup table, the handwriting synthesis unit 104 converts them into handwritten characters (three handwritten characters) using token information corresponding to the existing three character strings.

[0061] Furthermore, when there is a character example that has not been converted by the above-described token-based processing, the handwriting synthesis unit 104 generates a two-character character string of "ju", "ui", "ic", and "ce" (two-character character string CG4) from a three-character character string of "jui", "uic", and "ice" (three-character string CG3). The handwriting synthesis unit 104 determines whether any two character strings of "ju", "ui", "ic", and "ce" exist in the lookup table stored in the lookup table storage unit 45. If they exist in the lookup table, the handwriting synthesis unit 104 converts them into handwritten characters (two handwritten characters) using token information corresponding to the two existing character strings.

[0062] In the example shown in FIG. 7, it is assumed that "ui" exists in the reference table, and the token information corresponding to "ui" is decoded to generate handwritten characters HW42 corresponding to "ui". In addition, the handwriting synthesis unit 104 uses model-based processing (handwriting conversion processing) for the remaining text characters "j", "c", and "e" to generate handwritten characters HW41, HW43, and HW44 corresponding to the text characters "j", "c", and "e", respectively.

[0063] The handwriting synthesis unit 104 combines the generated handwritten characters HW41, HW42, HW43, and HW44 to generate a handwritten character HW4 corresponding to "juice."

[0064] In this way, in the token-based processing, the handwriting synthesis unit 104 extracts a string of N characters from the text candidates, and if the extracted string matches a text string stored in the lookup table storage unit 45, executes a first process (where N is an integer equal to or greater than 1) to convert the text string into handwritten characters based on token information corresponding to the matching text string. After the first process, the handwriting synthesis unit 104 then executes a second process in which it subtracts 1 from N and repeats the first process until the number of N characters becomes 1 character.

[0065] Furthermore, after the second process, the handwriting synthesis unit 104 executes a third process for generating handwritten characters from characters that do not match using a model-based process. That is, the handwriting synthesis unit 104 executes the first to third processes described above to synthesize text candidates into handwritten characters through hybrid processing that combines token-based processing and model-based processing.

[0066] Returning to the explanation of Figure 3, the display processing unit 105 is a functional unit that is realized, for example, by the processor 11 executing a program stored in the main memory 12 or the flash memory 13, and performs processing to display handwritten input, handwritten characters converted from text candidates, etc. on the display unit 21.

[0067] The display processing unit 105 displays, for example, the handwritten input acquired by the handwriting input processing unit 101 on the display unit 21. Furthermore, the display processing unit 105 displays, for example, a plurality of text candidates predicted by the prediction processing unit 103 together as display information indicating the predicted text candidates on the display unit 21. Furthermore, the display processing unit 105 displays, on the display unit 21, handwritten characters that correspond to a text candidate selected from the plurality of text candidates and that have been synthesized by the handwriting synthesis unit 104.

[0068] Furthermore, when displaying handwritten characters corresponding to text candidates, the display processing unit 105 checks the position and size of the handwritten characters to be displayed based on the bounding box of the handwritten input acquired by the handwriting input processing unit 101, and displays the handwritten characters at an appropriate position and size. Here, the bounding box of the handwritten input refers to a box-shaped (rectangular) display area that surrounds the handwritten input.

[0069] The display processing unit 105 may display display information indicating the text candidate predicted by the prediction processing unit 103 on the display unit 21, and may display the handwritten characters synthesized by the handwriting synthesis unit 104 as display information indicating the text candidate on the display unit 21 using lines of a color different from the trajectory of the handwritten input. In this case, the display processing unit 105 displays the handwritten characters of the text candidate on the display unit 21 using gray lines, for example, and when an acceptance operation to accept the text candidate is received by operating the touch sensor unit 22, changes the gray lines of the handwritten input candidate to black lines and displays the handwritten characters HW2.

[0070] In this way, the display processing unit 105 displays display information indicating the text candidate predicted by the prediction processing unit 103 on the display unit 21, and when the touch sensor unit 22 receives an acceptance operation to accept the text candidate, the handwritten characters corresponding to the text candidate synthesized by the handwriting synthesis unit 104 are displayed on the display unit 21.

[0071] In addition, the display processing unit 105 may display the text candidate on the display unit 21 in a specific font as display information indicating the text candidate, and when the touch sensor unit 22 receives an acceptance operation, the text candidate displayed in the specific font may be changed to handwritten characters synthesized by the handwriting synthesis unit 104.

[0072] The predictive learning unit 106 is a functional unit that is realized, for example, by the processor 11 executing a program stored in the main memory 12 or the flash memory 13. The predictive learning unit 106 executes machine learning processing using, for example, text history information stored in the text history storage unit 44 as learning data, and stores the learning result (next text prediction model) in the predictive learning result storage unit 42.

[0073] The token generation unit 107 is a functional unit that is realized, for example, by the processor 11 executing a program stored in the main memory 12 or the flash memory 13, and collects tokens from the user's handwritten input, associates the text strings of the collected tokens with token information, and stores the associations as a reference table in the reference table storage unit 45. Here, the process of generating tokens and a reference table by the token generating unit 107 will be described with reference to FIG.

[0074] FIG. 8 is a diagram showing an example of a process for generating tokens and a reference table in this embodiment. The example shown in FIG. 8 explains an example in which tokens are collected from handwritten characters HW5 ("quick") input by a user and a reference table is generated.

[0075] As shown in FIG. 8, the token generation unit 107 extracts tokens of handwritten character HW51 ("q"), handwritten character HW52 ("ui"), and handwritten character HW53 ("ck") from the handwritten character HW1 handwritten and input by the user through a token generation process, and decodes the handwritten character HW51, handwritten character HW52, and handwritten character HW53, respectively, to generate token information (decoded data) corresponding to each of them.

[0076] The token generation unit 107 also generates a reference table LT2 that associates the text strings of each token (text string "q", text string "ui", text string "ck") converted by the text recognition unit 102 with each piece of token information. The token generation unit 107 stores the generated reference table LT2 in the reference table storage unit 45.

[0077] In this way, the token generating unit 107 collects tokens based on the trajectory (ink stroke) of handwritten input, associates the token information with the text string corresponding to the token, and stores the information in the look-up table storage unit 45.

[0078] Next, the operation of the tablet terminal 1 according to this embodiment will be described with reference to the drawings. FIG. 9 is a flowchart showing an example of the operation of the tablet terminal 1 according to this embodiment.

[0079] As shown in FIG. 9, the tablet terminal 1 first determines whether or not contact of the pen 30 with the screen (display screen DF) has been detected (step S101). The handwriting input processing unit 101 of the tablet terminal 1 determines whether or not the touch sensor unit 22 has detected contact of the pen 30 with the display screen DF. If the touch sensor unit 22 has detected contact of the pen 30 (step S101: YES), the handwriting input processing unit 101 proceeds to step S102. If the touch sensor unit 22 has not detected contact of the pen 30 (step S101: NO), the handwriting input processing unit 101 returns the process to step S101.

[0080] In step S102, the handwriting input processing unit 101 acquires pen input data. That is, the handwriting input processing unit 101 acquires, as pen input data, position information of the pen 30 on the display screen DF detected by the touch sensor unit 22.

[0081] Next, the handwriting input processing unit 101 displays the trajectory of the handwritten pen input (step S103). The handwriting input processing unit 101 stores the acquired pen input data as a handwritten pen input in the handwriting input storage unit 41, and uses the display processing unit 105 to display the trajectory of the handwritten pen input on the display unit 21, for example, as in handwriting input HW6 shown in Fig. 10, which will be described later.

[0082] Next, the handwriting input processing unit 101 determines whether the pen input is complete (step S104). If the pen input is complete (step S104: YES), the handwriting input processing unit 101 proceeds to step S105. If the pen input is not complete (step S104: NO), the handwriting input processing unit 101 returns to step S101.

[0083] In step S105, the tablet terminal 1 converts the handwritten input into text data (text information). The text recognition unit 102 of the tablet terminal 1 converts, for example, the handwritten input HW6 shown in FIG. 10 into text data such as "how are." The text recognition unit 102 stores the converted text data in the text history storage unit 44 as history information of the text input.

[0084] Next, tablet terminal 1 predicts text candidates that follow the converted text data (step S106). Prediction processing unit 103 of tablet terminal 1 predicts text candidates that follow the text data converted by text recognition unit 102, for example, using the predictive learning result (next text prediction model) stored in predictive learning result storage unit 42. Prediction processing unit 103 predicts, for example, multiple text candidates.

[0085] Next, tablet terminal 1 displays a plurality of text candidates on display unit 21 (step S107). Display processing unit 105 of tablet terminal 1 displays a plurality of text candidates side by side on display unit 21, for example, as shown in display information (candidate CW1) in FIG.

[0086] FIG. 10 is a first diagram showing an example of the operation of the tablet terminal 1 according to this embodiment. The example shown in FIG. 10 shows a display example when the user performs handwriting input HW6 using the touch sensor unit 22.

[0087] As shown in Figure 10, the display processing unit 105 displays the user's handwritten input HW6 ("how are") on the display unit 21, and also displays multiple text candidates ("you", "we", "they") side by side vertically, as shown in candidate CW1 (display information).

[0088] The display processing unit 105 may sort the multiple text candidates CW1 based on probability or other factors and display them on the display unit 21, or may display the multiple text candidates CW1 in order of likelihood of selection.

[0089] Next, the tablet terminal 1 determines whether a text candidate has been selected (step S108). The display processing unit 105 determines whether one of the multiple text candidates (for example, "you," "we," and "they" shown in FIG. 10) has been selected by the user's operation of the touch sensor unit 22. If a text candidate has been selected (step S108: YES), the display processing unit 105 proceeds to step S110. If a text candidate has not been selected (step S108: NO), the display processing unit 105 proceeds to step S109. Note that the operation of the user operating the touch sensor unit 22 to select one of the multiple text candidates is an example of an acceptance operation for accepting the text candidate.

[0090] In step S109, the tablet terminal 1 determines whether or not contact with the screen (display screen DF) by the pen 30 other than for selecting a candidate has been detected. If the touch sensor unit 22 detects contact by the pen 30 other than for selecting a candidate (step S109: YES), the handwriting input processing unit 101 proceeds to step S102. If the handwriting input processing unit 101 has not detected contact by the pen 30 other than for selecting a candidate (step S109: NO), the handwriting input processing unit 101 returns the process to step S108.

[0091] In step S110, tablet terminal 1 synthesizes the text candidate with handwritten characters. Handwriting synthesis unit 104 of tablet terminal 1 synthesizes the text candidate predicted by prediction processing unit 103 with handwritten characters (for example, handwritten characters HW7 shown in FIG. 11) using hybrid processing that combines the token-based processing and model-based processing described above.

[0092] Next, the tablet terminal 1 updates the bounding box information of the current handwriting input trajectory (step S111). The display processing unit 105 of the tablet terminal 1 updates, for example, the position and size of the bounding box of the handwriting input HW6 (a box-shaped (rectangular) display area surrounding the handwriting input HW6) shown in FIG.

[0093] Next, the display processing unit 105 checks the position and size of the text candidate to be displayed (step S112). For example, the display processing unit 105 checks the position and size of the handwritten character HW7 to be displayed based on the position and size of the bounding box of the updated handwritten input HW6.

[0094] Next, the display processing unit 105 displays the handwritten character and confirms the handwritten input (step S113). The display processing unit 105 displays the handwritten character, for example, like the handwritten character HW7 shown in FIG. 11, and confirms the handwritten input.

[0095] FIG. 11 is a second diagram showing an example of the operation of the tablet terminal 1 according to this embodiment. The example shown in FIG. 11 shows a display example in which the text candidate "you" is selected from the plurality of text candidates shown in FIG. 10, and handwritten characters HW7 corresponding to the text candidate "you" are displayed.

[0096] As shown in FIG. 11, the display processing unit 105 displays on the display unit 21 handwritten characters HW7 that the handwriting synthesis unit 104 has converted from the selected text candidate "you" through hybrid processing. After the process of step S113, the display processing unit 105 returns the process to step S101.

[0097] Next, the detailed operation of the handwritten synthesis process in this embodiment will be described with reference to FIGS. FIG. 12 is a flowchart showing an example of handwriting synthesis processing of the tablet terminal 1 according to this embodiment.

[0098] 12, the handwriting synthesis unit 104 of the tablet terminal 1 first refers to the reference table and extracts matching tokens from the target text (e.g., text candidates) (step S201). The handwriting synthesis unit 104 extracts tokens included in the target text from the reference table stored in the reference table storage unit 45.

[0099] Next, the handwriting synthesis unit 104 generates handwritten characters corresponding to the matched tokens based on the lookup table (step S202). The handwriting synthesis unit 104 converts each token into handwritten characters using token-based processing. The handwriting synthesis unit 104 decodes the corresponding token information to generate handwritten characters.

[0100] Next, the handwriting synthesis unit 104 converts the remaining characters into handwritten characters using the conversion model (step S203). The handwriting synthesis unit 104 converts the remaining characters that do not match the tokens into handwritten characters one by one using a model-based process.

[0101] Next, the handwriting synthesis unit 104 generates a final synthesized handwritten character corresponding to the target text. The handwriting synthesis unit 104 synthesizes the handwritten character generated in the process of step S202 (token-based process) and the handwritten character generated in the process of step S203 (model-based process), and outputs the final handwritten character (handwritten character corresponding to the target text).

[0102] 13 is a diagram showing an example of handwriting synthesis processing by the tablet terminal 1 according to this embodiment. The example shown in Fig. 13 shows an example in which a text candidate CW2 ("not to be") is converted into a handwritten character HW3 for a handwritten character HW8 ("to be or") handwritten by a user.

[0103] In FIG. 13, the display processing unit 105 first displays the handwritten character HW8 ("to be or") and the text character string of the text candidate CW2 ("not to be") on the display unit 21 as shown in image G1.

[0104] Next, the handwriting synthesis unit 104 extracts, for example, a token TK1 ("to") and a token TK2 ("be") as shown in image G2 of FIG. Next, the handwriting synthesis unit 104 converts the token TK1 ("to") and the token TK2 ("be") into handwritten characters HW91 and HW92 using token-based processing, and the display processing unit 105 displays the handwritten characters HW91 and HW92 on the display unit 21, as shown in image G3 of Figure 13.

[0105] Next, the handwriting synthesis unit 104 generates handwritten character HW93 for the remaining character ("not") using model-based processing, and the display processing unit 105 displays the handwritten character HW93 on the display unit 21 as shown in image G4 of Figure 13. Finally, the display processing unit 105 displays the handwritten character HW9 corresponding to the text candidate CW2 ("not to be") on the display unit 21, as shown in image G5 in FIG.

[0106] 13, an example has been described in which the display processing unit 105 displays the text candidate CW2 on the display unit 21. However, the text candidate CW2 does not have to be displayed. Furthermore, the display processing unit 105 may be configured to display the handwritten characters HW9 on the display unit 21 all at once after generating the handwritten characters HW9.

[0107] Next, with reference to FIG. 14, the handwritten character conversion process using the token-based process will be described in detail. 14 is a flowchart showing an example of handwritten character conversion processing using tokens in the tablet terminal 1 according to this embodiment. The processing shown in FIG. 14 corresponds to the processing from step S201 to step S203 shown in FIG. 12 described above.

[0108] 14, the handwriting synthesis unit 104 of the tablet terminal 1 acquires a character string S to be synthesized (step S301). Here, the character string S is a text character string such as a text candidate.

[0109] Next, the handwriting synthesis unit 104 sets the number of characters N to the number of characters in the character string S (N=Len(S)) (step S302). Note that an integer of 1 or greater can be set (assigned) to the number of characters N.

[0110] Next, the handwriting synthesis unit 104 determines whether the number of characters N is 1 or more (N≧1) (step S303). If the number of characters N is 1 or more (N≧1) (step S303: YES), the handwriting synthesis unit 104 proceeds to step S304. If the number of characters N is not 1 or more (N=0) (step S303: NO), the handwriting synthesis unit 104 ends the process.

[0111] In step S304, the handwriting synthesis unit 104 extracts sub-character strings of N characters from the character string S. For example, the handwriting synthesis unit 104 extracts sub-character strings such as five character strings CG1 to two character strings CG4 shown in FIG.

[0112] Next, the handwriting synthesis unit 104 checks whether the extracted sub-character string exists in the reference table (step S305). The handwriting synthesis unit 104 refers to the reference table stored in the reference table storage unit 45 and checks whether there is a text character example that matches the extracted sub-character string.

[0113] Next, the handwriting synthesis unit 104 determines whether the sub-character strings are available using the reference table (step S306). That is, the handwriting synthesis unit 104 determines whether each sub-character string is registered as a token in the reference table. If the reference table determines that the sub-character strings are available (step S306: YES), the handwriting synthesis unit 104 proceeds to step S307. If the reference table determines that the sub-character strings are not available (the sub-character strings are not registered as tokens in the reference table) (step S306: NO), the handwriting synthesis unit 104 proceeds to step S309.

[0114] In step S307, the handwriting synthesis unit 104 saves the sub-strings in a list for later use. For example, the handwriting synthesis unit 104 stores the list of each sub-string in the storage unit 40.

[0115] Next, the handwriting synthesis unit 104 replaces the available substrings with handwritten characters (step S308). The handwriting synthesis unit 104 uses a lookup table to obtain token information corresponding to the substrings, and decodes the token information to generate handwritten characters corresponding to the substrings (tokens). After processing step S308, the handwriting synthesis unit 104 proceeds to step S309.

[0116] In step S309, the handwriting synthesis unit 104 determines whether any unused sub-character strings remain. If any unused sub-character strings remain (step S309: YES), the handwriting synthesis unit 104 proceeds to step S310. If any unused sub-character strings remain (step S309: NO), the handwriting synthesis unit 104 ends the process.

[0117] In step S310, the handwriting synthesis unit 104 subtracts "1" from the number of characters N (N=N-1). After processing in step S310, the handwriting synthesis unit 104 returns the process to step S303 and repeats the processes from step S303 to step S310 until the number of characters N becomes "0".

[0118] Note that a substring with a character count N of "1" corresponds to one text character, and the handwriting synthesis unit 104 converts the remaining one text character to which token-based processing could not be applied into a handwritten character using model-based processing.

[0119] Next, with reference to FIG. 15, the token generation process of the tablet terminal 1 according to this embodiment will be described in detail. FIG. 15 is a flowchart showing an example of a token generation process of the tablet terminal 1 according to this embodiment.

[0120] 15, the token generation unit 107 of the tablet terminal 1 acquires target ink strokes (handwriting of the pen 30) (step S401). The target ink strokes (handwriting of the pen 30) here are ink strokes (handwriting of the pen 30) handwritten and input by the user.

[0121] Next, the token generator 107 classifies the ink strokes (step S402). The token generator 107 classifies whether or not the ink strokes correspond to a character (or a character string), and extracts ink strokes that correspond to the character (or a character string).

[0122] Next, the token generating unit 107 sets the first stroke (handwriting) as the start (starting stroke of the token) (step S403).

[0123] Next, the token generating unit 107 examines each stroke from the start to the current stroke (step S404). The token generating unit 107 analyzes each stroke and examines whether each stroke represents a complete character.

[0124] Next, the token generating unit 107 determines whether the stroke represents a complete character (step S405). If the stroke represents a complete character (step S405: YES), the token generating unit 107 proceeds to step S406. If the stroke does not represent a complete character (step S405: NO), the token generating unit 107 proceeds to step S407.

[0125] In step S406, the token generation unit 107 generates a token for the stroke. The token generation unit 107 associates a text character string corresponding to the stroke with token information (encoded data) obtained by encoding the stroke, and adds the associated data to the reference table in the reference table storage unit 45. That is, the token generation unit 107 associates a text character string corresponding to the stroke with the token information, and stores the associated data in the reference table storage unit 45. After processing in step S406, the token generation unit 107 proceeds to step S407.

[0126] In step S407, the token generating unit 107 determines whether there are any remaining strokes. If there are any remaining strokes (step S407: YES), the token generating unit 107 proceeds to step S408. If there are any remaining strokes (step S407: NO), the token generating unit 107 ends the process.

[0127] In step S408, the token generating unit 107 sets the next stroke as the start (the starting stroke of the token). After the process of step S408, the token generating unit 107 returns the process to step S404 and repeats the processes of steps S404 to S408 until there are no remaining strokes.

[0128] As described above, the tablet terminal 1 (information processing device) according to this embodiment includes a touch sensor unit 22 (input unit), a display unit 21, a text recognition unit 102, a prediction processing unit 103, a handwriting synthesis unit 104, and a display processing unit 105. The touch sensor unit 22 (input unit) is capable of detecting handwritten input. The display unit 21 is capable of displaying the trajectory (ink strokes) of the handwritten input. The text recognition unit 102 recognizes text based on the handwritten input detected by the touch sensor unit 22. The prediction processing unit 103 predicts text candidates that follow the text recognized by the text recognition unit 102 based on the text. The handwriting synthesis unit 104 combines a first generation process (e.g., token-based processing) and a second generation process (e.g., model-based processing) to synthesize the text candidates predicted by the prediction processing unit 103 into handwritten characters. In the first generation process, the handwriting synthesis unit 104 generates handwritten characters based on tokens collected from the user's handwritten input, each token being a set of one or more handwritings. In the second generation process, the handwriting synthesis unit 104 generates handwritten characters based on a conversion model prepared in advance for converting text characters into handwritten characters. The display processing unit 105 displays the handwritten characters corresponding to the text candidates synthesized by the handwriting synthesis unit 104 on the display unit 21.

[0129] As a result, the tablet terminal 1 (information processing device) according to this embodiment combines a first generation process using tokens (e.g., token-based processing) with a second generation process using a conventional conversion model (e.g., model-based processing) to synthesize text candidates into handwritten characters, thereby reducing the delay in displaying handwritten characters.

[0130] For example, in the example shown in FIG. 7 where the text string "juice" is converted into handwritten characters and displayed, when only the conventional technique (the second generation process using a conversion model (e.g., model-based processing)) is used, the display delay time is 0.56 seconds to 0.59 seconds. In contrast, when the hybrid processing technique according to the present embodiment described above (a technique that combines the first generation process using tokens (e.g., token-based processing) and the second generation process using a conversion model (e.g., model-based processing)) is used, the display delay time is reduced to about half, at 0.17 seconds to 0.25 seconds, and the display delay of handwritten characters is significantly improved. In this way, the tablet terminal 1 according to the present embodiment can significantly reduce the display delay of handwritten characters. Therefore, the tablet terminal 1 according to the present embodiment can improve the productivity of handwritten input.

[0131] Furthermore, since the tokens are collected through handwritten input by the user, the tablet terminal 1 according to this embodiment can faithfully reproduce the user's handwritten characters by using the tokens.

[0132] The tablet terminal 1 according to this embodiment also includes a reference table storage unit 45 that stores a reference table (lookup table) that associates token information indicating tokens with text character strings. The handwriting synthesis unit 104 generates handwritten characters from character strings among the text candidates that match the text character strings stored in the reference table storage unit 45 by a first generation process (e.g., token-based processing). The handwriting synthesis unit 104 also generates handwritten characters from character strings among the text candidates that do not match the text character strings stored in the reference table storage unit 45 by a second generation process (e.g., model-based processing).

[0133] As a result, the tablet terminal 1 according to this embodiment can generate handwritten characters by a simple method using a reference table (lookup table) and the first generation process (for example, token-based process) at high speed. Therefore, the tablet terminal 1 according to this embodiment can further reduce the display delay of handwritten characters and further improve the productivity of handwritten input.

[0134] In this embodiment, the handwriting synthesis unit 104 executes a first process, a second process, and a third process to synthesize a text candidate into a handwritten character. As the first process, the handwriting synthesis unit 104 extracts a character string of N characters from a text candidate in a first generation process (e.g., a token-based process), and if the extracted character string matches a text string stored in the lookup table storage unit 45, converts the text string into a handwritten character based on token information corresponding to the matching text string (where N is an integer equal to or greater than 1). As the second process, the handwriting synthesis unit 104 subtracts 1 from N after the first process and repeats the first process until the number of N characters becomes 1 character. As the third process, the handwriting synthesis unit 104 generates a handwritten character from characters that did not match after the second process using a second generation process.

[0135] As a result, the tablet terminal 1 according to this embodiment can efficiently extract tokens from text candidates by executing the above-described first to third processes. Therefore, the tablet terminal 1 according to this embodiment can further reduce the display delay of handwritten characters, and can further improve the productivity of handwritten input.

[0136] The tablet terminal 1 according to this embodiment also includes a token generating unit 107 that collects tokens based on the trajectory of handwritten input, associates the token information with the text string corresponding to the token, and stores the associated information in the reference table storage unit 45.

[0137] As a result, the tablet terminal 1 according to this embodiment is equipped with the token generation unit 107, and is therefore able to extract tokens from the user's handwritten input and appropriately update the reference table. Therefore, the tablet terminal 1 according to this embodiment can generate appropriate handwritten characters according to the user.

[0138] Furthermore, in this embodiment, the display processing unit 105 displays display information indicating the text candidate predicted by the prediction processing unit 103 on the display unit 21, and when the touch sensor unit 22 receives an acceptance operation to accept the text candidate (for example, an operation to select one candidate from multiple text candidates), it displays the handwritten characters corresponding to the text candidate synthesized by the handwriting synthesis unit 104 on the display unit 21.

[0139] As a result, the tablet terminal 1 according to this embodiment can display text candidates and confirm them by accepting an acceptance operation, thereby improving the input efficiency of handwritten input and improving the productivity of handwritten input.

[0140] In addition, in this embodiment, the display processing unit 105 displays the text candidate in a specific font on the display unit 21 as display information indicating the text candidate, and when the touch sensor unit 22 receives an acceptance operation, the text candidate displayed in the specific font may be changed to handwritten characters synthesized by the handwriting synthesis unit 104.

[0141] As a result, the tablet terminal 1 according to this embodiment displays text candidates in a specific font on the display unit 21, and therefore handwritten characters can be generated in the background while the text candidates are being displayed. As a result, the tablet terminal 1 according to this embodiment can further reduce the delay in displaying handwritten characters.

[0142] Furthermore, a control method according to this embodiment is a control method for a tablet terminal 1 including a touch sensor unit 22 capable of detecting handwritten input and a display unit 21 capable of displaying the trajectory of the handwritten input, and includes a text recognition step, a prediction processing step, a handwriting synthesis step, and a display processing step. In the text recognition step, the text recognition unit 102 recognizes text based on the handwritten input detected by the touch sensor unit 22. In the prediction processing step, the prediction processing unit 103 predicts a text candidate that follows the text recognized by the text recognition unit 102 based on the text. In the handwriting synthesis step, the handwriting synthesis unit 104 combines a first generation process (e.g., token-based processing) that generates handwritten characters based on tokens collected from a user's handwritten input, each token being a set of one or more handwritings, with a second generation process (e.g., model-based processing) that generates handwritten characters based on a pre-prepared conversion model that converts text characters into handwritten characters, thereby synthesizing the text candidate predicted by the prediction processing unit 103 into handwritten characters. In the display processing step, the display processing unit displays on the display unit 21 the handwritten characters that are synthesized by the handwriting synthesis unit 104 and correspond to the text candidates.

[0143] As a result, the control method according to this embodiment has the same effect as the tablet terminal 1 described above, and can significantly reduce the delay in displaying handwritten characters, thereby improving the productivity of handwritten input.

[0144] The tablet terminal 1 (information processing device) according to the above-described embodiment may also be in the following form. The tablet terminal 1 (information processing device) includes a touch sensor unit 22 capable of detecting handwritten input, a display unit 21 capable of displaying the trajectory of the handwritten input, a main memory 12 (memory) that temporarily stores programs, and a processor 11 that executes the programs stored in the main memory 12. The processor 11 executes the programs stored in the main memory 12 to perform text recognition processing, prediction processing, handwriting synthesis processing, and display processing. As the text recognition processing, the processor 11 recognizes text based on the handwritten input detected by the touch sensor unit 22. As the prediction processing, the processor 11 predicts text candidates that follow the text based on the text recognized in the text recognition processing. As a handwriting synthesis process, processor 11 combines a first generation process (e.g., token-based process) that generates handwritten characters based on tokens collected by a user's handwriting input, each token being a set of one or more handwritings, with a second generation process (e.g., model-based process) that generates handwritten characters based on a pre-prepared conversion model that converts text characters into handwritten characters, to synthesize text candidates predicted by the prediction process into handwritten characters.As a display process, processor 11 displays on display unit 21 the handwritten characters corresponding to the text candidates synthesized by the handwriting synthesis process using a display processing unit.

[0145] As a result, the tablet terminal 1 according to this embodiment can significantly reduce the delay in displaying handwritten characters, thereby improving the productivity of handwritten input.

[0146] The present invention is not limited to the above-described embodiments, and can be modified within the scope of the present invention. For example, in each of the above embodiments, the information processing device is described as a tablet terminal 1, but the present invention is not limited to this. The information processing device may be, for example, a smartphone or a notebook personal computer equipped with a tablet mode.

[0147] Furthermore, in the above embodiment, an example was described in which the OS was Android (registered trademark), but this is not limited to this, and other OSs such as Windows (registered trademark) and iOS (registered trademark) may also be used.

[0148] Furthermore, in the above embodiment, the example has been described in which the operation medium is the pen 30, but this is not limitative and other operation mediums such as the user's finger may also be used.

[0149] Furthermore, in the above embodiment, an example was described in which the tablet terminal 1 is equipped with a predictive learning unit 106 and a token generation unit 107, but this is not limited to this, and the tablet terminal 1 may be configured to not have either or both of the predictive learning unit 106 and the token generation unit 107.

[0150] Furthermore, in each of the above embodiments, an example has been described in which the prediction processing unit 103 predicts text candidates using the predictive learning results (next text prediction model), but this is not limited to this, and other methods may be used, such as storing past inputs and predicting inputs similar to the past inputs as text candidates.

[0151] Each of the components of the tablet terminal 1 described above has an internal computer system. A program for implementing the functions of each of the components of the tablet terminal 1 described above may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform processing in each of the components of the tablet terminal 1 described above. Here, "reading a program recorded on a recording medium into a computer system and executing it" includes installing the program into a computer system. The term "computer system" here includes hardware such as an OS and peripheral devices.

[0152] Furthermore, a "computer system" may include multiple computer devices connected via a network, including communication lines such as the Internet, WAN, LAN, and dedicated lines. Furthermore, a "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Thus, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM.

[0153] The recording medium also includes internal or external recording media accessible from a distribution server for distributing the program. The program may be divided into multiple parts, downloaded at different times, and then combined by the various components of the tablet terminal 1, or each divided program may be distributed by a different distribution server. Furthermore, the term "computer-readable recording medium" also includes a medium that stores a program for a certain period of time, such as volatile memory (RAM) within a computer system that serves as a server or client when a program is transmitted over a network. The program may also be a medium that realizes part of the above-described functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-described functions in combination with a program already stored in the computer system.

[0154] Furthermore, some or all of the above-described functions may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each of the above-described functions may be individually implemented as a processor, or some or all of the functions may be integrated into a processor. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used. [Explanation of symbols]

[0155] 1 tablet device 10 Main control unit 11 processors 12. Main memory 13. Flash Memory 20 Touchscreen 21 Display section 22 Touch sensor section 23 Peripheral Devices 24 Audio System 25. Mike 26 Speaker 27 Baseband Chip 28 Radio Section 30 pens 40 Storage section 41 Handwriting input memory section 42 Predictive learning result memory unit 43 Synthetic information storage unit 44 Text history memory 45 Reference table storage unit 101 Handwriting input processing unit 102 Text Recognition Unit 103 Prediction processing unit 104 Handwriting synthesis unit 105 Display processing unit 106 Predictive Learning Department 107 Token Generation Unit

Claims

1. an input unit capable of detecting handwritten input; a display unit capable of displaying the trajectory of the handwritten input; a text recognition unit that recognizes text based on the handwritten input detected by the input unit; a prediction processing unit that predicts a text candidate that follows the text recognized by the text recognition unit, based on the text; a handwriting synthesis unit that combines a first generation process that generates handwritten characters based on tokens collected by handwriting input from a user, the tokens being a set of one or more handwritings, and a second generation process that generates handwritten characters based on a conversion model prepared in advance for converting text characters into handwritten characters, and synthesizes the text candidates predicted by the prediction processing unit into handwritten characters; a display processing unit that displays, on the display unit, the handwritten characters that correspond to the text candidates and that are synthesized by the handwriting synthesis unit; An information processing device comprising:

2. a reference table storage unit that stores a reference table that associates token information indicating the token with a text character string; The handwriting synthesis unit generating the handwritten characters from the character strings that match the text character strings stored in the reference table storage unit by the first generation process; The character that does not match the text character string stored in the reference table storage unit among the text candidates is used to generate the handwritten character by the second generation process. The information processing device according to claim 1 .

3. The handwriting synthesis unit a first process of extracting a character string of N characters from the text candidate in the first generation process, and converting the text string to the handwritten character based on the token information corresponding to the matching text string when the extracted character string matches the text string stored in the reference table storage unit (where N is an integer equal to or greater than 1); a second process of subtracting 1 from N after the first process and repeating the first process until the number of N characters becomes 1 character; a third process for generating the handwritten character from the unmatched characters by the second generation process after the second process; to synthesize text suggestions into your handwriting. The information processing device according to claim 2 .

4. a token generating unit that collects the tokens based on the trajectory of the handwritten input, associates the token information with the text string corresponding to the token, and stores the associated information in the reference table storage unit; The information processing device according to claim 2 .

5. The display processing unit display information indicating the text candidate predicted by the prediction processing unit on the display unit, and when the input unit receives an acceptance operation for accepting the text candidate, display the handwritten character corresponding to the text candidate synthesized by the handwriting synthesis unit on the display unit. The information processing device according to claim 1 .

6. The display processing unit displays the text candidate in a specific font on the display unit as display information indicating the text candidate, and when the input unit receives the acceptance operation, changes the text candidate displayed in the specific font to the handwritten character synthesized by the handwriting synthesis unit. The information processing device according to claim 5 .

7. A control method for an information processing device including an input unit capable of detecting handwritten input and a display unit capable of displaying a trajectory of the handwritten input, a text recognition unit recognizing text based on the handwritten input detected by the input unit; a step in which a prediction processing unit predicts a text candidate that follows the text recognized by the text recognition unit, based on the text; a step in which the handwriting synthesis unit combines a first generation process in which handwritten characters are generated based on tokens collected by handwriting input from a user, the tokens being a set of one or more handwritings, and a second generation process in which handwritten characters are generated based on a conversion model prepared in advance for converting text characters into handwritten characters, thereby synthesizing the text candidates predicted by the prediction processing unit into handwritten characters; a display processing unit displaying, on the display unit, the handwritten characters corresponding to the text candidates synthesized by the handwriting synthesis unit; A control method comprising:

Images