Information processing apparatus and control method

Abstract

An information processing device includes: an input unit capable of detecting a handwriting input; a display unit capable of displaying a trajectory of the handwriting input; a text recognition unit that recognizes a text on the basis of the handwriting input detected by the input unit; a prediction processing unit that predicts, on the basis of the text recognized by the text recognition unit, text candidates that follow the text; a handwriting synthesis unit which synthesizes the text candidates predicted by the prediction processing unit into handwritten characters; and an agreement processing unit that displays, on the display unit, display information indicating the text candidates predicted by the prediction processing unit, and displays, on the display unit, the handwritten characters corresponding to the text candidates synthesized by the handwritten synthesis unit when the input unit accepts an agreement operation for agreeing the text candidates.

Description

Technical Field

[0001] The present invention relates to an information processing device and a control method. Background Art

[0002] In recent years, it is known that information processing devices such as notebook personal computers (hereinafter referred to as notebook PCs) and tablet terminals can perform handwriting input using a pen, a finger, etc. (for example, see Patent Document 1). In such information processing devices, for example, a user interface that is easy to use can be realized by performing handwriting input on a text document, which deepens the understanding of the document with handwritten annotations added.

[0003] Patent document 1: Japanese Patent Application Publication No. 2019-507915.

[0004] However, in the conventional information processing apparatus as described above, there is a problem that the input efficiency of handwriting input is low compared to text input, and the productivity is low. Summary of the invention

[0005] The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide an information processing device and a control method capable of improving the productivity of handwriting input.

[0006] In order to solve the above-mentioned problems, one embodiment of the present invention is an information processing device, comprising: an input unit capable of detecting handwriting input; a display unit capable of displaying the trajectory of the handwriting input; a text recognition unit for recognizing text based on the handwriting input detected by the input unit; a prediction processing unit for predicting text candidates following the text recognized by the text recognition unit; a handwriting synthesis unit for synthesizing the text candidates predicted by the prediction processing unit into handwritten characters; and a consent processing unit for displaying display information indicating the text candidates predicted by the prediction processing unit on the display unit, and displaying the handwritten characters corresponding to the text candidates synthesized by the handwriting synthesis unit on the display unit when the input unit accepts a consent operation for agreeing to the text candidates.

[0007] Furthermore, one aspect of the present invention is the information processing device described above, wherein the prediction processing unit may predict the text candidate based on a user's past handwriting input history.

[0008] Furthermore, one aspect of the present invention is the information processing device described above, wherein the prediction processing unit can predict the text candidate sentence by sentence.

[0009] In addition, one embodiment of the present invention is that in the information processing device described above, the consent processing unit can display the handwritten characters synthesized by the handwriting synthesis unit on the display unit with a line of a color different from the trajectory of the handwriting input as the display information representing the text candidate, and the input unit, when accepting the consent operation, changes the handwritten characters displayed with the line of the different color to the same color as the trajectory of the handwriting input.

[0010] In addition, one embodiment of the present invention is that in the information processing device described above, the consent processing unit can display the text candidate on the display unit using a specific font as the display information representing the text candidate, and when the input unit accepts the consent operation, the text candidate displayed in the specific font is changed to the handwritten text.

[0011] In one aspect of the present invention, in the above-described information processing device, the handwriting synthesis unit may infer handwritten characters written by the user for the text candidate based on an input history of past handwriting inputs by the user, and synthesize the handwritten characters.

[0012] In addition, one embodiment of the present invention is a control method for an information processing device having an input unit capable of detecting handwriting input and a display unit capable of displaying a trajectory of the handwriting input, comprising the following steps: a text recognition unit recognizes text based on the handwriting input detected by the input unit; a prediction processing unit predicts text candidates following the text based on the text recognized by the text recognition unit; a handwriting synthesis unit synthesizes the text candidates predicted by the prediction processing unit into handwritten characters; and a consent processing unit displays display information indicating the text candidates predicted by the prediction processing unit on the display unit, and displays the handwritten characters corresponding to the text candidates synthesized by the handwriting synthesis unit on the display unit when the input unit accepts a consent operation for agreeing to the text candidates.

[0013] Effects of the Invention

[0014] According to the present invention, the productivity of handwriting input can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is an external view showing an example of a tablet terminal according to the first embodiment.

[0016] Figure 2 This is a diagram showing an example of a main hardware configuration of the tablet terminal according to the first embodiment.

[0017] Figure 3 This is a block diagram showing an example of the functional configuration of the tablet terminal according to the first embodiment.

[0018] Figure 4 This is a first diagram for explaining an example of the operation of the tablet terminal according to the first embodiment.

[0019] Figure 5 This is a second diagram for explaining an example of the operation of the tablet terminal according to the first embodiment.

[0020] Figure 6 The third diagram explains an example of the operation of the tablet terminal according to the first embodiment.

[0021] Figure 7 This is a flowchart showing an example of the operation of the tablet terminal according to the first embodiment.

[0022] Figure 8 This is a flowchart showing an example of the operation of the tablet terminal according to the second embodiment.

[0023] Fig. 9 This is a diagram for explaining an example of the operation of the tablet terminal according to the second embodiment.

[0024] Fig.10 This is a diagram for explaining a modified example of the operation of the tablet terminal according to the second embodiment. DETAILED DESCRIPTION

[0025] Hereinafter, an information processing device and a control method according to an embodiment of the present invention will be described with reference to the drawings.

[0026] [First embodiment]

[0027] Figure 1 1 is an external view showing an example of a tablet terminal 1 according to the present embodiment. In the present embodiment, the tablet terminal 1 is described as an example of an information processing device.

[0028] like Figure 1 As shown, the tablet terminal 1 includes a touch screen 20 on one main surface of a housing CS1 , and uses a pen 30 to execute various processes based on, for example, an application program that enables handwriting input.

[0029] The touch screen 20 includes a display unit 21 and a touch sensor unit 22 . The display unit 21 displays various information on the display screen DF.

[0030] The touch sensor unit 22 is arranged to overlap the display unit 21 , detects contact between the pen 30 (an example of an operation medium) and the display screen DF of the display unit 21 , and detects a contact position of the pen 30 .

[0031] In addition, details of the touch screen 20 , the display unit 21 , and the touch sensor unit 22 will be described later.

[0032] Next, refer to Figure 2 The main hardware configuration of the tablet terminal 1 will be described.

[0033] Figure 2 This is a diagram showing an example of a main hardware configuration of the tablet terminal 1 according to the present embodiment.

[0034] like Figure 2 As shown, the tablet terminal 1 includes a processor 11 , a main memory 12 , a flash memory 13 , a touch screen 20 , peripheral devices 23 , an audio system 24 , a microphone 25 , a speaker 26 , a baseband chip 27 , and a wireless unit 28 .

[0035] The processor 11 is, for example, an application processor including a CPU (Central Processing Unit) 11. The processor 11 controls the entire tablet terminal 1.

[0036] The main memory 12 is a writable memory used as a read area for the execution program of the processor 11 or a work area for writing processing data of the execution program. The main memory 12 is composed of, for example, a plurality of DRAM (Dynamic Random Access Memory) chips. The execution program includes an OS (Operating System), various device drivers for hardware operations of peripheral devices, various services / utilities, and application programs (application software).

[0037] The flash memory 13 is, for example, a flash EEPROM (Electrically Erasable Programmable Read Only Memory), and stores an OS, various drivers, various services / utilities, application programs (hereinafter, sometimes referred to as applications), and various data.

[0038] The display unit 21 is, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and performs display based on the drawing data (display data) output from the processor 11 .

[0039] The touch sensor unit 22 detects the contact and contact position of the pen 30 (operation medium) on the screen of the display unit 21 (on the display screen DF). The touch sensor unit 22 can detect the position of the pen 30 on the screen, for example, by an electrostatic capacitance method, an electromagnetic induction method, etc. The touch sensor unit 22 detects the detection position data (raw data) of the pen 30 at a predetermined detection interval (sampling period). The touch sensor unit 22 is an example of an input unit that can detect handwriting input by the pen 30.

[0040] The touch sensor unit 22 includes a storage unit such as a CPU, RAM, and ROM (not shown) inside, and the CPU executes firmware stored in the storage unit to perform various processes such as noise removal of the detection position data of the pen 30 .

[0041] The audio system 24 is, for example, an audio IC (Integrated Circuit), which inputs, records, plays, and outputs sound data. The audio system 24 is, for example, connected to a microphone 25 and a speaker 26. The audio system 24 outputs the sound data collected by the microphone 25 to the processor 11 or the baseband chip 27. In addition, the audio system 24 converts the sound data obtained from the processor 11 or the baseband chip 27 into a sound signal, and outputs it to the speaker 26.

[0042] The microphone 25 collects sounds around the tablet terminal 1. The microphone 25 collects sounds such as the user's voice when performing sound fusion with other terminals, for example.

[0043] The speaker 26 outputs various sounds to the outside of the tablet terminal 1. For example, when performing sound fusion with other terminals, the speaker 26 outputs (plays) sounds received from other terminals.

[0044] The baseband chip 27 is, for example, a dedicated IC for controlling wireless communications such as 4G (4th generation mobile communication system) and 5G (5th generation mobile communication system). The baseband chip 27 outputs, for example, sound data received using the wireless unit 28 to the speaker 26 via the audio system 24. In addition, the baseband chip 27 obtains, for example, sound data collected from the microphone 25 via the audio system 24, and outputs it through the mobile communication system using the wireless unit 28. In addition, the baseband chip 27 communicates input and output data of data communication based on the mobile communication system with the processor 11.

[0045] The wireless unit 28 is a wireless communication device including an antenna for performing wireless communication based on a mobile communication system.

[0046] The pen 30 is a pen-shaped operating medium, such as a touch pen, a stylus pen, etc. In addition, the pen 30 has, for example, a resonance circuit, and is configured to supply power through electromagnetic induction to a coil of the resonance circuit, and the position and pen angle of the pen 30 on the screen of the display unit 21 are detected by the resonance circuit.

[0047] Next, refer to Figure 3 The functional configuration of the tablet terminal 1 according to the present embodiment will be described.

[0048] Figure 3 This is a block diagram showing an example of the functional configuration of the tablet terminal 1 according to the present embodiment.

[0049] like Figure 3 As shown, the tablet terminal 1 includes a main control unit 10 , a touch screen 20 , a pen 30 , and a storage unit 40 .

[0050] The touch screen 20 includes a display unit 21 and a touch sensor unit 22 .

[0051] Storage unit 40 is realized by, for example, main memory 12 or flash memory 13. Storage unit 40 includes handwriting input storage unit 41, prediction learning result storage unit 42, synthesis learning result storage unit 43, text history storage unit 44, and handwriting history storage unit 45.

[0052] The handwriting input storage unit 41 is a storage unit realized by the main memory 12 , for example, and stores handwriting input data. The handwriting input data is, for example, pen input data, which is a movement trajectory of the pen 30 based on the detection position data of the pen 30 detected by the touch sensor unit 22 .

[0053] The prediction learning result storage unit 42 is a storage unit implemented by, for example, the flash memory 13, and stores the prediction learning result (next text prediction model). The prediction learning result is a prediction model for predicting a text candidate following the input text (for example, a word, a part of a sentence, etc.), and is, for example, the result of machine learning based on the user's past handwriting input or text input and fixed phrases.

[0054] The synthesis learning result storage unit 43 is a storage unit implemented by, for example, the flash memory 13, and stores the synthesis learning result (handwriting synthesis model). The synthesis learning result is a synthesis model for synthesizing input text (for example, text candidates described later) into the user's handwritten text, and is, for example, the result of machine learning based on the user's past handwriting input.

[0055] The text history storage unit 44 is a storage unit implemented by the flash memory 13, for example, and stores the history information of past text inputs. The text history information of text inputs stored in the text history storage unit 44 is used for learning the above-mentioned prediction learning result (next text prediction model).

[0056] The handwriting history storage unit 45 is a storage unit realized by, for example, the flash memory 13, and stores history information of past handwriting inputs. The history information of text inputs stored in the handwriting history storage unit 45 is used for learning the synthesis learning result (handwriting synthesis model) described above.

[0057] The main control unit 10 is a functional unit realized by, for example, the processor 11 executing a program stored in the main memory 12 or the flash memory 13, and performs various processes based on an OS (for example, Android (registered trademark)). The main control unit 10 includes a handwriting input processing unit 101, a text recognition unit 102, a prediction processing unit 103, a handwriting synthesis unit 104, an agreement processing unit 105, a prediction learning unit 106, and a handwriting synthesis learning unit 107.

[0058] The handwriting input processing unit 101 is a functional unit realized by, for example, the processor 11 executing a program stored in the main memory 12 or the flash memory 13, and performs handwriting input processing. The handwriting input processing unit 101 acquires the detection position data of the operating medium such as the pen 30 from the touch sensor unit 22 of the touch screen 20, and displays the movement trajectory (handwriting input) of the pen 30 based on the acquired detection position data on the display unit 21. The handwriting input processing unit 101, for example Figure 4 The figure shows the movement trajectory of the pen 30 (handwriting input).

[0059] Figure 4 This is the first diagram for explaining an example of the operation of the tablet terminal 1 according to this embodiment. Figure 4 , an example is shown in which the tablet terminal 1 displays the movement trajectory of the pen 30 input by handwriting by the user on the display unit 21 .

[0060] exist Figure 4 In FIG. 1 , the handwriting input HW1 indicates a display when the user inputs “Hello, how” by handwriting.

[0061] return Figure 3 As described above, the handwriting input processing unit 101 stores the handwriting input data in the handwriting input storage unit 41 .

[0062] The text recognition unit 102 is a functional unit realized by, for example, the processor 11 executing a program stored in the main memory 12 or the flash memory 13, and recognizes text based on handwriting input detected by the touch sensor unit 22. The text recognition unit 102 converts handwriting input data into text data using, for example, a known handwriting recognition technology.

[0063] The text recognition unit 102 associates the handwriting input data with the converted text data and stores the converted text data in the handwriting history storage unit 45 as handwriting input history information. The text recognition unit 102 stores the converted text data in the text history storage unit 44 as text input history information.

[0064] The prediction processing unit 103 is a functional unit realized by, for example, the processor 11 executing a program stored in the main memory 12 or the flash memory 13, and predicts a text candidate following the text recognized by the text recognition unit 102. The prediction processing unit 103 predicts a text candidate to be input next, for example, using the prediction learning result (next text prediction model) stored in the prediction learning result storage unit 42, for the text recognized by the text recognition unit 102.

[0065] That is, the prediction processing unit 103 predicts text candidates based on the user's past handwriting input history.

[0066] Furthermore, the prediction processing unit 103 may predict text candidates word by word or sentence by sentence (in units of sentences). In addition, the prediction processing unit 103 may predict text candidates in units of characters, phrases, or numeric strings (e.g., telephone numbers). That is, the text candidates may also be other contents including numbers, symbols, characters, words, and sentences.

[0067] The handwriting synthesis unit 104 is a functional unit realized by, for example, the processor 11 executing a program stored in the main memory 12 or the flash memory 13, and synthesizes the text candidates (contents of the text candidates) predicted by the prediction processing unit 103 into handwritten characters. The handwriting synthesis unit 104 synthesizes the text candidates predicted by the prediction processing unit 103 into handwritten characters using the synthesis learning results (handwriting synthesis model) stored in the synthesis learning result storage unit 43.

[0068] That is, the handwriting synthesis unit 104 estimates the handwritten characters written by the user for the text candidates based on the user's past handwriting input history, and synthesizes the handwritten characters. In addition, the style of the synthesized handwritten characters is selected by the user or is similar to the user's handwriting.

[0069] Note that the handwriting synthesis unit 104 may execute the handwriting synthesis process in parallel with the prediction processing unit 103 described later.

[0070] The consent processing unit 105 is a functional unit realized by, for example, the processor 11 executing a program stored in the main memory 12 or the flash memory 13, and displays the display information indicating the text candidate predicted by the prediction processing unit 103 on the display unit 21. The consent processing unit 105 displays, for example, the handwritten characters synthesized by the handwriting synthesis unit 104 on the display unit 21 in a line of a color different from the trajectory of the handwriting input as the display information indicating the text candidate. The consent processing unit 105 displays, for example, the handwritten characters synthesized by the handwriting synthesis unit 104 in a line of a color different from the trajectory of the handwriting input on the display unit 21 as the display information indicating the text candidate. Figure 5 As shown, the handwritten characters of the text candidates are displayed on the display unit 21 in gray lines.

[0071] Figure 5This is a second diagram for explaining an example of the operation of the tablet terminal 1 according to this embodiment. Figure 5 , an example is shown in which the consent processing unit 105 of the tablet terminal 1 displays handwritten characters of text candidates on the display unit 21 .

[0072] exist Figure 5 In FIG. 1 , the handwriting input candidate CW1 indicates “are you doing”, which is a candidate for the next handwriting input predicted with respect to the handwriting input HW1 (“Hello, how”) handwritten by the user.

[0073] like Figure 5 As shown, the consent processing unit 105 displays the handwriting input candidate CW1 in gray lines, for example. When displaying the handwriting input candidate CW1, the consent processing unit 105 may display it in other colors, in a dotted line or other types, or by blinking the display instead of the gray lines.

[0074] In addition, when displaying the handwriting input candidate CW1, the approval processing unit 105 checks the position and size of the displayed handwriting input candidate CW1 based on the bounding box of the handwriting input HW1, and displays the handwriting input candidate CW1 at an appropriate position and size. Here, the bounding box of the handwriting input HW1 refers to a box-shaped (rectangular) display area surrounding the handwriting input HW1.

[0075] Return again Figure 3 As described above, when the touch sensor unit 22 receives the consent operation of approving the text candidate, the consent processing unit 105 displays the handwritten characters corresponding to the text candidate synthesized by the handwriting synthesis unit 104 on the display unit 21. The consent processing unit 105 touches the display of the text candidate with the pen 30 or a finger (for example, Figure 5 The operation of the handwriting input candidate CW1) is performed, so that for example Figure 6 As shown, the handwritten characters of the text candidates are displayed on the display unit 21 .

[0076] Figure 6 The third diagram is a diagram for explaining an example of the operation of the tablet terminal 1 according to the present embodiment. Figure 6 , there is shown a display example when the consent processing unit 105 accepts the consent operation of the consent text candidate.

[0077] exist Figure 6In the figure, the handwriting input HW2 indicates that the candidate CW1 of the handwriting input is approved and becomes the formal handwriting input. When the approval operation is accepted, the handwriting input HW2 is changed from a gray line to a black line. That is, the approval processing unit 105 changes the gray line of the candidate CW1 of the handwriting input to a black line and displays the handwriting input HW2. In addition, the approval processing unit 105 stores the handwriting input HW2 as the formal handwriting input in the handwriting input storage unit 41.

[0078] In this way, the consent processing unit 105 displays the handwritten characters synthesized by the handwriting synthesis unit 104 on the display unit 21 with a line of a color different from the trajectory of the handwriting input (for example, gray) as display information indicating text candidates, and when the touch sensor unit 22 accepts the consent operation, the handwritten characters displayed with a line of a different color (for example, gray) are changed to the same color as the trajectory of the handwriting input.

[0079] Return again Figure 3 As described above, the prediction learning unit 106 is a functional unit implemented by, for example, the processor 11 executing a program stored in the main memory 12 or the flash memory 13. The prediction learning unit 106 performs a machine learning process for generating a prediction learning result (next text prediction model) of a prediction text candidate based on the user's past handwriting input history.

[0080] The prediction learning unit 106 executes machine learning processing using, for example, the text history information stored in the text history storage unit 44 as learning data, and stores the learning result (next text prediction model) in the prediction learning result storage unit 42 .

[0081] The handwriting synthesis learning unit 107 is a functional unit realized by, for example, the processor 11 executing a program stored in the main memory 12 or the flash memory 13. The handwriting synthesis learning unit 107 generates a handwriting synthesis model by machine learning based on the input history of the user's past handwriting inputs, which infers the handwritten characters written by the user from the text data and synthesizes the handwritten characters.

[0082] The handwriting synthesis learning unit 107 performs machine learning processing using, for example, past handwriting input history information stored in the handwriting history storage unit 45 as learning data, and stores the synthesis learning result (handwriting synthesis model) in the synthesis learning result storage unit 43 .

[0083] The prediction learning unit 106 and the handwriting synthesis learning unit 107 periodically execute machine learning processing at predetermined time intervals, for example.

[0084] In addition, in the present embodiment, the text recognition unit 102, the prediction processing unit 103, the handwriting synthesis unit 104, and the consent processing unit 105 may be functional units implemented by an application program or may be functional units implemented as processing by the OS. In addition, a part of the text recognition unit 102, the prediction processing unit 103, the handwriting synthesis unit 104, and the consent processing unit 105 may be implemented as processing of a device driver or internal processing of the touch sensor unit 22.

[0085] Next, the operation of the tablet terminal 1 according to the present embodiment will be described with reference to the drawings.

[0086] Figure 7 This is a flowchart showing an example of the operation of the tablet terminal 1 according to the present embodiment.

[0087] like Figure 7 As shown, the tablet terminal 1 first determines whether the contact of the pen 30 with the screen (display screen DF) is detected (step S101). The handwriting input processing unit 101 of the tablet terminal 1 determines whether the touch sensor unit 22 detects the contact of the pen 30 with the display screen DF. When the touch sensor unit 22 detects the contact of the pen 30 (step S101: Yes), the handwriting input processing unit 101 advances the process to step S102. In addition, when the touch sensor unit 22 does not detect the contact of the pen 30 (step S101: No), the handwriting input processing unit 101 returns the process to step S101.

[0088] In step S102 , the handwriting input processing unit 101 acquires pen input data. That is, the handwriting input processing unit 101 acquires position information on the display screen DF of the pen 30 detected by the touch sensor unit 22 as pen input data.

[0089] Next, the handwriting input processing unit 101 displays the trajectory of the stylus input (step S103). The handwriting input processing unit 101 stores the acquired pen input data as the stylus input in the handwriting input storage unit 41, and, for example, Figure 4 As in the handwriting input HW1 shown, the trajectory of the handwriting pen input is displayed on the display unit 21.

[0090] Next, the handwriting input processing unit 101 determines whether the pen input is completed (step S104). If the pen input is completed (step S104: Yes), the handwriting input processing unit 101 proceeds to step S105. If the pen input is not completed (step S104: No), the handwriting input processing unit 101 returns the process to step S101.

[0091] In step S105, the tablet terminal 1 converts the handwriting input into text data (text information). For example, the text recognition unit 102 of the tablet terminal 1 converts the handwriting input into text data (text information). Figure 4 The handwritten input HW1 shown is converted into text data such as "Hello, how". The text recognition unit 102 associates the handwritten input data with the converted text data and stores it as historical record information of the handwritten input in the handwritten history storage unit 45. In addition, the text recognition unit 102 stores the converted text data as historical record information of the text input in the text history storage unit 44.

[0092] Next, the tablet terminal 1 predicts text candidates for the subsequently converted text data (step S106). The prediction processing unit 103 of the tablet terminal 1 predicts, for example, text candidates for the text data subsequently converted by the text recognition unit 102 using the prediction learning result (next text prediction model) stored in the prediction learning result storage unit 42.

[0093] Next, the prediction processing unit 103 determines whether the prediction is OK (the prediction is completed normally) (step S107). When the prediction is OK (the prediction is completed normally) (step S107: Yes), the prediction processing unit 103 advances the process to step S108. In addition, when the prediction is not OK (the prediction is not completed normally) (step S107: No), the prediction processing unit 103 returns the process to step S101.

[0094] Next, in step S108, the tablet terminal 1 synthesizes the text candidates into handwritten characters. The handwritten synthesis unit 104 of the tablet terminal 1 synthesizes the text candidates predicted by the prediction processing unit 103 into handwritten characters using the synthesis learning result (handwritten synthesis model) stored in the synthesis learning result storage unit 43.

[0095] Next, the tablet terminal 1 updates the bounding box information of the current handwritten input trajectory (step S109). The consent processing unit 105 of the tablet terminal 1 updates, for example, Figure 4 the position and size of the bounding box (the display area of the box (rectangle) surrounding the handwritten input HW1) of the handwritten input HW1 shown.

[0096] Next, the consent processing unit 105 confirms the position and size of the displayed text candidates (step S110). The consent processing unit 105 confirms, for example, the position and size of the displayed handwritten input candidate CW1 based on the position and size of the updated bounding box of the handwritten input HW1.

[0097] Next, the consent processing unit 105 displays the handwritten characters of the text candidates in gray at the confirmed position and size (step S111). The consent processing unit 105, for example, Figure 5 displays the handwritten characters of the text candidates in gray on the display unit 21 as shown in the handwritten input candidate CW1.

[0098] Next, the consent processing unit 105 determines whether an operation of the consent candidate (consent operation) has been accepted (step S112). As the consent operation, the consent processing unit 105 determines whether an operation of touching with the pen 30 or a finger has been accepted through the touch sensor unit 22. If the consent processing unit 105 has accepted the consent operation (step S112: Yes), the processing proceeds to step S113. On the other hand, if the consent processing unit 105 has not accepted the consent operation (step S112: No), the processing proceeds to step S114.

[0099] The case where the consent operation is not accepted here includes the case where the consent operation is not accepted within a predetermined period and / or the case where the operation of the non-consent candidate is accepted.

[0100] In step S113, the consent processing unit 105 determines the handwritten characters in the text candidate as handwritten input. Figure 6 The gray line of the handwriting input candidate CW1 is changed to a black line as shown in the handwriting input HW2, and displayed on the display unit 21. In addition, the approval processing unit 105 stores the handwritten characters of the text candidate as handwriting input in the handwriting input storage unit 41. After the processing of step S113, the approval processing unit 105 returns the processing to step S101.

[0101] In step S114, the consent processing unit 105 deletes the display of the candidate. The consent processing unit 105 deletes the display of the candidate CW1 of the handwriting input, for example, returns to Figure 4 After the process of step S114, the consent processing unit 105 returns the process to step S101.

[0102] As described above, the tablet terminal 1 (information processing device) of this embodiment includes the touch sensor unit 22 (input unit), the display unit 21, the text recognition unit 102, the prediction processing unit 103, the handwriting synthesis unit 104, and the consent processing unit 105. The touch sensor unit 22 (input unit) can detect handwriting input. The display unit 21 can display the trajectory of handwriting input. The text recognition unit 102 recognizes text based on the handwriting input detected by the touch sensor unit 22. The prediction processing unit 103 predicts a text candidate following the text based on the text recognized by the text recognition unit 102. The handwriting synthesis unit 104 synthesizes the text candidate predicted by the prediction processing unit 103 into handwritten characters. The consent processing unit 105 displays display information indicating the text candidate predicted by the prediction processing unit 103 on the display unit 21, and displays the handwritten characters corresponding to the text candidate synthesized by the handwriting synthesis unit 104 on the display unit 21 when the touch sensor unit 22 accepts the consent operation of agreeing to the text candidate.

[0103] Thus, the tablet terminal 1 (information processing device) of the present embodiment can display text candidates and determine them by accepting the consent operation, so that the input efficiency of handwriting input can be improved and the productivity of handwriting input can be improved.

[0104] Furthermore, in the present embodiment, the prediction processing unit 103 predicts text candidates based on the user's past handwriting input history.

[0105] Thus, the tablet terminal 1 of the present embodiment can predict appropriate text candidates corresponding to the user's past usage. Therefore, the tablet terminal 1 of the present embodiment can further improve the productivity of handwriting input.

[0106] In addition, in the present embodiment, the prediction processing unit 103 predicts text candidates sentence by sentence.

[0107] As a result, the tablet terminal 1 according to the present embodiment predicts text candidates sentence by sentence, and thus the productivity of handwriting input can be further improved.

[0108] In addition, in the present embodiment, the consent processing unit 105 displays the handwritten text synthesized by the handwriting synthesis unit 104 on the display unit 21 with a line of a color different from the trajectory of the handwriting input (for example, gray) as display information representing a text candidate, and the touch sensor unit 22 changes the handwritten text displayed with a line of a different color to the same color as the trajectory of the handwriting input (for example, black) when the consent operation is accepted.

[0109] Thus, the tablet terminal 1 of the present embodiment can clearly distinguish between text candidates and handwriting input confirmed by the consent operation, and thus can further improve the productivity of handwriting input.

[0110] In the present embodiment, the handwriting synthesis unit 104 estimates the handwritten characters written by the user for the text candidates based on the user's past handwriting input history, and synthesizes the handwritten characters.

[0111] Thus, the tablet terminal 1 of this embodiment can display text candidates in a state close to the user's handwriting input, so that the text candidates can be displayed without any discomfort. Therefore, the tablet terminal 1 of this embodiment can improve the productivity of handwriting input without any discomfort.

[0112] In addition, the control method of the present embodiment is a control method for a tablet terminal 1 having a touch sensor unit 22 capable of detecting handwriting input and a display unit 21 capable of displaying a trajectory of handwriting input, and includes a text recognition step, a prediction processing step, a handwriting synthesis step, and an agreement processing step. In the text recognition step, the text recognition unit 102 recognizes text based on the handwriting input detected by the touch sensor unit 22. In the prediction processing step, the prediction processing unit 103 predicts a text candidate following the text based on the text recognized by the text recognition unit 102. In the handwriting synthesis step, the handwriting synthesis unit 104 synthesizes the text candidate predicted by the prediction processing unit 103 into handwritten text. In the agreement processing step, the agreement processing unit 105 displays display information indicating the text candidate predicted by the prediction processing unit 103 on the display unit 21, and the touch sensor unit 22 displays the handwritten text corresponding to the text candidate synthesized by the handwriting synthesis unit 104 on the display unit 21 when receiving an agreement operation of agreeing to the text candidate.

[0113] Therefore, the control method of this embodiment has the same effect as that of the tablet terminal 1 described above, and can improve the input efficiency of handwriting input and improve the productivity of handwriting input.

[0114] [Second embodiment]

[0115] Next, a tablet terminal 1 according to a second embodiment will be described with reference to the drawings. In the present embodiment, a modification example in which text candidates are displayed using text characters instead of handwritten characters will be described.

[0116] The appearance, hardware configuration, and functional configuration of the tablet terminal 1 of this embodiment are similar to those of the above-mentioned Figure 1 to Figure 3 The first embodiment shown is the same, so the description thereof is omitted here.

[0117] The consent processing unit 105 in this embodiment displays text candidates in a specific font as display information indicating the text candidates on the display unit 21 , and changes the text candidates displayed in the specific font to handwritten characters when the touch sensor unit 22 accepts a consent operation.

[0118] Next, refer to Figure 8 The operation of the tablet terminal 1 according to the present embodiment will be described.

[0119] Figure 8 This is a diagram showing an example of a main hardware configuration of the tablet terminal 1 according to the present embodiment.

[0120] exist Figure 8 In the process from step S201 to step S207, the process is the same as that described above. Figure 7 The processes from step S101 to step S107 shown are the same, so the description thereof is omitted here.

[0121] In step S208, the tablet terminal 1 updates the bounding box information of the current handwriting input trajectory. The consent processing unit 105 of the tablet terminal 1 updates, for example, Figure 4 The position and size of the bounding box (box-shaped (rectangular) display area surrounding the handwriting input HW1) of the handwriting input HW1 are shown.

[0122] Next, the approval processing unit 105 checks the position and size of the displayed text candidate (step S209). The approval processing unit 105 checks the position and size of the displayed text candidate based on the position and size of the bounding box of the handwriting input HW1 after the update, for example.

[0123] Next, the consent processing unit 105 displays the text candidate at the confirmed position and size (step S210). Fig. 9 The text candidates are displayed on the display unit 21 as shown in the handwriting input candidate CW2. Fig. 9 The display of text candidates according to the present embodiment will be described.

[0124] Fig. 9 This is a diagram for explaining an example of the operation of the tablet terminal 1 according to the present embodiment. Figure 5 The display shown executes Fig. 9 Display shown.

[0125] exist Fig. 9 In FIG. 1 , the handwriting input candidate CW2 indicates “are you doing”, which is a candidate for the next handwriting input predicted for the handwriting input HW1 (“Hello, how”) input by handwriting by the user.

[0126] like Fig. 9 As shown, the approval processing unit 105 displays the handwriting input candidate CW2 on the display unit 21 as text characters in a specific font, for example.

[0127] return Figure 8 Next, the consent processing unit 105 determines whether the operation of the consent candidate (consent operation) is accepted (step S211). As the consent operation, the consent processing unit 105 determines whether the operation of touching with the pen 30 or the finger is accepted through the touch sensor unit 22. If the consent processing unit 105 accepts the consent operation (step S211: Yes), the processing proceeds to step S212. In addition, if the consent processing unit 105 does not accept the consent operation (step S211: No), the processing proceeds to step S214.

[0128] In step S212 , the tablet terminal 1 synthesizes the text candidates into handwritten characters. The handwriting synthesis unit 104 of the tablet terminal 1 synthesizes the text candidates predicted by the prediction processing unit 103 and accepted for approval into handwritten characters using the synthesis learning results (handwriting synthesis model) stored in the synthesis learning result storage unit 43 .

[0129] Next, the consent processing unit 105 changes the text candidate to a handwritten character display to confirm the handwriting input (step S213). Fig. 9 The candidate CW2 for handwriting input shown is changed to the handwritten character ( Figure 6 The handwriting input HW2 shown in FIG. 1 is displayed on the display unit 21. In addition, the processing unit 105 accepts the synthesized handwriting ( Figure 6 The handwriting input HW2) shown is stored as handwriting input in the handwriting input storage unit 41. After the process of step S213, the approval processing unit 105 returns the process to step S201.

[0130] In step S214, the consent processing unit 105 deletes the display of the candidate. The consent processing unit 105 deletes the display of the candidate CW2 for handwriting input, for example, returns to Figure 4 After the process of step S214, the consent processing unit 105 returns the process to step S201.

[0131] As described above, in this embodiment, the consent processing unit 105 displays text candidates in a specific font as display information indicating the text candidates on the display unit 21, and changes the text candidates displayed in the specific font to handwritten characters when the touch sensor unit 22 receives a consent operation.

[0132] Thus, the tablet terminal 1 of the present embodiment achieves the same effects as those of the first embodiment described above, and can improve the input efficiency of handwriting input and improve the productivity of handwriting input.

[0133] In addition, in the tablet terminal 1 of this embodiment, since the text candidates are displayed on the display unit 21 using a specific font, handwritten characters can be generated in the background when the text candidates are displayed. As a result, the tablet terminal 1 of this embodiment can reduce the waiting time of the user and can display the handwritten characters at the moment of obtaining the user's consent.

[0134] Furthermore, the tablet terminal 1 of the present embodiment can clearly distinguish between text candidates and handwriting input for which the consent operation is confirmed, and thus can further improve the productivity of handwriting input.

[0135] In addition, the tablet terminal 1 (information processing device) of the first and second embodiments described above may also be in the following form. The tablet terminal 1 (information processing device) includes a touch sensor unit 22 capable of detecting handwriting input, a display unit 21 capable of displaying a trajectory of handwriting input, a main memory 12 (memory) for temporarily storing programs, and a processor 11 for executing programs stored in the main memory 12. The processor 11 performs text recognition processing, prediction processing, handwriting synthesis processing, and consent processing by executing the programs stored in the main memory 12. As a text recognition process, the processor 11 recognizes text based on handwriting input detected by the touch sensor unit 22. As a prediction process, the processor 11 predicts a text candidate following the text based on the text recognized by the text recognition process. As a handwriting synthesis process, the processor 11 synthesizes the text candidates predicted by the prediction process into handwritten text. As consent processing, processor 11 displays display information indicating text candidates predicted by prediction processing on display unit 21, and when touch sensor unit 22 accepts a consent operation for agreeing to the text candidate, handwritten characters corresponding to the text candidate synthesized by handwriting synthesis processing are displayed on display unit 21.

[0136] Thus, the tablet terminal 1 (information processing device) according to the first and second embodiments can improve the input efficiency of handwriting input and can improve the productivity of handwriting input.

[0137] In addition, the present invention is not limited to the above-mentioned embodiment, and can be modified within the scope not departing from the gist of the present invention.

[0138] For example, in the above-mentioned embodiments, the information processing device is described as a tablet terminal 1, but the present invention is not limited thereto. The information processing device may be, for example, a smartphone, a notebook personal computer having a tablet mode, or the like.

[0139] In the above-mentioned embodiments, an example in which the OS is Android (registered trademark) has been described, but the present invention is not limited to this, and other OSs such as Windows (registered trademark) and iOS (registered trademark) may be used.

[0140] In the above-mentioned embodiments, an example in which the operation medium is the pen 30 is described, but the present invention is not limited thereto, and other operation media such as a user's finger may be used.

[0141] In addition, in the above-mentioned embodiments, the example in which the tablet terminal 1 displays one text candidate on the display unit 21 is described, but the present invention is not limited to this, and a plurality of text candidates may be displayed on the display unit 21. In this case, the prediction processing unit 103 predicts a plurality of text candidates, and the agreement processing unit 105 displays the plurality of text predictions as follows: Fig.10The selected text candidates are displayed on the display unit 21 as shown, and the selected text candidates are combined into handwritten characters and displayed, thereby confirming the handwriting input. In addition, the consent processing unit 105 may display the plurality of text candidates in descending order of possibility of being selected.

[0142] here, Fig.10 This is a diagram for explaining a modified example of the operation of the tablet terminal 1 according to the second embodiment. Fig. 9 The display shown executes Fig.10 Display shown.

[0143] exist Fig.10 In FIG. 1 , text candidates CW3 show “you”, “we”, and “they”, which are a plurality of candidates for the next handwriting input predicted for handwriting input HW3 (“How are”) input by handwriting by the user.

[0144] like Fig.10 As shown, the consent processing unit 105 displays the plurality of text candidates CW3 on the display unit 21 in vertical text characters using a specific font, for example. Alternatively, the consent processing unit 105 may display the plurality of text candidates CW3 on the display unit 21 by ranking them by probability or other factors.

[0145] In the above-mentioned embodiments, the tablet terminal 1 is described as including the prediction learning unit 106 and the handwriting synthesis learning unit 107 . However, the present invention is not limited thereto, and the tablet terminal 1 may not include either or both of the prediction learning unit 106 and the handwriting synthesis learning unit 107 .

[0146] In addition, in each of the above-mentioned embodiments, an example is described in which the prediction processing unit 103 uses the prediction learning results (next text prediction model) to predict text candidates, but it is not limited to this. For example, other methods such as pre-storing past inputs and predicting inputs that are the same as past inputs as text candidates may also be used.

[0147] In addition, in each of the above-mentioned embodiments, an example is described in which the handwriting synthesis unit 104 uses the synthesis learning result (handwriting synthesis model) to synthesize handwritten text, but this is not limited to this. For example, other methods may also be used, such as storing the handwritten text that the user has previously handwritten input in a storage unit and obtaining the corresponding handwritten text to synthesize the handwritten text.

[0148] In addition, each component of the tablet terminal 1 has a computer system inside. Furthermore, the processing in each component of the tablet terminal 1 can also be performed by recording a program for realizing the functions of each component of the tablet terminal 1 on a computer-readable recording medium, and causing the computer system to read and execute the program recorded on the recording medium. Here, "causing the computer system to read and execute the program recorded on the recording medium" includes installing the program on the computer system. The "computer system" here refers to hardware including OS, peripheral devices, etc.

[0149] In addition, the "computer system" may also include a plurality of computer devices connected via a network including a communication line such as the Internet, WAN, LAN, or a dedicated line. In addition, the "computer-readable recording medium" refers to a portable medium such as a floppy disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built into the computer system. In this way, the recording medium storing the program may also be a non-temporary recording medium such as a CD-ROM.

[0150] In addition, the recording medium also includes an internal or external recording medium that can be accessed from a distribution server in order to distribute the program. In addition, the program can also be divided into multiple parts, and the composition of each component combined by the tablet terminal 1 after being downloaded at different times, or the distribution server for distributing each divided program is different. In addition, "computer-readable recording medium" refers to a medium that also includes a volatile memory (RAM) inside a computer system that keeps the program for a constant time, and the above-mentioned computer system is a server or client in the case of sending the program via a network. In addition, the above-mentioned program can also be used to implement a part of the above-mentioned functions. In addition, it can also be a so-called differential file (differential program), which implements the above-mentioned functions in combination with a program already recorded in the computer system.

[0151] In addition, part or all of the above functions may be implemented as an integrated circuit such as LSI (Large Scale Integration). Each of the above functions may be independently processed, or part or all may be integrated and processed. In addition, the method of integrated circuitization is not limited to LSI, and may also be implemented by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI emerges due to the advancement of semiconductor technology, an integrated circuit based on the technology may also be used.

[0152] Description of Reference Numerals

[0153] 1 tablet terminal; 10 main control unit; 11 processor; 12 main memory; 13 flash memory; 20 touch screen; 21 display unit; 22 touch sensor unit; 23 peripheral equipment; 24 audio system; 25 microphone; 26 speaker; 27 baseband chip; 28 wireless unit; 30 pen; 40 storage unit; 41 handwriting input storage unit; 42 prediction learning result storage unit; 43 synthesis learning result storage unit; 44 text history storage unit; 45 handwriting history storage unit; 101 handwriting input processing unit; 102 text recognition unit; 103 prediction processing unit; 104 handwriting synthesis unit; 105 consent processing unit; 106 prediction learning unit; 107 handwriting synthesis learning unit.

Claims

1. An information processing device comprising: An input unit capable of detecting handwriting input; A display unit capable of displaying the trajectory of the handwriting input; A text recognition unit that recognizes text based on the handwriting input detected by the input unit; a prediction processing unit that predicts a text candidate following the text based on the text recognized by the text recognition unit; a handwriting synthesis unit that synthesizes the text candidates predicted by the prediction processing unit into handwritten characters; and The consent processing unit displays display information indicating the text candidate predicted by the prediction processing unit on the display unit, and displays the handwritten characters corresponding to the text candidate synthesized by the handwriting synthesis unit on the display unit when the input unit accepts a consent operation for agreeing to the text candidate.

2. The information processing device according to claim 1, in, The prediction processing unit predicts the text candidate based on a user's past handwriting input history.

3. The information processing device according to claim 1, in, The prediction processing unit predicts the text candidates sentence by sentence.

4. The information processing device according to claim 1, in, The consent processing unit displays the handwritten characters synthesized by the handwriting synthesis unit on the display unit with a line of a color different from the trajectory of the handwriting input as the display information representing the text candidate, and when the input unit accepts the consent operation, changes the handwritten characters displayed with the line of the different color to the same color as the trajectory of the handwriting input.

5. The information processing device according to claim 1, in, The consent processing unit displays the text candidate on the display unit using a specific font as the display information indicating the text candidate, and changes the text candidate displayed in the specific font to the handwritten characters when the input unit accepts the consent operation.

6. The information processing device according to claim 1, in, The handwriting synthesis unit estimates handwritten characters written by the user for the text candidate based on a past handwriting input history of the user, and synthesizes the handwritten characters.

7. A control method for an information processing device having an input unit capable of detecting handwriting input and a display unit capable of displaying a trajectory of the handwriting input, comprising the following steps: The text recognition unit recognizes text based on the handwriting input detected by the input unit; The prediction processing unit predicts a text candidate following the text based on the text recognized by the text recognition unit; The handwriting synthesis unit synthesizes the text candidates predicted by the prediction processing unit into handwritten characters; and The consent processing unit displays display information indicating the text candidate predicted by the prediction processing unit on the display unit, and when the input unit accepts a consent operation for agreeing to the text candidate, displays the handwritten characters corresponding to the text candidate synthesized by the handwriting synthesis unit on the display unit.

Images