Information processing device, display method, program, and input system

Abstract

To provide an input device that has improved operability of handwriting input.SOLUTION: There is provided an input device 2 that comprises: a handwriting input unit that receives a handwriting input by using the position of a pen or a user's finger in contact with a display; and a display unit that displays the handwriting input received by the handwriting input unit on the display as a handwriting object. When no change occurs in the handwriting object for a first time, the input device 2 displays one or more operation commands based on the handwriting object.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention provides Information processing device, display method, Programs and input systems. [Background technology]

[0002] In general computer-controlled whiteboard devices or applications that allow handwritten input (hereinafter referred to as "input devices"), the input means is limited to a pen or finger. For this reason, an operation menu is provided so that the user can switch between a pen function for changing the color of text and an editing function for erasing text. Typically, the pen function menu allows the user to select color, thickness, etc., while the editing function menu allows the user to select erase, move, zoom, rotate, cut, copy, paste, etc. (See, for example, Patent Document 1).

[0003] Patent Document 1 discloses an input device that displays a menu for setting color, transparency, thickness, line type, stamp, and operation when a pen button is pressed. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional input devices have a problem in that handwriting input is sometimes difficult to use. For example, every time a user wants to edit or modify a handwritten object, they must open a pen function menu or an edit function menu. After opening the menu, they must then select the desired setting or function (color, thickness, line type, etc.).

[0005] In view of the above problems, the present invention has been made to improve the operability of handwriting input. technology The purpose is to provide. [Means for solving the problem]

[0006] In view of the above-described problems, the present invention provides an information processing apparatus capable of communicating with an input device via a network, the information processing apparatus including: a communication unit that receives, from the input device, data relating to handwritten input accepted by the input device, the data relating to the handwritten input being displayed on a display of the input device as a handwritten object; a handwriting recognition control unit that recognizes the data relating to the handwritten input received by the communication unit as a character string candidate; and a character string candidate recognized by the handwriting recognition control unit. Depending on the string that partially matches and an operation command recognition control unit that acquires operation command candidates, wherein the communication unit acquires the character string candidates recognized by the handwriting recognition control unit and the character string candidates recognized by the handwriting recognition control unit so that the input device can selectably display the character string candidates and the operation command candidates on the display when no change in the handwritten object displayed on the display occurs for a first period of time. Depending on the string that partially matches The operation command candidate is transmitted to the input device. death , When the operation command candidate is not selected for a second time period on the input device, when an operation to erase the display of the operation command candidate is received, or when a change occurs in the handwritten object, the displayed operation command candidate is erased, and when the display of the operation command candidate is erased under conditions other than when the operation command candidate is selected, information to erase the display of the operation command candidate is transmitted to the input device in order to maintain the display of the handwritten object. It is characterized by: [Effects of the Invention]

[0007] Improved usability of handwriting input technology can be provided. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an operation menu displayed by a conventional handwriting input device. [Figure 2] FIG. 1 is a diagram illustrating an example of a multi-function pen. [Figure 3] 10A and 10B are diagrams illustrating an example of display of operation command candidates displayed by a handwriting input device; [Figure 4] 1 is a diagram illustrating an overall configuration of an example of a handwriting input device. [Figure 5] FIG. 1 is a diagram illustrating a hardware configuration of an example of a handwriting input device. [Figure 6] FIG. 2 is a functional block diagram illustrating an example of functions of the handwriting input device in blocks. [Figure 7] FIG. 2 is a functional block diagram illustrating an example of functions relating to user authentication of the handwriting input device. [Figure 8] FIG. 10 is a diagram illustrating an example of predefined control data. [Figure 9] FIG. 4 is a diagram showing an example of dictionary data in a handwriting recognition dictionary unit. [Figure 10] FIG. 10 is a diagram illustrating an example of dictionary data in a character string conversion dictionary unit. [Figure 11] FIG. 10 is a diagram illustrating an example of dictionary data in a predictive conversion dictionary unit. [Figure 12] 10A and 10B are diagrams illustrating an example of operation command definition data and system definition data held by an operation command definition unit. [Figure 13] 10A and 10B are diagrams showing an example of operation command definition data in the case where a selected object is selected by a handwritten object; [Figure 14] 10A and 10B are diagrams showing an example of display of operation command candidates based on operation command definition data when there is no selected object; [Figure 15] 10A and 10B are diagrams illustrating an example of designating a selected object; [Figure 16] 10A and 10B are diagrams showing an example of display of operation command candidates based on operation command definition data when a handwritten object is present; [Figure 17] 10A and 10B are diagrams showing an example of display of operation command candidates based on operation command definition data when a handwritten object is present; [Figure 18] FIG. 10 is a sequence diagram (part 1) illustrating an example of a process in which the handwriting input device displays character string candidates and operation command candidates. [Figure 19] FIG. 10 is a sequence diagram (part 2) illustrating an example of a process in which the handwriting input device displays character string candidates and operation command candidates. [Figure 20] FIG. 10 is a sequence diagram (part 3) illustrating an example of a process in which the handwriting input device displays character string candidates and operation command candidates. [Figure 21]FIG. 10 is a sequence diagram (part 4) illustrating an example of a process in which the handwriting input device displays character string candidates and operation command candidates. [Figure 22] FIG. 5 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 5). [Figure 23] FIG. 10 is a sequence diagram (part 6) illustrating an example of a process in which the handwriting input device displays character string candidates and operation command candidates. [Figure 24] 10 is a diagram illustrating an example of a system configuration of a handwriting input system (embodiment 2); [Figure 25] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing system. [Figure 26] FIG. 2 is an example of a functional block diagram illustrating functions of a handwriting input system in blocks. [Figure 27] FIG. 10 is a sequence diagram (part 1) illustrating an example of a process in which the handwriting input device displays character string candidates and operation command candidates. [Figure 28] FIG. 10 is a sequence diagram (part 2) illustrating an example of a process in which the handwriting input device displays character string candidates and operation command candidates. [Figure 29] FIG. 10 is a sequence diagram (part 3) illustrating an example of a process in which the handwriting input device displays character string candidates and operation command candidates. [Figure 30] FIG. 10 is a sequence diagram (part 4) illustrating an example of a process in which the handwriting input device displays character string candidates and operation command candidates. [Figure 31] FIG. 5 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 5). [Figure 32] FIG. 10 is a sequence diagram (part 6) illustrating an example of a process in which the handwriting input device displays character string candidates and operation command candidates. [Figure 33] FIG. 10 is a sequence diagram (part 7) illustrating an example of a process in which the handwriting input device displays character string candidates and operation command candidates. [Figure 34] FIG. 10 is a diagram showing an example of an operation guide. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, as an example of an embodiment of the present invention, a handwriting input device 2 and a handwriting input method performed by the handwriting input device 2 will be described with reference to the drawings.

[0010] <Operation procedure of conventional handwriting input device> For convenience of explanation of the handwriting input device of this embodiment, first, the operating procedure of a conventional handwriting input device will be briefly explained.

[0011] 1 shows an operation menu displayed by a conventional handwriting input device. On the entire operation panel 101, there are displayed a pen function menu button 1021 for displaying a pen function menu 102, an edit function menu button 1031 for displaying an edit function menu, and an input / output function menu button 1041 for displaying an input / output function menu. The pen function menu 102 allows the user to select the pen color, thickness, and operation mode, the edit function menu 103 allows the user to select to erase, select, cut, copy, and paste an object, and the input / output function menu 104 allows the user to load a template, load a file, save to a file, print, etc.

[0012] As an example of the operating procedure of a conventional handwriting input device, when a user is handwriting straight lines or curves (hereinafter referred to as strokes), in order to erase multiple strokes and start writing again with the pen, the following operating procedure is required. Note that unless otherwise specified, pressing with the pen means pressing with the pen tip. (A1) The edit function menu button 1031 is pressed with a pen to display the edit function menu 103. (A2) When the user presses the "Clear" button 1032 on the edit function menu 103, the handwriting input device automatically erases the edit function menu 103. (A3) If the user writes with the pen across multiple strokes to be erased, the strokes will be erased. (A4) The pen function menu button 1021 is pressed with the pen.

[0013] This procedure erases the strokes to be erased, and the user can then continue to write strokes with the pen. Just erasing a stroke requires four steps. Although there are differences in the number of steps and screen transitions, handwriting input devices often have a similar operating system. This type of operating system is common to all computer products, and is based on an operating system where the user calls the functions they need. If the user does not know how to call up a function, they will not be able to operate it at all, so it is common for handwriting input devices to provide instructions on how to call up functions through tutorial or help functions.

[0014] Similarly, when input / output such as file or print becomes necessary, the user opens the input / output function menu 104. For example, to load a template, the following operation procedure is required. (B1) When the user presses the input / output function menu button 1041 with the pen, the input / output function menu 104 opens. (B2) When the user presses the "template selection" button 1042 on the input / output function menu 104 with the pen, the input / output function menu closes and the template selection window opens. (B3) Press the left or right button with the pen to scroll until the desired template is displayed. (B4) When the desired template is pressed with the pen, the template is loaded and the template selection window closes.

[0015] As described above, conventional handwriting input devices require users to follow step-by-step operating procedures. In other words, because handwriting input devices are based on an operating system that guides users to the next step in an explanatory manner, it is difficult for users to operate them intuitively.

[0016] As shown in Figure 2, it is also being considered to provide functions to the pen itself in order to reduce the number of operating procedures. Figure 2 shows an example of a multi-function pen (hereinafter referred to as an active pen 110). The active pen 110 is a pen with a built-in power source that can send commands to a handwriting input device. The active pen 110 in Figure 2 has one physical switch at the tip of the pen, one at the end of the pen, and two on the side of the pen; the tip is for writing, the end of the pen is for erasing, and the side of the pen is for assigning user functions.

[0017] The user can assign one of the pen side switches to the pen function menu 102 and the other to the edit function menu 103. Although it is convenient that the user can display the pen function menu 102 or the edit function menu 103 by pressing the side button of the active pen 110 held by the user, it still remains cumbersome for the user to press the side button to open the pen function menu 102 or the edit function menu 103 every time the object to be operated is changed.

[0018] Furthermore, if the switch on the end of the pen for erasure is used, the steps (A1), (A2), and (A4) described above are unnecessary, and the end of the pen can be used instead of the tip in step (A3), shortening the operation procedure required for erasure to one step. However, this causes another inconvenience in that it is costly to purchase and operate a special pen called the active pen 110. Furthermore, the active pen 110 requires battery replacement or charging, and depending on the usage situation, measures to prevent loss may also be necessary. [Example]

[0019] <Summary> Next, an overview of the handwriting input device 2 of this embodiment will be described with reference to Fig. 3. Fig. 3 is an example of a diagram for explaining a display example of operation command candidates (also referred to as operation command candidates) displayed by the handwriting input device 2. Fig. 3(a) shows operation command candidates that are displayed when the user has handwritten characters, and Fig. 3(b) shows operation command candidates that are displayed when the user has handwritten an encircling line. First, Fig. 3(a) will be described.

[0020] A. When the user writes the handwriting object 11 (here, a character) "gi" by handwriting with a pen, the handwriting input device 2 performs character recognition and converts it into "gi".

[0021] B. The handwriting input device 2 further converts "gi" into kanji. In Fig. 3(a), it is converted into "gi" as an example, but any kanji that is in the dictionary can be converted.

[0022] C. The handwriting input device 2 further converts "gi" into a predicted character string predicted from "gi." In FIG. 3(a), it is converted into "giji" as an example, but any character string contained in the dictionary can be converted.

[0023] D. The handwriting input device 2 displays operation command candidates including "gi", "gi" or "minutes". That is, operation command candidates 510 related to input / output functions such as "load minutes template" and "save in minutes folder" are displayed as operation commands associated with "gi", "gi" or "minutes". The number of operation command candidates is not limited to two. When an operation command candidate 510 is selected, the handwriting input device 2 executes file input / output according to the selected operation command.

[0024] Therefore, by handwriting "ぎ", the user can display the operation command candidates 510 such as "Load minutes template" or "Save to minutes folder", and can call up the input / output function shown in Figure 1 while handwriting.

[0025] Next, FIG. 3(b) will be described.

[0026] E. The user surrounds the already confirmed object 13 called "minutes" with the handwritten object 11 (here, a box line).

[0027] F. The handwriting input device 2 determines whether the handwritten object 11 and the determined object 13 satisfy a predetermined criterion (for example, whether they overlap).

[0028] G. If the handwritten object 11 and the finalized object 13 satisfy a predetermined criterion, the handwriting input device 2 displays candidate edit or modification operation commands 510. In Fig. 3(b), candidate operation commands 510 related to the editing function, such as "erase," "move," "rotate," and "select," are displayed. The number of candidate operation commands 510 is not limited to four.

[0029] Therefore, by encircling the confirmed object 13 with a surrounding line, the user can display operation command candidates 510 for changing the confirmed object 13, and can call up the editing function while still writing by hand.

[0030] In this way, the handwriting input device 2 of this embodiment can call input / output functions, editing functions, etc. while the user is writing by hand. The pen function can also be called in the same way. Therefore, the number of operation steps required for the user to call any function while writing by hand can be reduced.

[0031] <Terminology> The input means may be any means that allows handwriting by specifying coordinates on a touch panel, such as a pen, a human finger or hand, a rod-shaped member, etc. Also, it may be possible to input by eye gaze.

[0032] A stroke is a series of operations in which a user presses an input device against a display, moves it continuously, and then releases it from the display. Stroke data is information displayed on the display based on the trajectory of coordinates input by the input device. Stroke data may be interpolated as appropriate.

[0033] An object is a display object that is displayed on a display based on stroke data. Object means a target, but in this embodiment it means a display target.

[0034] A handwritten object is an object that is composed of one or more stroke data and is a display target that is handwritten by a user. A handwritten object rectangular area is an area indicated by a circumscribing rectangle that surrounds a handwritten object.

[0035] The determined object is an object that has been converted into a character code (font) by character recognition of a handwritten object and selected by the user, or a handwritten object that has been determined not to be converted into a character code (font).

[0036] The input device is a device that accepts input of information based on the contact position of an input means on a touch panel, and will be described as a handwriting input device 2 in this embodiment.

[0037] An operation command refers to a command that instructs the execution of a specific process, which is prepared for operating the handwriting input device 2. In this embodiment, an operation command for editing, modification, or input / output is taken as an example, but all commands that operate the handwriting input device 2, such as screen inversion, page switching, and setting of operation mode, are subject to the operation command.

[0038] <Overall configuration of the device> The overall configuration of a handwriting input device 2 according to an embodiment of the present invention will be described with reference to Fig. 4. Fig. 4 is a diagram showing the overall configuration of the handwriting input device 2. Here, an electronic whiteboard is shown as an example of the handwriting input device 2.

[0039] 4, a display 220 as an example of a display device is provided on the upper part of the handwriting input device 2. A user can input (draw) characters and the like on the display 220 using a pen 2500.

[0040] <Device hardware configuration> Next, the hardware configuration of the handwriting input device 2 will be described with reference to Fig. 5. As shown in the figure, the handwriting input device 2 has the configuration of an information processing device or a computer. Fig. 5 is an example of a hardware configuration diagram of the handwriting input device 2. As shown in Fig. 5, the handwriting input device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an SSD (Solid State Drive) 204.

[0041] Of these, the CPU 201 controls the overall operation of the handwriting input device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201. The SSD 204 stores various data, such as programs for the handwriting input device 2.

[0042] The handwriting input device 2 also includes a display controller 213 , a touch sensor controller 215 , a touch sensor 216 , a display 220 , and a power switch 222 .

[0043] The display controller 213 controls and manages screen display to output an output image to the display 220, etc. The touch sensor 216 detects contact of the display 220 with a pen 2500, a user's hand, etc. (the pen and the user's hand serve as input means). The touch sensor controller 215 controls processing by the touch sensor 216. The touch sensor 216 inputs and detects coordinates. For example, in the case of an optical method, two light receiving and emitting devices installed at both ends of the upper side of the display 220 emit multiple infrared rays parallel to the display 220, and receive the light that is reflected by a reflecting member installed around the display 220 and returns along the same optical path as the light emitted by the light receiving element. The touch sensor 216 outputs position information of the infrared rays radiated by the two light receiving and emitting devices that have been blocked by an object to the touch sensor controller 215, and the touch sensor controller 215 identifies the coordinate position that is the contact position of the object.

[0044] The power switch 222 is a switch for switching the power of the handwriting input device 2 on and off.

[0045] Furthermore, the handwriting input device 2 includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201 shown in FIG.

[0046] The touch sensor 216 is not limited to an optical type, and various detection means may be used, such as a capacitive touch panel that identifies the contact position by detecting changes in capacitance, a resistive film touch panel that identifies the contact position by voltage changes across two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by an object touching the display unit. The touch sensor 216 may be of a type that does not require an electronic pen to detect whether or not the pen tip is touching. In this case, a fingertip or a pen-shaped stick can be used to perform a touch operation. The pen 2500 does not have to be a long, thin pen-shaped panel.

[0047] <About the device functions> Next, functions of the handwriting input device 2 will be described with reference to Fig. 6. Fig. 6 is an example of a functional block diagram showing functions of the handwriting input device 2 in blocks. The handwriting input device 2 includes a handwriting input unit 21, a display unit 22, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a character string conversion control unit 28, a character string conversion dictionary unit 29, a predictive conversion control unit 30, a predictive conversion dictionary unit 31, an operation command recognition control unit 32, and an operation command definition unit 33. Each function of the handwriting input device 2 is realized by any of the components shown in Fig. 5 operating in response to an instruction from the CPU 201 in accordance with a program loaded from the SSD 204 onto the RAM 203.

[0048] The handwriting input unit 21 is realized by a touch sensor 216 or the like, and accepts handwriting input by the user. The handwriting input unit 21 converts the user's pen input d1 into pen operation data d2 (pen up, pen down, or pen coordinate data) and transmits it to the handwriting input display control unit 23. The pen coordinate data is periodically transmitted as discrete values, and coordinates between the discrete values ​​are interpolated.

[0049] The display unit 22 is realized by a display 220 or the like, and displays handwritten objects and operation menus. The display unit 22 converts the drawing data d3 written to the video memory by the handwriting input display control unit 23 into data according to the characteristics of the display 220, and transmits the data to the display 220.

[0050] The handwriting input display control unit 23 performs overall control over handwriting input and display. The handwriting input display control unit 23 processes the pen operation data d2 from the handwriting input unit 21 and transmits it to the display unit 22 for display. Details of the processing of the pen operation data d2 and the display of strokes will be described later with reference to Figs. 18 to 23.

[0051] The candidate display timer control unit 24 is a timer for controlling the display of selectable candidates. It starts or stops the timer to generate the timing for starting the display of selectable candidates and the timing for erasing the display. The candidate display timer control unit 24 receives a timer start request d4 (or a timer stop request in some cases) from the handwriting input display control unit 23, and transmits a timeout event d5 to the handwriting input display control unit 23.

[0052] The handwriting input saving unit 25 has a storage function for saving user data (handwritten objects / character string objects). The handwriting input saving unit 25 receives user data d6-1 from the handwriting input display control unit 23 and saves it in the handwriting input saving unit 25, and receives an acquisition request d6-2 from the handwriting input display control unit 23 and transmits the user data d7 saved in the handwriting input saving unit 25. The handwriting input saving unit 25 transmits position information d36 of the confirmed object to the operation command recognition control unit 32.

[0053] The handwriting recognition control unit 26 is a recognition engine that performs online handwriting recognition. Unlike a typical OCR (Optical Character Reader), it recognizes characters (not only Japanese but also multiple languages ​​such as English), numbers, symbols (%, $, &, etc.), shapes (lines, circles, triangles, etc.) in parallel with the user's pen operation. Various algorithms have been devised for recognition methods, but this embodiment uses known techniques and will not be described in detail.

[0054] The handwriting recognition control unit 26 receives the pen operation data d8-1 from the handwriting input display control unit 23, performs handwriting recognition, and stores handwriting-recognized character string candidates. The handwriting recognition control unit 26 also stores language character string candidates converted from the handwriting-recognized character string candidates d12 using the handwriting recognition dictionary unit 27. Separately, when an acquisition request d8-2 is received from the handwriting input display control unit 23, the handwriting recognition control unit 26 transmits the stored handwriting-recognized character string candidates and language character string candidates d9 to the handwriting input display control unit 23.

[0055] The handwriting recognition dictionary unit 27 stores dictionary data for language conversion in handwriting recognition. The handwriting recognition dictionary unit 27 receives handwriting recognition character string candidates d12 from the handwriting recognition control unit 26, converts them into linguistically probable language character string candidates d13, and transmits them to the handwriting recognition control unit 26. For example, in the case of Japanese, hiragana characters are converted into kanji characters or katakana characters.

[0056] The character string conversion control unit 28 controls the conversion of conversion character string candidates into character strings. A conversion character string is a character string that is likely to be generated by including a handwriting recognition character string or a language character string. The character string conversion control unit 28 receives handwriting recognition character strings and language character string candidates d11 from the handwriting recognition control unit 26, converts them into conversion character string candidates using the character string conversion dictionary unit 29, and stores them. Separately, when an acquisition request d14 is received from the handwriting input display control unit 23, the character string conversion control unit 28 transmits the stored conversion character string candidates d15 to the handwriting input display control unit 23.

[0057] The character string conversion dictionary unit 29 stores dictionary data for character string conversion. The character string conversion dictionary unit 29 receives handwriting-recognized character strings and language character string candidates d17 from the character string conversion control unit 28, and transmits converted character string candidates d18 to the character string conversion control unit 28.

[0058] The predictive conversion control unit 30 receives the handwriting-recognized character string and language character string candidate d10 from the handwriting recognition control unit 26 and receives the converted character string candidate d16 from the character string conversion control unit 28, and converts each of them into a predicted character string candidate using the predictive conversion dictionary unit 31. A predicted character string candidate is a character string that is likely to be generated by including the handwriting-recognized character string, language string, or converted character string. If an acquisition request d19 is separately received from the handwriting input display control unit 23, the predictive conversion control unit 30 transmits the predicted character string candidate d20 to the handwriting input display control unit 23.

[0059] The predictive conversion dictionary unit 31 stores dictionary data for predictive conversion. The predictive conversion dictionary unit 31 receives handwritten recognition character strings, language character string candidates, and conversion character string candidates d21 from the predictive conversion control unit 30, and transmits predicted character string candidates d22 to the predictive conversion control unit 30.

[0060] The operation command recognition control unit 32 receives the handwriting recognition character string and language character string candidates d30 from the handwriting recognition control unit 26, receives the converted character string candidates d28 from the character string conversion control unit 28, and receives the predicted character string candidates d29 from the predictive conversion control unit 30. Then, for each, it transmits an operation command conversion request d26 to the operation command definition unit 33 and receives the operation command candidates d27 from the operation command definition unit 33. The operation command recognition control unit 32 holds the operation command candidates d27.

[0061] If the operation command conversion request d26 partially matches the operation command definition, the operation command definition unit 33 sends operation command candidates d27 to the operation command recognition control unit 32.

[0062] The operation command recognition control unit 32 also receives pen operation data d24-1 from the handwriting input display control unit 23, transmits a request d23 to the handwriting input storage unit 25 to acquire position information of a confirmed object that was previously input and confirmed, and stores the confirmed object designated by the pen operation data as a selected object (including position information). The operation command recognition control unit 32 identifies a selected object that satisfies the position of the pen operation data d24-1 and a predetermined criterion. Separately, when an acquisition request d24-2 is received from the handwriting input display control unit 23, the operation command recognition control unit 32 transmits a selected object d25 identified as a candidate for the stored operation command to the handwriting input display control unit 23.

[0063] <About user authentication> In this embodiment, control is performed using the result of user authentication, so it is preferable that the handwriting input device 2 has a function for performing user authentication. Therefore, the function relating to user authentication will be described with reference to FIG.

[0064] Fig. 7 is a block diagram showing functions related to user authentication possessed by the handwriting input device 2. Note that Fig. 7 shows only the handwriting input display control unit 23 as a function related to the user authentication unit 34 (user authentication means), but the functions shown in Fig. 6 may be used as a result of user authentication.

[0065] The authentication information acquisition unit 35 acquires authentication information d31 from the user. The authentication information d31 may be an IC card number, a user ID and password, or biometric information such as a fingerprint. The user authentication unit 34 acquires authentication information d32 from the authentication information acquisition unit 35 and searches the user information DB 36 for authentication information d33. If a user matching the search is found, the user information d34 is acquired from the user information DB 36. The user information may be any information that represents the user's attributes, such as a user name, password, the user's computer name, department, or authority.

[0066] The user authentication unit 34 transmits the user information d35 to the handwriting input display control unit 23, so that the handwriting input display control unit 23 can execute operation commands using the user information. Operation commands using user information will be described with reference to FIG. 12.

[0067] The authentication function may be provided by an external authentication server or the like, in addition to the handwriting input device 2. In this case, the handwriting input device 2 transmits authentication information to the authentication server and acquires the authentication result and user information.

[0068] <About predefined control data> Next, the predefined control data used by the handwriting input device 2 for various processes will be described with reference to Fig. 8. Fig. 8 shows an example of the predefined control data. In the example of Fig. 8, control data is shown for each control item.

[0069] The selectable candidate display timer 401 defines the time until the selectable candidates are displayed (an example of a first time). This is because the selectable candidates are not displayed while handwriting. In FIG. 8, this means that the selectable candidates are displayed if the pen does not come down within TimerValue=500 [ms] after the pen is up. The selectable candidate display timer 401 is held by the candidate display timer control unit 24. The selectable candidate display timer 401 is used when the selectable candidate display timer starts in step S4.8 of FIG. 20, which will be described later.

[0070] The selectable candidate deletion timer 402 defines the time until the displayed selectable candidates are deleted (an example of a second time). This is to delete the selectable candidates if the user does not select a selectable candidate. In FIG. 8, this means that if a selectable candidate is not selected within TimerValue=5000 [ms] from the display of the selectable candidates, the selectable candidate display will be deleted. The selectable candidate deletion timer 402 is held by the candidate display timer control unit 24. The selectable candidate deletion timer 402 is used when the selectable candidate display deletion timer starts in step S4.8.1.8 of FIG. 21.

[0071] The handwritten object neighborhood rectangular area 403 defines a rectangular area that is considered to be near the handwritten object. In the example of FIG. 8, the handwritten object neighborhood rectangular area 403 is a rectangular area obtained by enlarging the handwritten object rectangular area by 50% (Horizontal) of the estimated character size in the horizontal direction and by 80% (Vertical) of the estimated character size in the vertical direction. In the example of FIG. 8, the rectangular area is a percentage of the estimated character size (specified as a percentage), but it can also be a fixed length by using units such as "mm". The handwritten object neighborhood rectangular area 403 is stored in the handwritten input storage unit 25. The estimated character size 405 is used to determine the overlap state between the handwritten object neighborhood rectangular area and the stroke rectangular area in step S4.2 of FIG. 19.

[0072] The estimated writing direction / character size determination condition 404 defines constants for determining the writing direction and the measurement direction of the character size. In the example of FIG. 8, if the difference between the time when the first stroke was added to the rectangular area of ​​the handwritten object and the time when the last stroke was added is MinTime=1000 [ms] or more, the difference between the horizontal distance (width) and the vertical distance (height) of the rectangular area of ​​the handwritten object is MinDiff=10 [mm] or more, and the horizontal distance is longer than the vertical distance, the estimated writing direction is determined to be "horizontal writing" and the estimated character size is determined to be the vertical distance. If the horizontal distance is shorter than the vertical distance, the estimated writing direction is determined to be "vertical writing" and the estimated character size is determined to be the horizontal distance. If these conditions are not met, the estimated writing direction is determined to be "horizontal writing" (DefaultDir="Horizontal") and the estimated character size is determined to be the longer of the horizontal distance and the vertical distance. The estimated writing direction / character size determination condition 404 is held in the handwriting input storage unit 25. The estimated writing direction / character size judgment condition 404 is used to obtain the estimated writing direction in step S4.8.1.5 of FIG. 21 and to obtain the character string object font in step S7.8.

[0073] The estimated character size 405 defines data for estimating the size of characters, etc. In the example of FIG. 8, this means that the estimated character size determined by the estimated writing direction / character size determination condition 404 is compared with the small character 405a (hereinafter referred to as the minimum font size) and the large character 405c (hereinafter referred to as the maximum font size) of the estimated character size 405. If the estimated character size is smaller than the minimum font size, the estimated character size is determined to be the minimum font size. If the estimated character size is larger than the maximum font size, the estimated character size is determined to be the maximum font size. Otherwise, the estimated character size is determined to be the character size of the medium character 405b. The estimated character size 405 is stored in the handwriting input storage unit 25. The estimated character size 405 is used to obtain the character string object font in step S7.8 of FIG. 23.

[0074] Specifically, the handwriting input storage unit 25 compares the estimated character size determined by the estimated writing direction / character size determination condition 404 with the FontSize of the estimated character size 405, and uses the font with the closest size. For example, if the estimated character size is 25 mm (FontSize for small characters) or less, it is determined to be a "small character," if the estimated character size is more than 25 mm but less than 50 mm (FontSize for medium characters), it is determined to be a "medium character," and if the estimated character size is more than 100 mm (FontSize for large characters), it is determined to be a "large character." "Small characters" 405a use a 25 mm Mincho font (FontStyle="Mincho" FontSize="25mm"), "medium characters" 405b use a 50 mm Mincho font (FontStyle="Mincho" FontSize="50mm"), and "large characters" 405c use a 100 mm Gothic font (FontStyle="Gothic" FontSize="100mm"). If you want more font sizes or styles, you can increase the number of estimated font sizes 405.

[0075] The crossing line determination condition 406 defines data used to determine whether multiple objects have been selected. In the example of FIG. 8, if the handwritten object is a single stroke, and the length of the long side of the handwritten object is 100 mm or more (MinLenLongSide="100mm"), the length of the short side is 50 mm or less (MaxLenShortSide="50mm"), and there is an object whose overlap rate with the handwritten object in the long side direction and short side direction is 80% or more (MinOverLapRate="80%") (a predetermined overlap rate or more), it is determined that multiple objects have been selected (selected objects). Instead of "more than or equal to," the overlap rate may be determined to be "greater than." The crossing line determination condition 406 is held by the operation command recognition control unit 32. The crossing line determination condition 406 is used in the crossing line determination for determining the selected object in step S4.7.6.2 of FIG. 20.

[0076] The encircling line determination condition 407 defines data used to determine whether an object is an encircling line. In the example of FIG. 8, the operation command recognition control unit 32 determines a fixed object whose overlap rate in the long side direction and short side direction of the handwritten object is 100% or more (MinOverLapRate="100%") as a selected object (a predetermined overlap rate or more). Instead of "above or above," the overlap rate may be determined to be "greater than." The encircling line determination condition 407 is held by the operation command recognition control unit 32. The encircling line determination condition 407 is used in the encircling line determination for determining the selected object in step S4.6.7.2 of FIG. 20.

[0077] Note that either the cross-line determination condition 406 or the enclosing line determination condition 407 may be given priority in the determination. For example, if the cross-line determination condition 406 is set to a relaxed value (to make it easier to select a cross-line) and the enclosing line determination condition 407 is set to a strict value (to make it possible to select only an enclosing line), the operation command recognition control unit 32 may give priority to the enclosing line determination condition 407 in the determination.

[0078] <Example of dictionary data> The dictionary data will be described with reference to Fig. 9 to Fig. 11. Fig. 9 is an example of dictionary data in the handwriting recognition dictionary unit 27, Fig. 10 is an example of dictionary data in the character string conversion dictionary unit 29, and Fig. 11 is an example of dictionary data in the predictive conversion dictionary unit 31. These dictionary data are used in steps S4.7.2 to S4.7.4 in Fig. 20, respectively.

[0079] In this embodiment, the conversion results using the dictionary data of the handwriting recognition dictionary unit 27 in FIG. 9 are called language string candidates, the conversion results using the dictionary data of the string conversion dictionary unit 29 in FIG. 10 are called converted string candidates, and the conversion results using the dictionary data of the predictive conversion dictionary unit 31 in FIG. 11 are called predicted string candidates. For each dictionary data, "before conversion" represents the string to be searched for in the dictionary data, "after conversion" represents the converted string corresponding to the string to be searched for, and "probability" represents the probability of selection by the user. The probability is calculated from the results of users selecting each string in the past. Therefore, the probability may be calculated for each user. Various algorithms have been devised as a method of calculating probability, but the calculation may be performed in an appropriate manner as appropriate, and details will be omitted. This embodiment is characterized by displaying string candidates in descending order of selection probability based on the estimated writing direction.

[0080] The dictionary data of the handwriting recognition dictionary unit 27 in Figure 9 shows that handwritten "gi" is converted to "gi" with a probability of 0.55 and to "gi" with a probability of 0.4, and handwritten "gishi" is converted to "gishi" with a probability of 0.5 and to "gishi" with a probability of 0.45. The same applies to other "before conversion" character strings. In Figure 9, the "before conversion" character string is handwritten hiragana, but characters other than hiragana may also be registered as "before conversion."

[0081] 10 indicates that the character string "" is converted to "" with a probability of 0.95, and the character string "" is converted to "" with a probability of 0.85. The same applies to other "before conversion" character strings.

[0082] The dictionary data of the predictive conversion dictionary unit 31 in Fig. 11 indicates that the character string "minutes" is converted to "addressee of the minutes" with a probability of 0.65, and the character string "skill test" is converted to "approval of skill test" with a probability of 0.75. In the example of Fig. 11, all character strings before conversion are kanji, but characters other than kanji may also be registered.

[0083] The dictionary data is not language-dependent, and any character strings may be registered before and after conversion.

[0084] <Operation command definition data held by the operation command definition section> Next, the operation command definition data used by the operation command recognition control unit 32 will be described with reference to Fig. 12 and Fig. 13. Fig. 12 shows an example of the operation command definition data and system definition data held by the operation command definition unit 33.

[0085] Fig. 12(a) shows an example of operation command definition data. The operation command definition data in Fig. 12(a) is an example of operation command definition data when no selected object is selected by a handwritten object, and applies to all operation commands that operate the handwriting input device 2. The operation command in Fig. 12(a) has an operation command name (Name), a string (String) that partially matches a string candidate, and an operation command string (Command) to be executed. "%~%" in the operation command string is a variable, and is associated with system definition data as shown in Fig. 12(b). In other words, "%~%" is replaced with the system definition data shown in Fig. 12(b).

[0086] First, the operation command definition data 701 indicates that the operation command name is "Read minutes template," the string that partially matches the string candidate is "minutes" or "template," and the operation command string to be executed is "ReadFile https: / / %username%:%password%@server.com / template / minutes.pdf." In this example, the operation command string to be executed contains the system definition data "%~%," and "%username%" and "%password%" are replaced by the system definition data 704 and 705, respectively. Therefore, the operation command string to be ultimately executed is the string "ReadFile https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf," which indicates that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf" is to be read (ReadFile).

[0087] The operation command definition data 702 indicates that the operation command name is "Save to minutes folder," the strings that partially match the string candidate are "minutes" or "save," and the operation command string to be executed is "WriteFile https: / / %username%:%password%@server.com / minutes / %machinename%_%yyyy-mm-dd%.pdf." As with the operation command definition data 701, "%username%," "%password%," and "%machinename%" in the operation command string are replaced with system definition data 704 to 706, respectively. Note that "%yyyy-mm-dd%" indicates that it should be replaced with the current date. For example, if the current date is September 26, 2018, it should be replaced with "2018-09-26." The final operation command to be executed is "WriteFile https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf", which indicates that the minutes should be saved (WriteFile) in the file "https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf".

[0088] The operation command definition data 703 indicates that the operation command name is "Print", the string that partially matches the string candidate is "Print" or "Print", and the operation command string to be executed is "PrintFile https: / / %username%:%password%@server.com / print / %machinename%-"%yyyy-mm-dd%.pdf". When the operation command string is replaced in the same way as in the operation command definition data 702, the operation command to be executed finally becomes "PrintFile https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf", which indicates that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf" will be printed (PrintFile). In other words, the file is sent to the server. When the user makes the printer communicate with the server and specifies a file, the printer prints the contents of the file on paper.

[0089] In this way, the operation command definition data 701 to 703 can be identified from the character string candidates, and the operation command can be displayed by the user's handwriting. Also, if the user authentication is successful, "%username%", "%password%", etc. in the operation command definition data are replaced with the user information, making it possible to input and output files associated with the user.

[0090] If user authentication is not performed (including the case where authentication fails but the user can still use the handwriting input device 2), the handwriting input device 2 replaces the username and password with the pre-set password of the handwriting input device 2, such as "%username%" and "%password%." Therefore, file input and output can be performed in association with the handwriting input device 2 without user authentication.

[0091] Next, we will explain the operation command definition data when there is a handwritten object, that is, the operation command definition data for editing and modification. Fig. 13 shows an example of the operation command definition data when there is a selected object selected by a handwritten object. The operation command definition data in Fig. 13 has the operation command name (Name), the group name (Group) of operation command candidates, and the operation command string to be executed (Command).

[0092] The operation command definition data 707 defines operation commands for editing (Group="Edit"), and is an example of definition data for the names of operation commands for editing: "erase," "move," "rotate," and "select." In other words, these operation commands are displayed for the selected object, allowing the user to select the desired operation command.

[0093] The operation command definition data 708 defines operation commands for the decoration system (Group="Decorate"), and is an example of definition data for the names of operation commands for the decoration system: "bold", "thin", "larger", "smaller", and "underline". These operation commands are displayed for the selected object, allowing the user to select the desired operation command. In addition, color operation commands may also be displayed.

[0094] Therefore, when the user selects a selection object with a handwritten object, the operation command definition data 707 and 708 are specified, and the user can display the operation command by handwriting.

[0095] <Example of displaying operation command candidates> Fig. 14 shows an example of displaying operation command candidates based on the operation command definition data (when no selected object is present) of Fig. 12(a). The procedure for displaying operation command candidates will be described in detail with reference to the sequence diagrams of Figs. 18 to 23.

[0096] FIG. 14 shows an example of an operation guide 500 and selectable candidates 530 displayed by the operation guide 500. The operation guide 500 is displayed when the user writes a handwritten object 504 by hand (when the selectable candidate display timer times out). The operation guide 500 has an operation header 520, operation command candidates 510, handwriting-recognized character string candidates 506, conversion character string candidates 507, predicted character string candidates 508, and a handwritten object rectangular area display 503. The selectable candidates 530 are the operation command candidates 510, handwriting-recognized character string candidates 506 (including language string candidates 571, if any), conversion character string candidates 507, and predicted character string candidates 508. The selectable candidates 530 excluding the operation command candidates 510 are referred to as character string candidates 539.

[0097] The operation header 520 has buttons 501, 509, 502, and 505. Button 501 accepts an operation to switch predictive conversion on and off. In the example of FIG. 14, when the user presses button 501 displaying "Predict," handwriting input unit 21 accepts the press and notifies handwriting input display control unit 23 of this, and display unit 22 changes the display to button 501 displaying "Kana." After the change, predictive conversion is not performed (converted character string candidate 507 and predicted character string candidate 508 are not displayed), and operation command candidate 510, handwritten recognition character string candidate 506, and language character string candidate (kanji converted from kana to kanji, such as 'gi') are displayed.

[0098] Button 502 operates the page of the candidate display. In the example of FIG. 14, there are three candidate display pages, and the first page is currently displayed. Button 505 accepts erasure of the operation guide 500. When the user presses button 505, the handwriting input unit 21 accepts it and notifies the handwriting input display control unit 23 of that fact, and the display unit 22 erases the display except for the handwritten object. Button 509 accepts erasure of all displays. When the user presses button 509, the handwriting input unit 21 accepts it and notifies the handwriting input display control unit 23 of that fact, and the display unit 22 erases all displays shown in FIG. 14, including the handwritten object, allowing the user to start handwriting again from the beginning.

[0099] In Fig. 14, handwritten object 504 is the character "gi" handwritten by the user. A handwritten object rectangular area display 503 is displayed surrounding handwritten object 504. The display procedure will be explained with reference to the sequence diagrams of Figs. 18 to 23. In the example of Fig. 14, handwritten object rectangular area display 503 is displayed in a dotted line frame.

[0100] The character string candidates are arranged in descending order of probability in the handwriting recognition character string candidate 506, the converted character string candidate 507, and the predicted character string candidate 508. The character "gi" in the handwriting recognition character string candidate 506 is a candidate for the recognition result. In this example, the character "gi" is correctly recognized.

[0101] The converted character string candidate 507 is a character string candidate converted from the language character string candidate. In this example, "skill test" is an abbreviation for "technical mass production trial." The predicted character string candidate 508 is a predicted character string candidate converted from a handwritten recognition character string, a language character string candidate, or a converted character string candidate. In this example, "Approve the skill test" and "Address to send the minutes" are displayed.

[0102] The operation command candidate 510 is a candidate operation command selected based on the operation command definition data 701 to 703 in Fig. 11(a). In the example in Fig. 14, the first character of the line "》" 511 is shown to be a candidate operation command. In Fig. 14, there is no selection data to be selected by the handwritten object 504 "ぎ", and the character string candidate for "ぎ" "meiji" (minutes) partially matches the operation command definition data 701 and 702 in Fig. 11(a), so it is displayed as the operation command candidate 510.

[0103] When the user selects "Load minutes template," the operation command defined in operation command definition data 701 is executed, and when the user selects "Save in minutes folder," the operation command defined in operation command definition data 702 is executed. In this way, operation command candidates are displayed only when operation command definition data containing the converted character string is found, so they may not always be displayed.

[0104] As shown in FIG. 14, character string candidates and operation command candidates are displayed simultaneously (together), so the user can freely select either the character string candidate or the operation command that he or she intended to input.

[0105] <Example of specifying a selected object> The handwriting input device 2 of this embodiment allows the user to select a fixed object by handwriting and specify a selected object, which becomes a target for editing or modification.

[0106] Fig. 15 is an example diagram for explaining an example of designating a selected object. In Fig. 15, a handwritten object 11 is displayed with a solid black line, a handwritten object rectangular area 12 is displayed with gray shading, a confirmed object 13 is displayed with a black line, and a selected object rectangular area 14 is displayed with a dotted line. Note that lowercase letters are added to the symbols when distinguishing between them. In addition, the crossing line determination condition 406 or the enclosing line determination condition 407 of the predefined control data shown in Fig. 8 is used as the determination condition for determining whether a confirmed object is a selected object (whether or not there is a predetermined relationship).

[0107] 15(a) shows an example in which the user specifies two horizontally written fixed objects 13a and 13b with a crossover line (hand-drawn object 11a). In this example, the length H1 of the short side and the length W1 of the long side of the hand-drawn object rectangular area 12a satisfy the crossover line determination condition 406, and the overlap rate with the fixed objects 13a and 13b also satisfies the crossover line determination condition 406. Therefore, both the fixed objects 13a and 13b, "minutes" and "giji," are specified as selected objects.

[0108] 15B shows an example in which a horizontally written fixed object 13c is specified by an encircling line (hand-drawn object 11b). In this example, only the fixed object 13c, "minutes," whose overlap rate with the fixed object 13c and the hand-drawn object rectangular area 12c satisfies the encircling line determination condition 407, is specified as the selected object.

[0109] 15(c) is an example in which multiple vertically written determined objects 13d and 13e are specified by a crossing line (hand-drawn object 11c). In this example, similar to FIG. 15(a), the length H1 of the short side and the length W1 of the long side of the hand-drawn object rectangular area 12d satisfy the crossing line determination condition 406, and the overlapping rates with the two determined objects 13d and 13e, "minutes" and "giji", respectively, satisfy the crossing line determination condition 406, so both the determined objects 13d and 13e, "minutes" and "giji", are specified as selected objects.

[0110] 15(d) shows an example in which a vertically written fixed object 13f is specified by enclosing it (handwritten object 11d). In this example, just like in FIG. 15(b), only the fixed object 13f, "minutes," is specified as the selected object.

[0111] <Example of displaying operation command candidates> Figure 16 shows an example of displaying operation command candidates based on the operation command definition data when there is the handwritten object shown in Figure 13. Figure 16(a) shows candidates for editing operation commands, and Figure 16(b) shows candidates for modification operation commands. Figure 16(a) also shows an example in which a selected object is specified by the handwritten object 11a in Figure 15(a).

[0112] As shown in Figures 16(a) and 16(b), the operation command candidates displayed following the first character "»" 511 are a main menu 550. The name of the operation command last executed or the name of the first operation command in the operation command definition data is displayed in the main menu 550. The first character "»" 511a on the first line is a candidate for an editing-related operation command, and the first character "»" 511b on the second line is a candidate for a modification-related operation command.

[0113] The "〉" 512 at the end of the line indicates that there is a sub-menu (an example of a sub-menu button). The "〉" 512a in the first line displays the sub-menu (last selected) of the candidate operation commands in the editing system, and the "〉" 512b in the second line displays the remaining sub-menus of the candidate operation commands in the modification system. When the user presses "〉" 512, the sub-menu 560 is displayed on its right side. All the operation commands defined in the operation command definition data are displayed in the sub-menu 560. In the display example of FIG. 16(a), the sub-menu 560 corresponding to the "〉" 512a in the first line is also displayed since the main menu is displayed. It may be displayed by pressing the "〉" 512a in the first line.

[0114] When the user presses any operation command name with the pen, the Command of the operation command definition data associated with the operation command name is executed by the handwritten input display control unit 23 for the selected object. That is, when "Delete" 521 is selected, "Delete" is executed, when "Move" 522 is selected, "Move" is executed, when "Rotate" 523 is selected, "Rotate" is executed, and when "Select" 524 is selected, "Select" is executed respectively.

[0115] For example, when the user presses "Delete" 521 with the pen, "Minutes" and "Provisional" can be deleted. When "Move" 522, "Rotate" 523, or "Select" 524 is pressed, the bounding box (the circumscribed rectangle of the selected object) is displayed. With "Move" 522 and "Rotate" 523, it can be moved or rotated by the drag operation of the pen, and with "Select" 524, other operations on the bounding box can be executed.

[0116] The strings "—" 541, "—," 542, "~" 543, "→" 544, "⇒" 545, which are candidate strings other than the candidate operation commands, are the recognition results of the strike-through line (handwritten object 11a). When the user intends to input a string rather than an operation command, the candidate string can be selected.

[0117] In Fig. 16(b), pressing ">" 512b on the second line displays the submenu of Fig. 16(b). Similar to Fig. 16(a), the display example of Fig. 16(b) also displays a main menu 550 and a submenu 560. Based on the operation command definition data of Fig. 13, when "thicken" 531 is selected, the handwriting input display control unit 23 executes "Thick," when "thinner" 532 is selected, "Thin," when "bigger" 533 is selected, "Large," when "smaller" 534 is selected, "Small," and when underline 535 is selected, "Underline," respectively, for the selected object.

[0118] Note that fixed values ​​are separately defined for how thick to make the line when "Thicken" 531 is selected, how thin to make the line when "Thinner" 532 is selected, how large to make the line when "Larger" 533 is selected, how small to make the line when "Smaller" 534 is selected, and the type of line when underline 535 is selected. Alternatively, it would be better if a separate selection menu is opened when the submenu in Figure 16(b) is selected, allowing the user to make adjustments.

[0119] When the user presses "thicken" 531 with the pen, the handwriting input display control unit 23 thickens the lines that make up the final objects 13a and 13b, "minutes" and "giji." When the user presses "thin" 532 with the pen, the handwriting input display control unit 23 thins the lines that make up "minutes" and "giji." When the user presses "larger" 533 with the pen, the handwriting input display control unit 23 enlarges the lines. When the user presses "smaller" 534 with the pen, the handwriting input display control unit 23 shrinks the lines. When the user presses "underline" 535 with the pen, the handwriting input display control unit 23 adds an underline.

[0120] FIG. 17 shows an example of displaying operation command candidates based on operation command definition data when there is a handwritten object as shown in FIG. 13. The difference from FIGS. 16(a) and 16(b) is that FIGS. 17(a) and 17(b) show an example in which a selection object is specified by the handwritten object 11b (encircling line) of FIG. 15(b). As can be seen by comparing FIGS. 16 and 17, there is no difference in the operation command candidates displayed depending on whether the handwritten object is a line or an encircling line. The handwritten input display control unit 23 displays operation command candidates on the display unit 22 when a selection object is specified. However, the handwritten object may be recognized and the operation command candidates displayed may be changed depending on the handwritten object. In this case, the recognized handwritten object (such as a one or a circle) is associated with operation command definition data as shown in FIG. 13.

[0121] In FIG. 17, the character string candidates other than the operation command candidates, “○” 551, “∞” 552, “0” 553, “00” 554, and “ロ” 555, are the recognition results of the enclosing line (handwritten object 11b), and if the user intends to input a character string instead of an operation command, the user can select a character string candidate.

[0122] <Operation procedure> The operation of the handwriting input device 2 will be described with reference to the above configuration and Figs. 18 to 23. Figs. 18 to 23 are sequence diagrams illustrating an example of the process in which the handwriting input device 2 displays character string candidates and operation command candidates. The process in Fig. 18 starts when the handwriting input device 2 is started (when an application is started). Note that, due to space limitations, Figs. 18 to 23 use symbols to indicate the functions of Fig. 6.

[0123] S1: First, the handwriting input display control unit 23 sends a handwriting object start signal to the handwriting input storage unit 25. The handwriting input storage unit 25 allocates a handwriting object area (a memory area for storing a handwritten object). The handwriting object area may be allocated after the user brings the pen into contact with the handwriting input unit 21.

[0124] S2: Next, the user touches the pen to the handwriting input unit 21. The handwriting input unit 21 detects the pen down and transmits it to the handwriting input display control unit 23.

[0125] S2.1: The handwriting input display control unit 23 transmits a stroke start signal to the handwriting input storage unit 25, and the handwriting input storage unit 25 reserves a stroke area.

[0126] S3: When the user moves the pen while keeping it in contact with the handwriting input unit 21, the handwriting input unit 21 transmits the pen coordinates to the handwriting input display control unit 23.

[0127] S3.1: The handwriting input display control unit 23 transmits pen coordinate complement display data (data for interpolating discrete pen coordinates) to the display unit 22. The display unit 22 uses the pen coordinate complement display data to interpolate the pen coordinates and display a line.

[0128] S3.2: The handwriting input display control unit 23 transmits the pen coordinates and the time of reception thereof to the handwriting input storage unit 25. The handwriting input storage unit 25 adds the pen coordinates to the stroke. While the user is moving the pen, the handwriting input unit 21 periodically repeats transmitting the pen coordinates to the handwriting input display control unit 23, and therefore the processes of steps S3 to S3.2 are repeated until the pen is lifted.

[0129] S4: When the user removes the pen from the handwriting input unit 21, the handwriting input unit 21 transmits a pen-up signal to the handwriting input display control unit 23.

[0130] S4.1: The handwriting input display control unit 23 sends a stroke end signal to the handwriting input storage unit 25, and the handwriting input storage unit 25 confirms the pen coordinates of the stroke. Once the pen coordinates of the stroke have been confirmed, it becomes impossible to add pen coordinates to the stroke.

[0131] S4.2: Next, the handwriting input display control unit 23 transmits the overlap status of the handwritten object neighborhood rectangular area and the stroke rectangular area to the handwriting input saving unit 25 based on the handwritten object neighborhood rectangular area 403. The handwriting input saving unit 25 calculates the overlap status and transmits the overlap status to the handwriting input display control unit 23.

[0132] The subsequent steps S4.3 to S4.5 are executed if the handwritten object neighborhood rectangular area and the stroke rectangular area do not overlap.

[0133] S4.3: If the rectangular area near the handwritten object and the rectangular area of ​​the stroke do not overlap, one handwritten object is determined, and the handwritten input / display control unit 23 transmits a command to clear the stored data to the handwriting recognition control unit 26.

[0134] S4.3.1 to S4.3.3: The handwriting recognition control unit 26 sends a retained data clear command to the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32. The data relating to character string candidates and operation command candidates retained by the handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 are cleared. Note that at the time of clearing, the last handwritten stroke has not been added to the handwritten object.

[0135] S4.4: The handwriting input display control unit 23 sends a handwriting object end notification to the handwriting input saving unit 25. The handwriting input saving unit 25 finalizes the handwritten object. Finalizing a handwritten object means that one handwritten object has been completed (no more strokes can be added).

[0136] S4.5: The handwriting input display control unit 23 transmits a handwriting object start signal to the handwriting input storage unit 25. In preparation for the start of handwriting of the next handwritten object (pen down), the handwriting input storage unit 25 reserves a new handwritten object area.

[0137] S4.6: Next, the handwriting input display control unit 23 sends a stroke addition to the handwriting input saving unit 25 for the stroke completed in step S4.1. If steps S4.3 to S4.5 have been executed, the added stroke is the first stroke of the handwritten object, and the handwriting input saving unit 25 adds the stroke data to the starting handwritten object. If steps S4.3 to S4.5 have not been executed, the added stroke is added to the handwritten object that is already being handwritten.

[0138] S4.7: Next, the handwriting input display control unit 23 transmits a stroke addition to the handwriting recognition control unit 26. The handwriting recognition control unit 26 adds the stroke data to the stroke data storage area where the character string candidate is stored.

[0139] S4.7.1: The handwriting recognition control unit 26 executes handwriting recognition on this stroke data storage area.

[0140] S4.7.2: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidates, which are the execution results, to the handwriting recognition dictionary unit 27. The handwriting recognition dictionary unit 27 transmits the linguistically likely language character string candidates to the handwriting recognition control unit 26.

[0141] S4.7.3: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidate and the received language character string candidate to the character string conversion control unit 28.

[0142] S4.7.3.1: The character string conversion control unit 28 transmits the handwriting recognition character string candidates and the language character string candidates to the character string conversion dictionary unit 29. The character string conversion dictionary unit 29 transmits the conversion character string candidates to the character string conversion control unit 28.

[0143] S4.7.3.2: The character string conversion control unit 28 transmits the received converted character string candidates to the predictive conversion control unit 30.

[0144] S4.7.3.2.1: The predictive conversion control unit 30 transmits the received converted character string candidates to the predictive conversion dictionary unit 31. The predictive conversion dictionary unit 31 transmits the predicted character string candidates to the predictive conversion control unit 30.

[0145] S4.7.3.2.2: The predictive conversion control unit 30 transmits the received predicted character string candidate to the operation command recognition control unit 32.

[0146] S4.7.3.2.2.1: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) matching the predicted character string candidate.

[0147] Thereafter, the same processing is carried out up to the transmission of operation command candidates in steps S4.7.3.3 to S4.7.5.1. S4.7.3.3: The character string conversion control unit 28 transmits the received conversion character string candidate to the operation command recognition control unit 32.

[0148] S4.7.3.3.1: The operation command recognition control unit 32 transmits the received conversion character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) that matches the conversion character string candidate.

[0149] S4.7.4: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidates and the language character string candidates to the predictive conversion control unit 30.

[0150] S4.7.4.1: The predictive conversion control unit 30 transmits the handwritten recognition character string candidate and the received language character string candidate to the predictive conversion dictionary unit 31. The predictive conversion dictionary unit 31 transmits the predicted character string candidate to the predictive conversion control unit 30.

[0151] S4.7.4.2: The predictive conversion control unit 30 transmits the received predicted character string candidate to the operation command recognition control unit 32.

[0152] S4.7.4.2.1: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) matching the predicted character string candidate.

[0153] S4.7.5: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidates and the received language character string candidates to the operation command recognition control unit 32.

[0154] S4.7.5.1: The operation command recognition control unit 32 transmits the handwriting recognition character string candidate and the received language character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) that matches the language character string candidate.

[0155] S4.7.6: Next, the handwriting recognition control unit 26 sends the stroke addition to the operation command recognition control unit 32.

[0156] S4.7.6.1: The operation command recognition control unit 32 transmits the position information of the confirmed object to the handwriting input storage unit 25. The handwriting input storage unit 25 transmits the position information of the confirmed object to the operation command recognition control unit 32.

[0157] S4.7.6.2: To determine the selected object, the operation command recognition control unit 32 determines whether or not the position information of the stroke received from the handwriting recognition control unit 26 when the stroke was added in step S4.7.6 and the position information of the confirmed object received from the handwriting input storage unit 25 have a predetermined relationship based on the crossing line determination condition 406 and the encircling line determination condition 407, and stores the confirmed object that can be determined to be selected as the selected object. In addition, in this case, since the selected object is specified, candidates for input / output operation commands are obtained from the operation command definition unit 33.

[0158] In addition, the handwriting recognition control unit 26, the string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 each store handwriting recognition string candidates, language string candidates, conversion string candidates, predicted string candidates, operation command candidates, and data related to the selected object so that they can be obtained in subsequent steps S4.8.1.2 to 4.8.1.4.

[0159] S4.8: Immediately after transmitting the stroke addition command to the handwriting recognition control unit 26 in step S4.7, the handwriting input display control unit 23 transmits a selectable candidate display timer start command to the candidate display timer control unit 24. The candidate display timer control unit 24 starts this timer.

[0160] The following steps S5 to S5.2 are executed if a pen-down occurs before a certain time has elapsed (before the timer times out).

[0161] S5: If the user touches the pen to the handwriting input unit 21 before the timer times out, the handwriting input unit 21 transmits a pen-down signal (the same event as in step S2) to the handwriting input display control unit 23.

[0162] S5.1: The handwriting input display control unit 23 transmits a stroke start signal (same as step S2.1) to the handwriting input storage unit 25. The subsequent sequence is the same as that from step S2.1 onwards.

[0163] S5.2: Furthermore, the handwriting input display control unit 23 transmits a selectable candidate display timer stop command to the candidate display timer control unit 24. The candidate display timer control unit 24 stops the timer because the timer is no longer necessary since pen-down has been detected.

[0164] Steps S4.8.1 to S7.14 are executed if the pen does not go down before a certain time has elapsed (before the timer times out), and therefore the character string candidates and operation command candidates shown in FIG.

[0165] 4.8.1: If the user does not bring the pen into contact with the handwriting input unit 21 while the selectable candidate display timer is in progress, the candidate display timer control unit 24 transmits a timeout to the handwriting input display control unit 23 .

[0166] S4.8.1.1: The handwriting input display control unit 23 transmits a request to obtain handwriting recognition / language character string candidates to the handwriting recognition control unit 26. The handwriting recognition control unit 26 transmits the handwriting recognition / language character string candidates currently held to the handwriting input display control unit 23.

[0167] S4.8.1.2: The handwriting input display control unit 23 transmits a request to obtain a conversion character string candidate to the character string conversion control unit 28. The character string conversion control unit 28 transmits the conversion character string candidate currently held to the handwriting input display control unit 23.

[0168] S4.8.1.3: The handwriting input display control unit 23 transmits a predicted character string candidate acquisition request to the predictive conversion control unit 30. The predictive conversion control unit 30 transmits the predicted character string candidates currently held to the handwriting input display control unit 23.

[0169] S4.8.1.4: The handwriting input display control unit 23 transmits the acquired operation command candidates to the operation command recognition control unit 32. The operation command recognition control unit 32 transmits the currently held operation command candidates and the selected object to the handwriting input display control unit 23.

[0170] S4.8.1.5: Furthermore, the handwriting input display control unit 23 transmits the estimated writing direction to the handwriting input storage unit 25. The handwriting input storage unit 25 determines the estimated writing direction from the stroke addition time, horizontal distance, and vertical distance of the rectangular area of ​​the handwritten object, and transmits the estimated writing direction to the handwriting input display control unit 23.

[0171] S4.8.1.6: The handwriting input display control unit 23 creates selectable candidate display data as shown in Figure 14 from these handwriting recognition string candidates ('gi' in Figure 14), language string candidates (for example, 'gi', although not displayed in Figure 14), converted string candidates ('minutes', 'skill test' in Figure 14), predicted string candidates ('Settle the skill test', 'Address to send the minutes' in Figure 14), operation command candidates ('Load minutes template', 'Save in minutes folder' in Figure 14), each selection probability, and estimated writing direction, and displays the selectable candidate display data as shown in Figure 14, consisting of string candidates and operation command candidates, by sending it to the display unit 22.

[0172] S4.8.1.7: The handwriting input display control unit 23 also transmits rectangular area display data (rectangular frames) of the handwritten object and the selected object (handwritten object rectangular area display 503 in FIG. 14) to the display unit 22.

[0173] S4.8.1.8: The handwriting input display control unit 23 transmits a selectable candidate display deletion timer start signal to the candidate display timer control unit 24 to delete the selectable candidate display data after a certain time has elapsed since its display. The candidate display timer control unit 24 starts this timer.

[0174] Steps S6 to S6.3 are executed when the user erases the selectable candidate display displayed on the display unit 22 while the selectable candidate deletion timer is running, or when a change occurs in the handwritten object (i.e., when a stroke of the handwritten object is added, deleted, moved, deformed, or divided), or when no candidate is selected before the timer times out.

[0175] Furthermore, steps S6 to S6.1 are executed when the candidate display is erased or a change in the handwritten object occurs.

[0176] S6: The handwriting input unit 21 transmits to the handwriting input display control unit 23 a notice that the selectable candidate display has been deleted or that the handwritten object has been changed.

[0177] S6.1: The handwriting input display control unit 23 sends a command to stop the selectable candidate deletion timer. The candidate display timer control unit 24 stops the timer. This is because an operation has been performed on the handwritten object within a certain period of time, making the timer unnecessary.

[0178] S6.2: The handwriting input display control unit 23 sends a command to erase the selectable candidate display data to the display unit 22, thereby erasing the display.

[0179] S6.3: The handwriting input display control unit 23 erases the display by transmitting a command to erase the rectangular area display data of the handwritten object and the selected object to the display unit 22. Therefore, when the display of the operation command candidate is erased under conditions other than the selection of the operation command candidate, the handwritten object remains displayed as is.

[0180] 4.8.1.8.1: On the other hand, if the selectable candidate display is not deleted or the handwritten object is not changed while the selectable candidate deletion timer is started (if the user does not perform a pen operation), the candidate display timer control unit 24 sends 4.8.1.8.1: Timeout to the handwriting input display control unit 23.

[0181] Similarly, after the selectable candidate display deletion timer times out, the handwriting input display control unit 23 executes steps S6.2 and S6.3. This is because the selectable candidate display data and the rectangular area display data of the handwritten object and the selected object may be deleted after a certain period of time has elapsed.

[0182] If the user selects a selectable candidate while the selectable candidate deletion timer is running, steps S7 to S7.14 are executed.

[0183] S7: If the user selects a selectable candidate while the selectable candidate deletion timer is running, the handwriting input unit 21 transmits the character string candidate or operation command candidate selection to the handwriting input display control unit 23.

[0184] S7.1: The handwriting input display control unit 23 transmits a selectable candidate display deletion timer stop command to the candidate display timer control unit 24. The candidate display timer control unit 24 stops this timer.

[0185] S7.2: Next, the handwriting input / display control unit 23 sends a command to clear the stored data to the handwriting recognition control unit 26.

[0186] S7.2.1: The handwriting recognition control unit 26 sends a retained data clear command to the character string conversion control unit 28.

[0187] S7.2.2: The handwriting recognition control unit 26 sends a command to clear the stored data to the predictive conversion control unit 30.

[0188] S7.2.3: The handwriting recognition control unit 26 sends a command to clear the stored data to the operation command recognition control unit 32.

[0189] The handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 clear the data relating to character string candidates and operation command candidates that have been held up to that point.

[0190] S7.3: Next, the handwriting input display control unit 23 transmits a command to erase the selectable candidate display data to the display unit 22, thereby erasing the display.

[0191] S7.4: The handwriting input display control unit 23 sends a command to erase the rectangular area display data of the handwritten object and the selected object to the display unit 22, thereby erasing the display.

[0192] S7.5: The handwriting input display control unit 23 erases the display by transmitting to the display unit 22 a command to erase the handwritten object display data and the pen coordinate complement display data transmitted in step S3.1. This is because the handwritten object, etc., is no longer necessary since a character string candidate or an operation command candidate has been selected.

[0193] S7.6: The handwriting input display control unit 23 transmits a request to delete the handwritten object to the handwriting input storage unit 25.

[0194] If a character string candidate is selected, steps S7.7 to S7.9 are executed.

[0195] S7.7: If a character string candidate is selected, the handwriting input display control unit 23 transmits a character string object addition to the handwriting input saving unit 25.

[0196] S7.8: Furthermore, the handwriting input display control unit 23 transmits a character string object font acquisition command to the handwriting input saving unit 25. The handwriting input saving unit 25 selects a predefined font based on the estimated character size of the handwritten object and transmits the selected font to the handwriting input display control unit 23.

[0197] S7.9: Next, the handwriting input display control unit 23 uses the defined font received from the handwriting input storage unit 25 to send character string object display data to the display unit 22 to be displayed in the same position as the handwritten object, thereby displaying the character string object.

[0198] If a candidate operation command is selected, steps S7.10 to S7.13 are executed, and if there is a selected object, steps S7.10 to S7.12 are executed.

[0199] S7.10: When a candidate for an operation command for a selected object is selected (when a selected object exists), the handwriting input display control unit 23 erases the display by sending a command to erase the display data of the selected object to the display unit 22. This is to erase the original selected object.

[0200] S7.11: Next, the handwriting input display control unit 23 transmits an operation command for executing the selected object to the handwriting input saving unit 25. The handwriting input saving unit 25 transmits display data of the newly selected object (display data after editing or modification) to the handwriting input display control unit 23.

[0201] S7.12: Next, the handwriting input display control unit 23 transmits the selected object display data to the display unit 22, thereby re-displaying the selected object after the operation command is executed.

[0202] If there is no selected object (if an input / output operation command is selected), step S7.13 is executed.

[0203] S7.13: When an input / output operation command is selected, the handwriting input display control unit 23 executes the operation command character string (Command) of the operation command definition data corresponding to the operation command selected by the user. Note that, when the user authentication unit 34 succeeds in authenticating the user, the handwriting input display control unit 23 sets information about the successfully authenticated user in % to % of the operation command and executes it.

[0204] S7.14: For the next handwritten object, the handwritten input display control unit 23 sends a handwritten object start signal to the handwritten input saving unit 25. The handwritten input saving unit 25 reserves a handwritten object area. After this, the processes of steps S2 to S7.14 are repeated.

[0205] <Summary> As described above, the handwriting input device 2 of this embodiment can display candidates for any operation command by the user writing a handwritten object by hand or encircling a fixed object with a surrounding line. Therefore, any function (edit function, input / output function, pen function, etc.) can be called from the handwritten state. This eliminates the need for step-by-step operations of pressing a menu button to call a desired function, and reduces the number of operation steps required from the user writing to calling a desired function. [Example]

[0206] In this embodiment, a handwriting input system in a system form in which an information processing system on a network performs processing such as handwriting recognition and returns the processing result to a handwriting input device 2 will be described.

[0207] In the description of this embodiment, components or contents of figures with the same symbols as in embodiment 1 perform the same functions, so the description of components that have already been described may be omitted or only the differences may be described.

[0208] 24 is an example of a system configuration diagram of a handwriting input system 100. The handwriting input system 100 has a handwriting input device 2 and an information processing system 10 that can communicate with each other via a network N.

[0209] The handwriting input device 2 is installed in a facility such as a company's office, and is connected to a LAN or Wi-Fi installed within the facility. The information processing system 10 is installed in, for example, a data center. The handwriting input device 2 is connected to the Internet i via a firewall 8, and the information processing system 10 is also connected to the Internet i via a high-speed LAN or the like within the data center.

[0210] The handwriting input device 2 may be connected to the Internet i using wireless communication such as a telephone network. In this case, the wireless communication may be 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), or the like.

[0211] The information processing system 10 has one or more information processing devices, and the one or more information processing devices act as servers to provide services to the handwriting input device 2. A server is a computer or software that performs the function of providing information and processing results in response to requests from clients. As will be described later, the information processing system 10 receives pen coordinates from the handwriting input device 2 and transmits information required to display the operation guide 500 shown in Fig. 14 to the handwriting input device 2.

[0212] The server-side system is sometimes called a cloud system. A cloud system is a system that uses cloud computing, which is a form of usage in which resources on a network are used without being aware of specific hardware resources. Cloud systems are not necessarily deployed on the Internet. In Figure 24, the information processing system is deployed on the Internet, but it may also be deployed on a local network (called on-premise in this case).

[0213] Moreover, in some embodiments, information processing system 10 includes multiple computing devices, such as a server cluster, configured to communicate with each other over any type of communications link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0214] The handwriting input device 2 may have the same configuration as in the first embodiment, but in this embodiment, it is sufficient to have a touch panel, a display, and a communication function. The handwriting input device 2 may include multiple computing devices configured to communicate with each other.

[0215] In this embodiment, a general information processing device such as a PC or tablet can execute a web browser or a dedicated application. The web browser or the dedicated application communicates with the information processing system 10. When the web browser is running, the user inputs or selects the URL of the information processing system 10 to connect the handwriting input device to the information processing system 10. The handwriting input device 2 executes a web application provided by the information processing system 10 in the web browser. The web application refers to software or its mechanism that operates through cooperation between a program written in a programming language (for example, JavaScript (registered trademark)) that runs on the web browser and a program on the web server side.

[0216] When the dedicated application runs, it connects to a pre-registered URL of the information processing system 10. Since the dedicated application has a program and a user interface, the program sends and receives necessary information to and from the information processing system 10 and displays it on the user interface.

[0217] The communication method may be a general-purpose communication protocol such as HTTP, HTTPs, or WebSocket, or a dedicated communication protocol.

[0218] <Hardware configuration example> The hardware configuration of the handwriting input device 2 may be the same as that shown in Fig. 5. In this embodiment, an example of the hardware configuration of the information processing system 10 will be described.

[0219] Fig. 25 is a hardware configuration diagram of the information processing system 10. As shown in Fig. 25, the information processing system 10 is constructed by a computer, and includes a CPU 601, a ROM 602, a RAM 603, an HD 604, an HDD (Hard Disk Drive) controller 605, a display 606, an external device connection I / F (Interface) 608, a network I / F 609, a bus line 610, a keyboard 611, a pointing device 612, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 616.

[0220] Of these, the CPU 601 controls the overall operation of the information processing system 10. The ROM 602 stores programs, such as an IPL, used to drive the CPU 601. The RAM 603 is used as a work area for the CPU 601. The HD 604 stores various data, such as programs. The HDD controller 605 controls the reading and writing of various data from and to the HD 604 under the control of the CPU 601. The display 606 displays various information, such as a cursor, menu, window, text, or image. The external device connection I / F 608 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 609 is an interface for data communication using a communication network. The bus line 610 is an address bus, a data bus, or the like for electrically connecting the components, such as the CPU 601, shown in FIG. 25.

[0221] The keyboard 611 has a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 612 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 614 controls reading and writing of various data from and to a DVD-RW 613, which is an example of a removable recording medium. Note that this is not limited to DVD-RW, and may be DVD-R, etc. The media I / F 616 controls reading and writing (storing) of data from and to a recording medium 615, such as a flash memory.

[0222] <About the device functions> Next, functions of the handwriting input system 100 will be described with reference to Fig. 26. Fig. 26 is an example of a functional block diagram showing functions of the handwriting input system 100 in blocks. Note that the description of Fig. 26 will mainly focus on the differences from Fig. 6.

[0223] In this embodiment, the handwriting input device 2 has a display unit 22, a display control unit 41, a handwriting input unit 21, and a communication unit 42. Each function of the handwriting input device 2 is a function or means realized by operating any of the components shown in Fig. 5 in response to an instruction from the CPU 201 in accordance with a program loaded from the SSD 204 onto the RAM 203.

[0224] The function of the handwriting input unit 21 in this embodiment may be the same as in embodiment 1. The handwriting input unit 21 converts the user's pen input d1 into pen operation data da (pen up, pen down, or pen coordinate data) and transmits it to the display control unit 41.

[0225] The display control unit 41 controls the display of the handwriting input device 2. First, the display control unit 41 performs interpolation calculations on the coordinates between the discrete values ​​of the pen coordinate data, which are discrete values, and converts the pen coordinate data from pen-down to pen-up into one stroke db, and transmits it to the display unit 22.

[0226] Furthermore, the display control unit 41 transmits the pen operation data dc to the communication unit 42 and acquires various display data dd from the communication unit 42. The display data includes information for displaying the operation guide 500 in FIG. 14. The display control unit 41 transmits the display data de to the display unit 22.

[0227] The communication unit 42 transmits the pen operation data dc to the information processing system 10, receives various display data dd from the information processing system 10, and transmits the data to the display control unit 41.

[0228] The function of the display unit 22 may be the same as in Example 1. The display unit 22 displays the stroke db and the display data de. The display unit 22 converts the stroke db and the display data de written to the video memory by the display control unit 41 into data according to the characteristics of the display 220 and transmits them to the display 220.

[0229] <<Functions of the information processing system>> The information processing system 10 includes a communication unit 43, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a character string conversion control unit 28, a character string conversion dictionary unit 29, a predictive conversion control unit 30, a predictive conversion dictionary unit 31, an operation command recognition control unit 32, and an operation command definition unit 33. Each function of the information processing system 10 is realized by any of the components shown in Fig. 25 operating in response to an instruction from the CPU 601 in accordance with a program loaded from the HD 604 onto the RAM 603.

[0230] The communication unit 43 receives pen operation data dc from the handwriting input device 2, transmits pen operation data df to the handwriting input display control unit 23, receives display data dd from the handwriting input display control unit 23 and transmits it to the handwriting input device 2.

[0231] Other functions are the same as those in the first embodiment, or even if they are different, it will not hinder the explanation of this embodiment.

[0232] Furthermore, with regard to the function of authenticating a user, the information processing system 10 performs authentication using a general method.

[0233] <Operation procedure> The operation of the handwriting input system 100 will be described using the above configuration and Figs. 27 to 33. Figs. 27 to 33 are sequence diagrams illustrating an example of processing in which the handwriting input device 2 displays character string candidates and operation command candidates. The processing in Fig. 27 starts when the handwriting input device 2 is started (a web browser or a dedicated application is started) and communication with the information processing system 10 is established. The overall flow in Figs. 27 to 33 may be the same as that in Figs. 18 to 23.

[0234] S1: When communication is established, the handwriting input display control unit 23 transmits a handwriting object start to the handwriting input storage unit 25 in order to reserve a memory area of ​​the handwriting input device 2. The handwriting input storage unit 25 reserves a handwriting object area (a memory area for storing handwritten objects). The handwriting object area may be reserved after the user brings the pen into contact with the handwriting input unit 21.

[0235] S2a: Next, the user touches the pen to the handwriting input unit 21. The handwriting input unit 21 detects the pen down and transmits it to the display control unit 41.

[0236] S2b: The display control unit 41 transmits a pen-down to the communication unit 42 to notify the information processing system 10 of the pen-down.

[0237] S2c: The communication unit 42 transmits a pen-down signal to the information processing system 10.

[0238] S2d: The communication unit 43 of the information processing system 10 receives the pen-down signal and transmits it to the handwriting input display control unit 23.

[0239] S2.1: The handwriting input display control unit 23 transmits a stroke start signal to the handwriting input storage unit 25, and the handwriting input storage unit 25 reserves a stroke area.

[0240] S3a: When the user moves the pen while keeping it in contact with the handwriting input unit 21, the handwriting input unit 21 transmits the pen coordinates to the display control unit 41.

[0241] S3b: The display control unit 41 transmits the pen coordinates to the communication unit 42 to notify the information processing system 10 of the pen coordinates.

[0242] S3c: The communication unit 42 transmits the pen coordinates to the information processing system 10.

[0243] S3d: The communication unit 43 of the information processing system 10 receives the pen coordinates and transmits them to the handwriting input display control unit 23.

[0244] S3.1: The display control unit 41 transmits pen coordinate complement display data (data for interpolating discrete pen coordinates) to the display unit 22. The display unit 22 uses the pen coordinate complement display data to interpolate the pen coordinates and display a line. The processing in step S3.2 is the same as in the first embodiment.

[0245] S4a: When the user removes the pen from the handwriting input unit 21, the handwriting input unit 21 transmits a pen-up message to the display control unit 41.

[0246] S4b: The display control unit 41 transmits a pen-up event to the communication unit 42 to notify the information processing system 10 of the pen-up event.

[0247] S4c: The communication unit 42 transmits a pen-up signal to the information processing system 10.

[0248] S4d: The communication unit 43 of the information processing system 10 receives the pen-up signal and transmits it to the handwriting input display control unit 23.

[0249] The subsequent steps S4.1 to S4.3.3 and steps S4.6 to S4.7.6.2 in FIG. 29 are the same as those in the first embodiment.

[0250] S5a: If the user touches the pen to the handwriting input unit 21 before the timer times out, the handwriting input unit 21 sends a pen-down (the same event as in step S2) to the display control unit 41. The processes in steps S5b to S5d may be the same as those in steps S2b to S2d. The processes in steps S5.1 to S4.8.1.5 are the same as those in the first embodiment.

[0251] S4.8.1.6a: The handwriting input display control unit 23 creates selectable candidate display data including each character string candidate, operation command candidate, each selection probability, and estimated writing direction shown in Figure 14, and transmits the selectable candidate display data consisting of the character string candidates and operation command candidates to the communication unit 43.

[0252] S4.8.1.6b: The communication unit 43 transmits the selectable candidate display data to the handwriting input device 2.

[0253] S4.8.1.6c: The communication unit 42 of the handwriting input device 2 receives the selectable candidate display data and transmits it to the display control unit 41.

[0254] S4.8.1.6d: The display control unit 41 has received the selectable candidate display data, and transmits it to the display unit 22 to display it.

[0255] S4.8.1.7a: The handwriting input display control unit 23 also transmits rectangular area display data (rectangular frames) of the handwritten object and the selected object (handwritten object rectangular area display 503 in FIG. 14) to the communication unit 43.

[0256] S4.8.1.7b: The communication unit 43 transmits the rectangular area display data to the handwriting input device 2.

[0257] S4.8.1.7c: The communication unit 42 of the handwriting input device 2 receives the rectangular area display data and transmits it to the display control unit 41.

[0258] S4.8.1.7d: The display control unit 41 has received the rectangular area display data, and transmits it to the display unit 22 to display it. The process of step S4.8.1.8 is the same as in the first embodiment.

[0259] S6a: When the user erases a selectable candidate or writes an additional character by hand on the handwritten object, the handwriting input unit 21 notifies the display control unit 41 that the selectable candidate display has been erased or the handwritten object has been changed.

[0260] S6b: The display control unit 41 transmits to the communication unit 42 to notify the information processing system 10 of the occurrence of deletion of the selectable candidate display or change in the handwritten object.

[0261] S6c: The communication unit 42 transmits to the information processing system 10 the occurrence of the deletion of the selectable candidate display or the change of the handwritten object.

[0262] S6d: The communication unit 43 of the information processing system 10 receives the notification of deletion of the selectable candidate display or change in the handwritten object, and transmits the notification to the handwritten input display control unit 23. The processes of steps S6.1 and S4.8.1.8.1 are the same as those in the first embodiment.

[0263] S6.2a: The handwriting input display control unit 23 transmits a command to the communication unit 43 to erase the selectable candidate display data.

[0264] S6.2b: The communication unit 43 transmits a command to the handwriting input device 2 to erase the selectable candidate display data.

[0265] S6.2c: The communication unit 42 of the handwriting input device 2 receives the command to delete the selectable candidate display data, and transmits it to the display control unit 41.

[0266] S6.2d: The display control unit 41 has received the command to erase the selectable candidate display data, and transmits the command to the display unit 22 to erase the selectable candidates.

[0267] S6.3a: The handwriting input display control unit 23 transmits to the communication unit 43 a command to erase the rectangular area display data of the handwritten object and the selected object.

[0268] S6.3b: The communication unit 43 transmits to the handwriting input device 2 a command to erase the rectangular area display data of the handwritten object and the selected object.

[0269] S6.3c: The communication unit 42 of the handwriting input device 2 receives the rectangular area display data deletion of the handwritten object and the selected object, and transmits it to the display control unit 41.

[0270] S6.3d: The display control unit 41 has received the command to erase the rectangular area display data of the handwritten object and the selected object, and erases the rectangular areas of the handwritten object and the selected object by transmitting the command to the display unit 22. Therefore, when the display of the operation command candidate is erased under conditions other than the selection of the operation command candidate, the handwritten object remains displayed as is.

[0271] If the user selects a selectable candidate while the selectable candidate deletion timer is running, steps S7 to S7.14 are executed.

[0272] S7a: If the user selects a selectable candidate while the selectable candidate deletion timer is starting, the handwriting input unit 21 transmits the character string candidate or the operation command candidate selection to the display control unit 41.

[0273] S7b: The display control unit 41 transmits the character string candidate or the operation command candidate selection to the communication unit 42 to notify the information processing system 10.

[0274] S7c: The communication unit 42 transmits the character string candidate or the selected operation command candidate to the information processing system 10.

[0275] S7d: The communication unit 43 of the information processing system 10 receives the character string candidate or the selected operation command candidate, and transmits it to the handwriting input display control unit 23. The processes of steps S7.1 to S7.2.3 are the same as those in the first embodiment.

[0276] S7.3a: Next, the handwriting input display control unit 23 transmits a command to the communication unit 43 to erase the selectable candidate display data.

[0277] S7.3b: The communication unit 43 transmits a command to the handwriting input device 2 to erase the selectable candidate display data.

[0278] S7.3c: The communication unit 42 of the handwriting input device 2 receives the command to delete the selectable candidate display data, and transmits it to the display control unit 41.

[0279] S7.3d: The display control unit 41 has received the command to erase the selectable candidate display data, and thus causes the display unit 22 to erase the selectable candidates.

[0280] S7.4a: The handwriting input display control unit 23 transmits to the communication unit 43 a command to erase the rectangular area display data of the handwritten object and the selected object.

[0281] S7.4b: The communication unit 43 transmits a command to erase the rectangular area display data to the handwriting input device 2.

[0282] S7.4c: The communication unit 42 of the handwriting input device 2 receives the rectangular area display data deletion command and transmits it to the display control unit 41.

[0283] S7.4d: The display control unit 41 has received the command to erase the rectangular area display data, and therefore causes the display unit 22 to erase the rectangular area.

[0284] S7.5a: The handwriting input display control unit 23 transmits a command to the communication unit 43 to erase the handwritten object display data.

[0285] S7.5b: The communication unit 43 transmits a command to erase the handwritten object display data to the handwriting input device 2.

[0286] S7.5c: The communication unit 42 of the handwriting input device 2 receives the command to erase the handwritten object display data, and transmits it to the display control unit 41.

[0287] S7.5d: Having received the command to erase the handwritten object display data, the display control unit 41 erases the handwritten object and the pen coordinate complement display data on the display unit 22. The process of step S7.6 may be the same as in the first embodiment.

[0288] When a character string candidate is selected, steps S7.7 to S7.9 are executed. The processes in steps S7.7 and S7.8 may be the same as those in the first embodiment.

[0289] S7.9a: Next, the handwriting input display control unit 23 uses the predefined font received from the handwriting input storage unit 25 to transmit to the communication unit 43 character string object display data to be displayed in the same position as the handwritten object.

[0290] S7.9b: The communication unit 43 transmits the character string object display data to the handwriting input device 2.

[0291] S7.9c: The communication unit 42 of the handwriting input device 2 receives the character string object display data and transmits it to the display control unit 41.

[0292] S7.9d: The display control unit 41 has received the character string object display data, and causes the display unit 22 to display the character string object.

[0293] If a candidate operation command is selected, steps S7.10 to S7.13 are executed, and if there is a selected object, steps S7.10 to S7.12 are executed.

[0294] S7.10a: When a candidate for an operation command for a selected object is selected (when a selected object exists), the handwriting input display control unit 23 transmits a command to erase the display data of the selected object to the communication unit 43. This is to erase the original selected object.

[0295] S7.10b: The communication unit 43 transmits a command to erase the display data of the selected object to the handwriting input device 2.

[0296] S7.10c: The communication unit 42 of the handwriting input device 2 receives the command to erase the selected object display data, and transmits it to the display control unit 41.

[0297] S7.10d: The display control unit 41 has received the command to erase the selected object display data, and therefore causes the display unit 22 to erase the selected object.

[0298] S7.11: Next, the handwriting input display control unit 23 transmits an operation command for executing the selected object to the handwriting input saving unit 25. The handwriting input saving unit 25 transmits display data of the newly selected object (display data after editing or modification) to the handwriting input display control unit 23.

[0299] S7.12a: Next, the handwriting input display control unit 23 transmits the selected object display data to the communication unit 43.

[0300] S7.12b: The communication unit 43 transmits the selected object display data to the handwriting input device 2.

[0301] S7.12c: The communication unit 42 of the handwriting input device 2 receives the selected object display data and transmits it to the display control unit 41.

[0302] S7.12d: Having received the selected object display data, the display control unit 41 causes the display unit 22 to re-display the selected object after the operation command is executed. The processes of steps S7.13 and S7.14 may be the same as those in the first embodiment.

[0303] In this way, even in a system configuration in which the handwriting input device 2 and the information processing system 10 communicate with each other, it is possible to achieve the same effects as in the first embodiment. Note that the process flows in Figs. 27 to 33 are merely examples, and processes resulting from communication between the handwriting input device 2 and the information processing system 10 may be included, or processes may be omitted. Furthermore, the handwriting input device 2 may perform part of the processes performed by the information processing system 10. For example, the handwriting input device 2 may perform processes related to deletion.

[0304] <Example of operation guide display> The operation guide 500 for the handwriting recognition system may be similar to that shown in Fig. 14. Naturally, the design can be changed. In this embodiment, an example of an operation guide displayed when English is handwritten will be described.

[0305] FIG. 34 shows an example of an operation guide 500. When the information processing system 10 recognizes a foreign language such as English, it is desirable that it can also recognize the type of language. However, in order to improve the recognition accuracy, there is a method for identifying the type of language handwritten by the user. For example, the user selects the type of language from a menu displayed by the handwriting input device 2 and notifies the information processing system 10. Alternatively, the user may register user attributes including the locale (language used) in the information processing system 10, and the type of language may be identified when the user logs in. Alternatively, if the information processing system 10 is on the Internet, the country or region may be identified based on the IP address of the handwriting input device 2, and the information processing system 10 may determine the language used in the country or region.

[0306] (1) In Figure 34, for example, the user handwrote "pl (pee-er)."

[0307] (2) The handwriting recognition control unit 26, handwriting recognition dictionary unit 27, character string conversion control unit 28, character string conversion dictionary unit 29, predictive conversion control unit 30, and predictive conversion dictionary unit 31 are provided for English, and the handwriting recognition control unit 26 recognizes "pl" as an English word. The information processing system 10 recognizes the character code "pl" and transmits converted character string candidates 507 that include "pl" as part of the character string and predicted character string candidates 508 that include the converted character string to the handwriting input device 2. This enables the handwriting input device 2 to display words and phrases (selectable candidates) that start with "pl," such as "play," "place," "placard," "placenta," and "play ground." Note that, although omitted in FIG. 34, operation commands may also be displayed.

[0308] (3) When the user selects "play," the operation guide 500 and the handwritten object 504 disappear, and "play" is displayed.

[0309] <Summary> Therefore, in this embodiment, handwriting recognition can be performed in the same manner as in the first embodiment in the handwriting input system 100 in which the handwriting input device 2 and the information processing system 10 communicate with each other.

[0310] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0311] For example, although an electronic whiteboard has been described as an example in the present embodiment, the present invention can be suitably applied to any information processing device having a touch panel. Furthermore, devices having similar functions to an electronic whiteboard are also referred to as electronic whiteboards, electronic information boards, interactive boards, etc. Examples of information processing devices equipped with a touch panel include output devices such as PJs (Projectors), digital signage, HUDs (Head Up Display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, network appliances, notebook PCs (Personal Computers), mobile phones, smartphones, tablet devices, game consoles, PDAs (Personal Digital Assistants), digital cameras, wearable PCs, and desktop PCs.

[0312] In this embodiment, the coordinates of the pen tip are detected by detecting the coordinates of the pen tip on a touch panel, but the coordinates of the pen tip may also be detected by ultrasonic waves. The pen emits light and also emits ultrasonic waves, and the handwriting input device 2 calculates the distance based on the arrival time of the ultrasonic waves. The position of the pen can be identified based on the direction and distance. The projector draws (projects) the trajectory of the pen as a stroke.

[0313] In this embodiment, when a selected object is present, candidate edit and modification operation commands are displayed, and when no selected object is present, candidate input / output operation commands are displayed. However, candidate edit and modification operation commands and candidate input / output operation commands may be displayed simultaneously.

[0314] In addition, the configuration examples in Figures 6, 26, etc. are divided according to main functions to make it easier to understand the processing by the handwriting input device 2. The method of dividing the processing units and the names of the processing units do not limit the present invention. The processing of the handwriting input device 2 can be divided into more processing units depending on the processing content. Also, it can be divided so that one processing unit includes more processes.

[0315] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions. [Explanation of symbols]

[0316] 2. Handwriting input device 21 Handwriting input section 22 Display section 23 Handwriting input display control unit 24 Candidate display timer control section 25 Handwriting input storage section 26 Handwriting recognition control unit 27 Handwriting recognition dictionary section 28 String conversion control section 29 String Conversion Dictionary 30 Predictive conversion control unit 31 Predictive conversion dictionary section 32 Operation command recognition control section 33 Operation command definition section 34 User authentication section 35 Authentication information acquisition unit [Prior art documents] [Patent documents]

[0317] [Patent Document 1] Japanese Patent Application Publication No. 2018-026185

Claims

1. An information processing device capable of communicating with an input device via a network, a communication unit that receives, from the input device, data relating to handwritten input accepted by the input device, the data relating to the handwritten input being displayed on a display by the input device as a handwritten object; a handwriting recognition control unit that recognizes the data related to the handwriting input received by the communication unit as a character string candidate; an operation command recognition control unit that acquires operation command candidates corresponding to character strings that partially match the character string candidates recognized by the handwriting recognition control unit, The communication unit When no change in the handwritten object displayed on the display occurs for a first period of time, the input device selectably displays the character string candidates and the operation command candidates on the display, transmitting the character string candidate recognized by the handwriting recognition control unit and the operation command candidate corresponding to a character string that partially matches the character string candidate recognized by the handwriting recognition control unit to the input device; when the operation command candidate is not selected on the input device for a second period of time, when an operation to erase the display of the operation command candidate is received, or when a change occurs in the handwritten object, erasing the displayed operation command candidate; When the display of the operation command candidate is erased under a condition other than the condition in which the operation command candidate is selected, in order to maintain the display of the handwritten object, The information processing apparatus is characterized in that it transmits information for erasing the display of the operation command candidates to the input device.

2. The communication unit transmitting the character string candidates and the operation command candidates to the input device when no change in the handwritten object displayed on the display occurs for the first period of time; 2. The information processing device according to claim 1.

3. when the operation command candidates and the character string candidates are not selected on the input device for the second time period, or when an operation to erase the display of the operation command candidates and the character string candidates is received, erasing the display of the operation command candidates and the character string candidates; The information processing device according to claim 1, characterized in that when the display of the operation command candidate and the character string candidate is erased under conditions other than the selection of the operation command candidate and the character string candidate, information for erasing the display of the operation command candidate and the character string candidate is sent to the input device in order to maintain the display of the handwritten object.

4. the character string candidates are letters, numbers, symbols, or graphics, 4. The information processing apparatus according to claim 3, wherein the communication unit transmits the operation command candidates to the input device together with letters, numbers, symbols, or graphics.

5. storing a constant to replace a variable associated with the operation command; replacing a variable associated with the operation command corresponding to a character string that partially matches the character string candidate recognized by the handwriting recognition control unit with the constant; 5. The information processing apparatus according to claim 1, wherein the operation command is executed with the variables replaced with the constants.

6. having a user authentication means for authenticating a user, If the user authentication means is successful in authentication, information about the successfully authenticated user is set to the constant; replacing a variable associated with the operation command corresponding to a character string that partially matches the character string candidate recognized by the handwriting recognition control unit with information about the user; 6. The information processing apparatus according to claim 5, wherein the operation command is executed with the variables replaced with information about the user.

7. The information processing device according to claim 2, characterized in that, when the position information of a confirmed object that has already been handwritten has a predetermined relationship with the handwritten object, the operation command recognition control unit recognizes the operation command related to the confirmed object that has a predetermined relationship with the handwritten object.

8. 8. The information processing apparatus according to claim 7, wherein the operation command recognition control unit recognizes the operation command for editing or modifying the fixed object that has a predetermined relationship with the handwritten object.

9. When the position information of each of a plurality of determined objects that have already been handwritten has a predetermined relationship with one of the handwritten objects, 9. The information processing apparatus according to claim 7, wherein the operation command recognition control unit recognizes the operation commands relating to a plurality of the determined objects that have a predetermined relationship with one of the handwritten objects.

10. The information processing device according to any one of claims 7 to 9, characterized in that, when an already handwritten confirmed object and the handwritten object overlap with each other at an overlap rate equal to or greater than a predetermined rate, the operation command recognition control unit determines that the position information of the already handwritten confirmed object and the handwritten object are in the predetermined relationship.

11. 2. The information processing device according to claim 1, wherein the condition other than that for selecting the operation command candidate is when an operation to erase the display of the operation command candidate is received or when a change occurs in the handwritten object.

12. A display method performed by an information processing device capable of communicating with an input device via a network, a communication process for receiving, from the input device, data relating to handwritten input accepted by the input device, the data relating to the handwritten input being displayed on a display by the input device as a handwritten object; a handwriting recognition control process for recognizing the data related to the handwriting input received through the communication process as a character string candidate; an operation command recognition control process for acquiring operation command candidates corresponding to a character string that partially matches the character string candidate recognized by the handwriting recognition control process; The communication process includes: When no change in the handwritten object displayed on the display occurs for a first period of time, the input device selectably displays the character string candidates and the operation command candidates on the display, transmitting the character string candidate recognized by the handwriting recognition control process and the operation command candidate corresponding to a character string that partially matches the character string candidate recognized by the handwriting recognition control process to the input device; when the operation command candidate is not selected on the input device for a second period of time, when an operation to erase the display of the operation command candidate is received, or when a change occurs in the handwritten object, erasing the displayed operation command candidate; When the display of the operation command candidate is erased under a condition other than the condition in which the operation command candidate is selected, in order to maintain the display of the handwritten object, a display method comprising transmitting information to the input device to erase the display of the operation command candidates;

13. An information processing device capable of communicating with an input device via a network, a communication unit that receives, from the input device, data relating to handwritten input accepted by the input device, the data relating to the handwritten input being displayed on a display by the input device as a handwritten object; a handwriting recognition control unit that recognizes the data related to the handwriting input received by the communication unit as a character string candidate; and causing the handwriting recognition control unit to function as an operation command recognition control unit that acquires operation command candidates corresponding to character strings that partially match the character string candidates recognized by the handwriting recognition control unit, The communication unit When no change in the handwritten object displayed on the display occurs for a first period of time, the input device selectably displays the character string candidates and the operation command candidates on the display, transmitting the character string candidate recognized by the handwriting recognition control unit and the operation command candidate corresponding to a character string that partially matches the character string candidate recognized by the handwriting recognition control unit to the input device; when the operation command candidate is not selected on the input device for a second period of time, when an operation to erase the display of the operation command candidate is received, or when a change occurs in the handwritten object, erasing the displayed operation command candidate; When the display of the operation command candidate is erased under a condition other than the condition in which the operation command candidate is selected, in order to maintain the display of the handwritten object, a program for transmitting information for erasing the display of the operation command candidates to the input device;

14. An input system in which an input device and an information processing device can communicate with each other via a network, The input device is a handwriting input unit that accepts handwriting input; a first communication unit that transmits data related to the handwriting input accepted by the handwriting input unit to the information processing device; a display control unit that displays the data related to the handwritten input received by the handwritten input unit on a display as a handwritten object, The information processing device includes: a second communication unit that receives the data related to the handwriting input transmitted from the first communication unit; a handwriting recognition control unit that recognizes the data related to the handwriting input received by the second communication unit as a character string candidate; an operation command recognition control unit that acquires operation command candidates corresponding to character strings that partially match the character string candidates recognized by the handwriting recognition control unit, The second communication unit When no change in the handwritten object displayed on the display occurs for a first period of time, the input device selectably displays the character string candidates and the operation command candidates on the display, transmitting the character string candidate recognized by the handwriting recognition control unit and the operation command candidate corresponding to a character string that partially matches the character string candidate recognized by the handwriting recognition control unit to the input device; when the operation command candidate is not selected on the input device for a second period of time, when an operation to erase the display of the operation command candidate is received, or when a change occurs in the handwritten object, erasing the displayed operation command candidate; When the display of the operation command candidate is erased under a condition other than the condition in which the operation command candidate is selected, in order to maintain the display of the handwritten object, An input system comprising: a display unit configured to display a list of operation command candidates; a display unit configured to display the list of operation command candidates;

Images