Information processing apparatus, display method, program, input system

Abstract

To provide an input device that has improved operability of handwriting input.SOLUTION: There is provided an input device 2 that comprises: a handwriting input unit that receives a handwriting input by using the position of a pen or a user's finger in contact with a display; and a display unit that displays the handwriting input received by the handwriting input unit on the display as a handwriting object. When no change occurs in the handwriting object for a first time, the input device 2 displays one or more operation commands based on the handwriting object.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an input device, an input method, a program, and an input system. [Background technology]

[0002] In a typical computer-controlled whiteboard device or application that allows handwritten input (hereinafter referred to as an input device), the input means is limited to a pen or a finger. For this reason, an operation menu is provided so that the user can switch between a pen function for changing the color of characters, etc., and an editing function for erasing characters, etc. Usually, the pen function menu allows the selection of color, thickness, etc., and the editing function menu allows the selection of erasure, movement, magnification, rotation, cut, copy, paste, etc. (see, for example, Patent Document 1).

[0003] Patent Document 1 discloses an input device that displays a menu for setting a color, transparency, thickness, line type, stamp, and operation when a pen button is pressed. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional input devices have a problem in that the operability of handwriting input is sometimes poor. For example, every time a user wants to edit or modify a handwritten object, the user must open a pen function menu or an edit function menu. After opening the menu, the user must further select the desired setting or function (color, thickness, line type, etc.).

[0005] In view of the above-mentioned problems, an object of the present invention is to provide an input device with improved operability for handwriting input. [Means for solving the problem]

[0006] In consideration of the above problems, the present invention provides an input device comprising a handwriting input unit that accepts handwritten input using the position of a pen or a user's finger in contact with a display, and a display unit that displays the handwritten input accepted by the handwriting input unit as a handwritten object on the display, and is characterized in that if no change occurs in the handwritten object for a first period of time, one or more operation commands are displayed based on the handwritten object. Effect of the Invention

[0007] It is possible to provide an input device with improved operability for handwritten input. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an operation menu displayed by a conventional handwriting input device. [Diagram 2] FIG. 1 illustrates an example of a multi-function pen. [Diagram 3] 1 is a diagram illustrating an example of a display of operation command candidates displayed by a handwriting input device; [Figure 4] 1 is a diagram illustrating an example of the overall configuration of a handwriting input device. [Diagram 5] FIG. 2 is a diagram illustrating a hardware configuration of an example of a handwriting input device. [Figure 6] FIG. 2 is a functional block diagram of an example showing functions of the handwriting input device in the form of blocks; [Figure 7] FIG. 2 is a functional block diagram illustrating an example of functions related to user authentication of the handwriting input device. [Figure 8] FIG. 11 is a diagram illustrating an example of pre-defined control data. [Figure 9] 5 is a diagram showing an example of dictionary data in a handwriting recognition dictionary section; FIG. [Figure 10] FIG. 13 is a diagram illustrating an example of dictionary data of a character string conversion dictionary unit. [Figure 11] FIG. 4 is a diagram showing an example of dictionary data in a predictive conversion dictionary section; [Figure 12]4A and 4B are diagrams illustrating an example of operation command definition data and system definition data held in an operation command definition unit. [Figure 13] 13 is a diagram showing an example of operation command definition data in a case where a selected object is selected by a handwritten object. FIG. [Figure 14] 13A and 13B are diagrams showing an example of display of operation command candidates based on operation command definition data when there is no selected object; [Figure 15] 11 is a diagram illustrating an example of designating a selected object; [Figure 16] 13A and 13B are diagrams showing an example of display of operation command candidates based on operation command definition data when a handwritten object is present; [Figure 17] 13A and 13B are diagrams showing an example of display of operation command candidates based on operation command definition data when a handwritten object is present; [Figure 18] FIG. 11 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 1). [Figure 19] FIG. 2 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 2). [Figure 20] FIG. 11 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 3). [Figure 21] FIG. 4 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 4). [Figure 22] FIG. 5 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 5). [Diagram 23] FIG. 6 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 6). [Figure 24] FIG. 11 is a diagram showing an example of a system configuration of a handwriting input system (embodiment 2); [Diagram 25] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing system. [Figure 26] FIG. 2 is an example of a functional block diagram illustrating functions of the handwriting input system in the form of blocks. [Figure 27] FIG. 11 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 1). [Figure 28] FIG. 2 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 2). [Figure 29] FIG. 11 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 3). [Diagram 30] FIG. 4 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 4). [Diagram 31] FIG. 5 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 5). [Diagram 32] FIG. 6 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 6). [Diagram 33] FIG. 7 is a sequence diagram of an example illustrating a process in which the handwriting input device displays character string candidates and operation command candidates (part 7). [Diagram 34] FIG. 11 is a diagram showing an example of an operation guide. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, as an example of an embodiment of the present invention, a handwriting input device 2 and a handwriting input method performed by the handwriting input device 2 will be described with reference to the drawings.

[0010] <Operation procedure of conventional handwriting input device> For convenience of explanation of the handwriting input device of this embodiment, first, an operating procedure of a conventional handwriting input device will be briefly explained.

[0011] 1 shows an operation menu displayed by a conventional handwriting input device. On the entire operation panel 101, a pen function menu button 1021 for displaying a pen function menu 102, an edit function menu button 1031 for displaying an edit function menu, an input / output function menu button 1041 for displaying an input / output function menu, etc. are displayed. In the pen function menu 102, the user can select the color, thickness, and operation mode of the pen, in the edit function menu 103, the user can select erasing, selecting, cutting, copying, and pasting objects, and in the input / output function menu 104, the user can load a template, load a file, save to a file, print, etc.

[0012] As an example of the operation procedure of a conventional handwriting input device, when a user is handwriting straight lines or curves (hereinafter referred to as strokes) and wants to erase multiple strokes and start writing again with the pen, the following operation procedure is required. Note that, unless otherwise specified, pressing with the pen means pressing with the pen tip. (A1) The edit function menu button 1031 is pressed with a pen to display the edit function menu 103. (A2) When the user presses the “Delete” button 1032 on the edit function menu 103, the handwriting input device automatically deletes the edit function menu 103. (A3) When the user writes with the pen across multiple strokes that are to be erased, the strokes are erased. (A4) The pen function menu button 1021 is pressed with the pen.

[0013] This process erases the strokes to be erased, and the user can then use the pen to write new strokes. Just erasing a stroke requires four steps. Although there are differences in the number of steps and screen transitions, handwriting input devices tend to have similar operation systems. Furthermore, this type of operation system is common to all computer products, and is based on an operation system that allows the user to call up the functions they need. If the user does not know how to call up a function, they will not be able to operate it at all, so it is common for handwriting input devices to provide instructions on how to call up functions through tutorial or help functions.

[0014] Similarly, when input / output such as file or print becomes necessary, the user opens the input / output function menu 104. For example, the following operation procedure is required to load a template. (B1) When the user presses the input / output function menu button 1041 with a pen, the input / output function menu 104 opens. (B2) When the user presses the "template selection" button 1042 in the input / output function menu 104 with the pen, the input / output function menu closes and a template selection window opens. (B3) Press the left or right button with the pen to scroll until the desired template is displayed. (B4) When the desired template is pressed with the pen, the template is loaded and the template selection window closes.

[0015] In this way, conventional handwriting input devices require users to follow step-by-step operating procedures. In other words, because handwriting input devices are based on an operating system that guides users to the next step in an explanatory manner, it is difficult for users to operate them intuitively.

[0016] As shown in Fig. 2, it is also being considered to provide functions to the pen itself in order to reduce the number of operating procedures. Fig. 2 shows an example of a multi-function pen (hereinafter referred to as active pen 110). Active pen 110 is a pen that has a built-in power source and can send commands to a handwriting input device. Active pen 110 in Fig. 2 has one physical switch at the tip of the pen, one at the end of the pen, and two on the side of the pen; the tip of the pen is for writing, the end of the pen is for erasing, and the side of the pen is for allocating user functions.

[0017] The user can assign one of the pen side switches to the pen function menu 102 and the other to the edit function menu 103. Although this is convenient in that the pen function menu 102 or the edit function menu 103 can be displayed by pressing the side button of the active pen 110 held by the user, the inconvenience of the user having to press the side button to open the pen function menu 102 or the edit function menu 103 every time the object to be operated is changed remains unchanged.

[0018] Furthermore, if the switch at the end of the pen for erasure is used, the above steps (A1), (A2), and (A4) are unnecessary, and the end of the pen can be used instead of the tip of the pen in step (A3), shortening the operation procedure required for erasure to one step, but this causes another inconvenience in that it is costly to purchase and operate a special pen called the active pen 110. Furthermore, the active pen 110 requires battery replacement or charging, and measures to prevent loss are also necessary depending on the usage situation. EXAMPLES

[0019] <Summary> Next, an overview of the handwriting input device 2 of this embodiment will be described with reference to Fig. 3. Fig. 3 is an example of a diagram for explaining a display example of operation command candidates (also called operation command candidates) displayed by the handwriting input device 2. Fig. 3(a) shows operation command candidates displayed when the user has handwritten characters, and Fig. 3(b) shows operation command candidates displayed when the user has handwritten an encircling line. First, Fig. 3(a) will be described.

[0020] A. When the user writes the handwritten object 11 (here a character) "gi" with a pen, the handwriting input device 2 performs character recognition and converts it to "gi".

[0021] B. The handwriting input device 2 further converts "gi" to a Chinese character. In Fig. 3(a), as an example, it is converted to "yi", but any Chinese character in the dictionary can be converted.

[0022] C. The handwriting input device 2 further converts "yi" to the predicted character string predicted from "yi". In Fig. 3(a), as an example, it is converted to "minutes", but any character string in the dictionary can be converted.

[0023] D. The handwriting input device 2 displays candidates for operation commands including "gi", "yi", or "minutes". That is, as operation command candidates associated with "gi", "yi", or "minutes", candidates 510 for operation commands such as "load minutes template" and "save to minutes folder" related to input / output functions are displayed. The candidates for operation commands are not limited to two. When a candidate 510 for an operation command is selected, the handwriting input device 2 executes file input / output according to the selected operation command.

[0024] Therefore, as a user, by writing "gi", it is possible to display candidates 510 for operation commands such as "load minutes template" or "save to minutes folder", and from the state of writing, the input / output functions shown in Fig. 1 can be called.

[0025] Next, Fig. 3(b) will be described.

[0026] E. The user surrounds the determined object 13 "minutes" with a handwritten object 11 (here a bounding box).

[0027] F. The handwriting input device 2 determines whether the handwritten object 11 and the determined object 13 satisfy a predetermined criterion (for example, whether they overlap).

[0028] G. When the handwritten object 11 and the finalized object 13 satisfy a predetermined criterion, the handwriting input device 2 displays candidates 510 for editing or modification operation commands. In Fig. 3(b), the operation command candidates 510 "delete", "move", "rotate", and "select" related to the editing function are displayed. The number of operation command candidates 510 is not limited to four.

[0029] Therefore, by encircling the confirmed object 13 with a line, the user can display operation command candidates 510 for changing the confirmed object 13, and can call up an editing function while still in the handwriting state.

[0030] In this way, the handwriting input device 2 of the present embodiment can call input / output functions, editing functions, etc. while the user is writing by hand. The pen function can be similarly called. Therefore, the number of operation steps required for the user to call any function while writing by hand can be reduced.

[0031] <Terminology> The input means may be any means that allows handwriting by specifying coordinates on a touch panel, such as a pen, a human finger or hand, a rod-shaped member, etc. Also, eye-gaze input may be possible.

[0032] A series of operations in which a user presses an input means against a display, moves it continuously, and then releases it from the display is called a stroke. Stroke data is information that is displayed on the display based on the trajectory of coordinates input by the input means. Stroke data may be appropriately interpolated.

[0033] An object that is displayed on a display based on stroke data is called an object. Object means a target, but in this embodiment, it means a display target.

[0034] A handwritten object is an object that is composed of one or more stroke data and is a display target that is handwritten by a user. A handwritten object rectangular area is an area that is indicated by a circumscribing rectangle that surrounds a handwritten object.

[0035] The determined object is an object that has been converted into a character code (font) by character recognition of a handwritten object and selected by a user, or a handwritten object that has been determined not to be converted into a character code (font).

[0036] The input device is a device that accepts input of information based on the contact position of an input means on a touch panel, and will be described as a handwriting input device 2 in this embodiment.

[0037] An operation command refers to a command for instructing the execution of a specific process, which is prepared for operating the handwriting input device 2. In this embodiment, as an example, an editing, modification, or input / output operation command is taken, but all commands for operating the handwriting input device 2, such as screen inversion, page switching, and operation mode setting, are subject to the operation command.

[0038] <Overall configuration of the device> The overall configuration of the handwriting input device 2 according to the embodiment of the present invention will be described with reference to Fig. 4. Fig. 4 is a diagram showing the overall configuration of the handwriting input device 2. Here, an electronic whiteboard is shown as an example of the handwriting input device 2.

[0039] 4, a display 220 as an example of a display device is provided on the upper part of the handwriting input device 2. A user can input (draw) characters or the like on the display 220 using a pen 2500.

[0040] <Hardware configuration of the device> Next, the hardware configuration of the handwriting input device 2 will be described with reference to Fig. 5. As shown in the figure, the handwriting input device 2 has the configuration of an information processing device or a computer. Fig. 5 is an example of a hardware configuration diagram of the handwriting input device 2. As shown in Fig. 5, the handwriting input device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an SSD (Solid State Drive) 204.

[0041] Of these, the CPU 201 controls the overall operation of the handwriting input device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201. The SSD 204 stores various data such as programs for the handwriting input device 2.

[0042] The handwriting input device 2 also includes a display controller 213 , a touch sensor controller 215 , a touch sensor 216 , a display 220 , and a power switch 222 .

[0043] The display controller 213 controls and manages the screen display to output an output image to the display 220, etc. The touch sensor 216 detects that the pen 2500, the user's hand, etc. (the pen and the user's hand are input means) have touched the display 220. The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 inputs and detects coordinates. For example, in the case of an optical type, the method of inputting and detecting the coordinates is a method in which two light receiving and emitting devices installed at both ends of the upper side of the display 220 emit a plurality of infrared rays parallel to the display 220, and receive the light reflected by a reflecting member installed around the display 220 and returning on the same optical path as the optical path of the light emitted by the light receiving element. The touch sensor 216 outputs position information of the infrared rays radiated by the two light receiving and emitting devices that have been blocked by an object to the touch sensor controller 215, and the touch sensor controller 215 specifies the coordinate position that is the contact position of the object.

[0044] The power switch 222 is a switch for switching the power of the handwriting input device 2 ON / OFF.

[0045] Furthermore, the handwriting input device 2 includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201 shown in FIG.

[0046] The touch sensor 216 is not limited to an optical type, and may be a capacitive touch panel that identifies the contact position by detecting a change in capacitance, a resistive film touch panel that identifies the contact position by a voltage change between two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by a contact object touching the display unit. The touch sensor 216 may be of a type that does not require an electronic pen to detect the presence or absence of a touch with the tip of the pen. In this case, a fingertip or a pen-shaped stick can be used to perform a touch operation. The pen 2500 does not need to be a long, thin pen type.

[0047] <About the device functions> Next, the functions of the handwriting input device 2 will be described with reference to FIG. 6. FIG. 6 is an example of a functional block diagram showing the functions of the handwriting input device 2 in a block form. The handwriting input device 2 includes a handwriting input unit 21, a display unit 22, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a character string conversion control unit 28, a character string conversion dictionary unit 29, a predictive conversion control unit 30, a predictive conversion dictionary unit 31, an operation command recognition control unit 32, and an operation command definition unit 33. Each function of the handwriting input device 2 is a function or means realized by operating any of the components shown in FIG. 5 by an instruction from the CPU 201 according to a program expanded from the SSD 204 onto the RAM 203.

[0048] The handwriting input unit 21 is realized by a touch sensor 216 etc., and accepts handwriting input by a user. The handwriting input unit 21 converts the user's pen input d1 into pen operation data d2 (pen up, pen down, or pen coordinate data) and transmits it to the handwriting input display control unit 23. The pen coordinate data is periodically transmitted as discrete values, and the coordinates between the discrete values ​​are interpolated.

[0049] The display unit 22 is realized by a display 220 or the like, and displays handwritten objects and operation menus. The display unit 22 converts the drawing data d3 written in the video memory by the handwriting input display control unit 23 into data according to the characteristics of the display 220, and transmits it to the display 220.

[0050] The handwriting input display control unit 23 performs overall control over handwriting input and display. The handwriting input display control unit 23 processes pen operation data d2 from the handwriting input unit 21, and transmits the data to the display unit 22 for display. Details of the processing of the pen operation data d2 and the display of strokes will be described later with reference to Figs. 18 to 23.

[0051] The candidate display timer control unit 24 is a display control timer for selectable candidates. It starts or stops the timer to generate the timing for starting the display of selectable candidates and the timing for erasing the display. The candidate display timer control unit 24 receives a timer start request d4 (or a timer stop request) from the handwriting input display control unit 23, and transmits a timeout event d5 to the handwriting input display control unit 23.

[0052] The handwriting input saving unit 25 has a storage function for saving user data (handwritten objects / character string objects). The handwriting input saving unit 25 receives user data d6-1 from the handwriting input display control unit 23 and saves it in the handwriting input saving unit 25, and receives an acquisition request d6-2 from the handwriting input display control unit 23 and transmits the user data d7 saved in the handwriting input saving unit 25. The handwriting input saving unit 25 transmits position information d36 of the confirmed object to the operation command recognition control unit 32.

[0053] The handwriting recognition control unit 26 is a recognition engine that performs online handwriting recognition. Unlike a typical OCR (Optical Character Reader), it recognizes characters (not only Japanese but also multiple languages ​​such as English), numbers, symbols (%, $, &, etc.), figures (lines, circles, triangles, etc.), etc. in parallel with the user's pen operation. Various algorithms have been devised for the recognition method, but details will be omitted since known techniques can be used in this embodiment.

[0054] The handwriting recognition control unit 26 receives the pen operation data d8-1 from the handwriting input display control unit 23, performs handwriting recognition, and holds the handwriting-recognized character string candidates. The handwriting recognition control unit 26 also holds the language character string candidates converted from the handwriting-recognized character string candidates d12 using the handwriting recognition dictionary unit 27. When the handwriting recognition control unit 26 separately receives an acquisition request d8-2 from the handwriting input display control unit 23, it transmits the held handwriting-recognized character string candidates and language character string candidates d9 to the handwriting input display control unit 23.

[0055] The handwriting recognition dictionary unit 27 stores dictionary data for language conversion in handwriting recognition. The handwriting recognition dictionary unit 27 receives handwriting recognition character string candidates d12 from the handwriting recognition control unit 26, converts them into linguistically probable language character string candidates d13, and transmits them to the handwriting recognition control unit 26. For example, in the case of Japanese, hiragana is converted into kanji or katakana.

[0056] The character string conversion control unit 28 controls the conversion of the conversion character string candidates into character strings. A conversion character string is a character string that is likely to be generated by including a handwriting recognition character string or a language character string. The character string conversion control unit 28 receives the handwriting recognition character string and the language character string candidates d11 from the handwriting recognition control unit 26, converts them into conversion character string candidates using the character string conversion dictionary unit 29, and stores them. When the character string conversion control unit 28 separately receives an acquisition request d14 from the handwriting input display control unit 23, it transmits the stored conversion character string candidates d15 to the handwriting input display control unit 23.

[0057] The character string conversion dictionary unit 29 stores dictionary data for character string conversion. The character string conversion dictionary unit 29 receives handwritten recognition character strings and language character string candidates d17 from the character string conversion control unit 28, and transmits converted character string candidates d18 to the character string conversion control unit 28.

[0058] The predictive conversion control unit 30 receives the handwriting-recognized character string and the language character string candidate d10 from the handwriting recognition control unit 26, and receives the converted character string candidate d16 from the character string conversion control unit 28, and converts each of them into a predicted character string candidate using the predictive conversion dictionary unit 31. A predicted character string candidate is a character string that is likely to be generated by including a handwriting-recognized character string, a language string, or a converted character string. When an acquisition request d19 is separately received from the handwriting input display control unit 23, the predictive conversion control unit 30 transmits the predicted character string candidate d20 to the handwriting input display control unit 23.

[0059] The predictive conversion dictionary unit 31 stores dictionary data for predictive conversion. The predictive conversion dictionary unit 31 receives handwritten recognition character strings, language character string candidates, and conversion character string candidates d21 from the predictive conversion control unit 30, and transmits predicted character string candidates d22 to the predictive conversion control unit 30.

[0060] The operation command recognition control unit 32 receives the handwriting recognition character string and language character string candidates d30 from the handwriting recognition control unit 26, receives the converted character string candidates d28 from the character string conversion control unit 28, and receives the predicted character string candidates d29 from the predictive conversion control unit 30. Then, for each of them, the operation command conversion request d26 is transmitted to the operation command definition unit 33, and the operation command candidates d27 are received from the operation command definition unit 33. The operation command recognition control unit 32 holds the operation command candidates d27.

[0061] If the operation command conversion request d26 partially matches the operation command definition, the operation command definition unit 33 transmits operation command candidates d27 to the operation command recognition control unit 32.

[0062] The operation command recognition control unit 32 also receives pen operation data d24-1 from the handwriting input display control unit 23, transmits a request d23 to the handwriting input storage unit 25 to obtain position information of a previously input and confirmed object, and stores the confirmed object designated by the pen operation data as a selected object (including position information). The operation command recognition control unit 32 identifies a selected object that satisfies the position of the pen operation data d24-1 and a predetermined criterion. When the operation command recognition control unit 32 separately receives an acquisition request d24-2 from the handwriting input display control unit 23, it transmits the selected object d25, which has been identified as a candidate for the stored operation command, to the handwriting input display control unit 23.

[0063] <About user authentication> In this embodiment, since control is performed using the result of user authentication, it is preferable that the handwriting input device 2 has a function of performing user authentication. For this reason, a function relating to user authentication will be described with reference to FIG.

[0064] Fig. 7 is a block diagram showing functions related to user authentication possessed by the handwriting input device 2. Note that, in Fig. 7, only the handwriting input display control unit 23 is shown as a function related to the user authentication unit 34 (user authentication means), but the functions shown in Fig. 6 may be used as the result of user authentication.

[0065] The authentication information acquisition unit 35 acquires authentication information d31 from the user. The authentication information d31 may be an IC card number, a user ID and password, or biometric information such as a fingerprint. The user authentication unit 34 acquires authentication information d32 from the authentication information acquisition unit 35 and searches for authentication information d33 in the user information DB 36. If a user matching the search is found, user information d34 is acquired from the user information DB 36. The user information may be any information that represents the attributes of the user, and may be, for example, a user name, a password, the user's computer name, a department, or authority.

[0066] The user authentication unit 34 transmits the user information d35 to the handwriting input display control unit 23, so that the handwriting input display control unit 23 can execute an operation command using the user information. The operation command using the user information will be described with reference to FIG.

[0067] The authentication function may be provided by an external authentication server or the like, in addition to the handwriting input device 2. In this case, the handwriting input device 2 transmits authentication information to the authentication server and acquires the authentication result and user information.

[0068] <About predefined control data> Next, the predefined control data used by the handwriting input device 2 for various processes will be described with reference to Fig. 8. Fig. 8 shows an example of the predefined control data. In the example of Fig. 8, the control data is shown for each control item.

[0069] The selectable candidate display timer 401 defines the time until selectable candidates are displayed (an example of a first time). This is because selectable candidates are not displayed while handwriting. In FIG. 8, this means that if pen-down does not occur within TimerValue=500 [ms] after pen-up, selectable candidates are displayed. The selectable candidate display timer 401 is held by the candidate display timer control unit 24. The selectable candidate display timer 401 is used when the selectable candidate display timer starts in step S4.8 of FIG. 20, which will be described later.

[0070] The selectable candidate deletion timer 402 defines the time until the displayed selectable candidates are deleted (an example of a second time). This is to delete the selectable candidates if the user does not select a selectable candidate. In FIG. 8, this means that if a selectable candidate is not selected within TimerValue=5000 [ms] from the display of the selectable candidates, the selectable candidate display is deleted. The selectable candidate deletion timer 402 is held by the candidate display timer control unit 24. The selectable candidate deletion timer 402 is used when the selectable candidate display deletion timer starts in step S4.8.1.8 in FIG. 21.

[0071] The handwritten object vicinity rectangular area 403 defines a rectangular area that is considered to be near the handwritten object. In the example of FIG. 8, the handwritten object vicinity rectangular area 403 is a rectangular area obtained by enlarging the handwritten object rectangular area by 50% (Horizontal) of the estimated character size in the horizontal direction and by 80% (Vertical) of the estimated character size in the vertical direction. In the example of FIG. 8, the rectangular area is a percentage of the estimated character size (specified as a percentage), but it is also possible to set the length to a fixed length by using units such as "mm". The handwritten object vicinity rectangular area 403 is stored in the handwritten input storage unit 25. The estimated character size 405 is used in step S4.2 of FIG. 19 to determine the overlap state between the handwritten object vicinity rectangular area and the stroke rectangular area.

[0072] The estimated writing direction / character size determination condition 404 defines a constant for determining the measurement direction of the writing direction and the character size. In the example of FIG. 8, if the difference between the time when the first stroke was added to the rectangular area of ​​the handwritten object and the time when the last stroke was added is MinTime=1000 [ms] or more, the difference between the horizontal distance (width) and the vertical distance (height) of the rectangular area of ​​the handwritten object is MinDiff=10 [mm] or more, and the horizontal distance is longer than the vertical distance, it means that the estimated writing direction is determined to be "horizontal writing" and the estimated character size is determined to be the vertical distance. If the horizontal distance is shorter than the vertical distance, it means that the estimated writing direction is determined to be "vertical writing" and the estimated character size is determined to be the horizontal distance. If the above conditions are not met, it means that the estimated writing direction is determined to be "horizontal writing" (DefaultDir="Horizontal") and the estimated character size is determined to be the longer of the horizontal distance and the vertical distance. The estimated writing direction / character size determination condition 404 is held by the handwritten input storage unit 25. The estimated writing direction / character size judgment condition 404 is used in obtaining an estimated writing direction in step S4.8.1.5 in FIG. 21 and in obtaining a character string object font in step S7.8.

[0073] The estimated character size 405 defines data for estimating the size of characters, etc. In the example of FIG. 8, this means that the estimated character size determined by the estimated writing direction / character size determination condition 404 is compared with the small character 405a (hereinafter referred to as the minimum font size) and the large character 405c (hereinafter referred to as the maximum font size) of the estimated character size 405. If the estimated character size is smaller than the minimum font size, the estimated character size is determined to be the minimum font size. If the estimated character size is larger than the maximum font size, the estimated character size is determined to be the maximum font size. Otherwise, the character size is determined to be the medium character 405b. The estimated character size 405 is held by the handwriting input storage unit 25. The estimated character size 405 is used in obtaining the character string object font in step S7.8 of FIG. 23.

[0074] Specifically, the handwriting input storage unit 25 compares the estimated character size determined by the estimated writing direction / character size determination condition 404 with the FontSize of the estimated character size 405, and uses the font with the closest size. For example, if the estimated character size is 25 mm (FontSize of small characters) or less, it is determined to be a "small character", if the estimated character size is more than 25 mm and less than 50 mm (FontSize of medium characters) it is determined to be a "medium character", and if the estimated character size is more than 100 mm (FontSize of large characters), it is determined to be a "large character". The "small character" 405a uses a 25 mm font in Mincho style (FontStyle="Mincho" FontSize="25mm"), the "medium character" 405b uses a 50 mm font in Mincho style (FontStyle="Mincho" FontSize="50mm"), and the "large character" 405c uses a 100 mm font in Gothic style (FontStyle="Gothic" FontSize="100mm"). If you want more font sizes or styles, you can increase the number of estimated character sizes 405.

[0075] The crossing line determination condition 406 defines data used to determine whether multiple objects are selected. If the handwritten object is a single stroke, and in the example of FIG. 8, the handwritten object has a long side length of 100 mm or more (MinLenLongSide="100mm"), a short side length of 50 mm or less (MaxLenShortSide="50mm"), and an object has an overlap rate of 80% or more (MinOverLapRate="80%") in the long side direction and short side direction with the handwritten object (a predetermined overlap rate or more), it is determined that multiple objects are selected (selected objects). Instead of "above or below," it may be determined that the overlap rate is "greater than." The crossing line determination condition 406 is held by the operation command recognition control unit 32. The crossing line determination condition 406 is used in the crossing line determination for determining the selected object in step S4.7.6.2 of FIG. 20.

[0076] The encircling line determination condition 407 defines data used to determine whether an object is an encircling line. In the example of FIG. 8, the operation command recognition control unit 32 determines a confirmed object whose overlap rate in the long side direction and short side direction of a handwritten object is 100% or more (MinOverLapRate="100%") as a selected object (a predetermined overlap rate or more). The overlap rate may be determined to be "greater than" rather than "above or equal." The encircling line determination condition 407 is held by the operation command recognition control unit 32. The encircling line determination condition 407 is used in the encircling line determination for determining the selected object in step S4.6.7.2 of FIG. 20.

[0077] Either of the crossing line determination condition 406 and the encircling line determination condition 407 may be given priority in the determination. For example, if the crossing line determination condition 406 is loosened (to make it easier to select a crossing line) and the encircling line determination condition 407 is set to a strict value (to make it possible to select only an encircling line), the operation command recognition control unit 32 may give priority to the encircling line determination condition 407 in the determination.

[0078] <An example of dictionary data> The dictionary data will be described with reference to Fig. 9 to Fig. 11. Fig. 9 is an example of dictionary data in the handwriting recognition dictionary unit 27, Fig. 10 is an example of dictionary data in the character string conversion dictionary unit 29, and Fig. 11 is an example of dictionary data in the predictive conversion dictionary unit 31. These dictionary data are used in steps S4.7.2 to S4.7.4 in Fig. 20, respectively.

[0079] In this embodiment, the conversion result by the dictionary data of the handwriting recognition dictionary unit 27 in FIG. 9 is called a language string candidate, the conversion result by the dictionary data of the string conversion dictionary unit 29 in FIG. 10 is called a converted string candidate, and the conversion result by the dictionary data of the predictive conversion dictionary unit 31 in FIG. 11 is called a predicted string candidate. "Before conversion" of each dictionary data is a string to search the dictionary data, "after conversion" is a converted string corresponding to the string to be searched, and "probability" represents the probability of selection by the user. The probability is calculated from the result of the user selecting each string in the past. Therefore, the probability may be calculated for each user. Various algorithms have been devised as a method of calculating the probability, but it is sufficient to calculate it by an appropriate method as appropriate, and details will be omitted. This embodiment is characterized in that the string candidates are displayed in descending order of selection probability from the estimated writing direction.

[0080] The dictionary data of the handwriting recognition dictionary unit 27 in Fig. 9 indicates that handwritten "gi" is converted to "gi" with a probability of 0.55 and to "gi" with a probability of 0.4, and handwritten "gishi" is converted to "gishi" with a probability of 0.5 and to "gishi" with a probability of 0.45. The same is true for other "before conversion" character strings. In Fig. 9, the "before conversion" character string is handwritten hiragana, but characters other than hiragana may be registered as "before conversion".

[0081] 10, the dictionary data in the character string conversion dictionary unit 29 indicates that the character string "" is converted to "" with a probability of 0.95, and the character string "" is converted to "" with a probability of 0.85. The same is true for the other "pre-conversion" character strings.

[0082] The dictionary data of the predictive conversion dictionary unit 31 in Fig. 11 indicates that the character string "Minutes" is converted to "Destination of minutes" with a probability of 0.65, and the character string "Skill test" is converted to "Approve skill test" with a probability of 0.75. In the example in Fig. 11, all the character strings before conversion are kanji, but characters other than kanji may be registered.

[0083] The dictionary data is not language-dependent, and any character string may be registered before and after conversion.

[0084] <Operation command definition data held by the operation command definition section> Next, the operation command definition data used by the operation command recognition control unit 32 will be described with reference to Fig. 12 and Fig. 13. Fig. 12 shows an example of the operation command definition data and system definition data held by the operation command definition unit 33.

[0085] Fig. 12(a) shows an example of operation command definition data. The operation command definition data in Fig. 12(a) is an example of operation command definition data when there is no selected object selected by a handwritten object, and covers all operation commands that operate the handwriting input device 2. The operation command in Fig. 12(a) has an operation command name (Name), a string (String) that partially matches a string candidate, and an operation command string (Command) to be executed. "%~%" in the operation command string is a variable, and is associated with system definition data as shown in Fig. 12(b). In other words, "%~%" is replaced with the system definition data shown in Fig. 12(b).

[0086] First, the operation command definition data 701 indicates that the operation command name is "Read minutes template", the string that partially matches the string candidate is "minutes" or "template", and the operation command string to be executed is "ReadFile https: / / %username%:%password%@server.com / template / minutes.pdf". In this example, the operation command string to be executed contains the system definition data "%~%", and "%username%" and "%password%" are replaced with the system definition data 704 and 705, respectively. Therefore, the operation command string to be finally executed is the string "ReadFile https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf", which indicates that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf" is to be read (ReadFile).

[0087] Operation command definition data 702 indicates that the operation command name is "Save in minutes folder", the string that partially matches the string candidate is "Minutes" or "Save", and the operation command string to be executed is "WriteFile https: / / %username%:%password%@server.com / minutes / %machinename%_%yyyy-mm-dd%.pdf". As with operation command definition data 701, "%username%", "%password%", and "%machinename%" in the operation command string are replaced with system definition data 704 to 706, respectively. Note that "%yyyy-mm-dd%" indicates that it should be replaced with the current date. For example, if the current date is September 26, 2018, it should be replaced with "2018-09-26". The final operation command to be executed is "WriteFile https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf", which indicates that the minutes will be saved (WriteFile) to the file "https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf".

[0088] The operation command definition data 703 indicates that the operation command name is "Print", the character string that partially matches the character string candidate is "Print" or "Print", and the operation command character string to be executed is "PrintFile https: / / %username%:%password%@server.com / print / %machinename%-"%yyyy-mm-dd%.pdf". When the operation command character string is replaced in the same manner as the operation command definition data 702, the operation command to be finally executed becomes "PrintFile https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf", which indicates that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf" is to be printed (PrintFile). In other words, the file is sent to the server. When the user causes the printer to communicate with the server and specifies a file, the printer prints the contents of the file on paper.

[0089] In this way, the operation command definition data 701-703 can be identified from the character string candidates, so that the user can display the operation command by handwriting. Also, if the user authentication is successful, "%username%", "%password%", etc. of the operation command definition data are replaced with the user information, making it possible to input and output files in association with the user.

[0090] If user authentication is not performed (including the case where authentication fails but the user can use the handwriting input device 2), the handwriting input device 2 replaces it with "%username%", "%password%", etc., that are preset for the handwriting input device 2. Therefore, file input and output becomes possible in association with the handwriting input device 2 even without user authentication.

[0091] Next, we will explain the operation command definition data when there is a handwritten object, that is, the operation command definition data for editing and modification. Fig. 13 shows an example of the operation command definition data when there is a selected object selected by a handwritten object. The operation command definition data in Fig. 13 has the operation command name (Name), the group name (Group) of the operation command candidates, and the operation command string (Command) to be executed.

[0092] The operation command definition data 707 defines the operation commands of the edit system (Group="Edit"), and is an example of the definition data of the edit system operation command names "Delete", "Move", "Rotate", and "Select". In other words, these operation commands are displayed for the selected object, and the user can select the desired operation command.

[0093] The operation command definition data 708 defines the operation commands of the modification system (Group="Decorate"), and is an example of definition data for the names of the operation commands of the modification system, "Thicken", "Thinner", "Larger", "Smaller", and "Underline". These operation commands are displayed for the selected object, and the user can select the desired operation command. In addition, color operation commands may be displayed.

[0094] Therefore, when the user selects a selection object with a handwritten object, the operation command definition data 707, 708 is specified, so that the user can display the operation command by handwriting.

[0095] <Example of display of operation command candidates> Fig. 14 shows an example of displaying operation command candidates based on the operation command definition data (when no selected object is present) in Fig. 12(a). The display procedure for the operation command candidates will be described in detail with reference to the sequence diagrams of Figs. 18 to 23.

[0096] 14 shows an example of an operation guide 500 and selectable candidates 530 displayed by the operation guide 500. The operation guide 500 is displayed when the user handwrites a handwritten object 504 (due to a timeout of the selectable candidate display timer). The operation guide 500 has an operation header 520, operation command candidates 510, handwritten recognition character string candidates 506, converted character string candidates 507, predicted character string candidates 508, and a handwritten object rectangular area display 503. The selectable candidates 530 are the operation command candidates 510, the handwritten recognition character string candidates 506 (including language character string candidates 571, if any), the converted character string candidates 507, and the predicted character string candidates 508. The selectable candidates 530 excluding the operation command candidates 510 are referred to as character string candidates 539.

[0097] The operation header 520 has buttons 501, 509, 502, and 505. Button 501 accepts an operation to switch predictive conversion on and off. In the example of FIG. 14, when the user presses button 501 displaying "Predict", handwriting input unit 21 accepts it and notifies handwriting input display control unit 23 of the same, and display unit 22 changes the display to button 501 displaying "Kana". After the change, predictive conversion is not performed (converted character string candidate 507 and predicted character string candidate 508 are not displayed), and operation command candidate 510, handwritten recognition character string candidate 506, and language character string candidate (kanji converted from kana to kanji, such as 'gi') are displayed.

[0098] Button 502 operates the page of the candidate display. In the example of FIG. 14, there are three candidate display pages, and the first page is currently displayed. Button 505 accepts the deletion of the operation guide 500. When the user presses button 505, the handwriting input unit 21 accepts it and notifies the handwriting input display control unit 23 of the same, and the display unit 22 erases the display except for the handwritten objects. Button 509 accepts the deletion of all displays. When the user presses button 509, the handwriting input unit 21 accepts it and notifies the handwriting input display control unit 23 of the same, and the display unit 22 erases all displays shown in FIG. 14, including the handwritten objects, allowing the user to start over with handwriting.

[0099] In Fig. 14, a handwritten object 504 is the character "gi" handwritten by the user. A handwritten object rectangular area display 503 is displayed surrounding the handwritten object 504. The display procedure will be described with reference to the sequence diagrams of Figs. 18 to 23. In the example of Fig. 14, the handwritten object rectangular area display 503 is displayed in a dotted line frame.

[0100] The character string candidates are arranged in descending order of probability in the handwriting recognition character string candidate 506, the converted character string candidate 507, and the predicted character string candidate 508. The handwriting recognition character string candidate 506 "gi" is a candidate for the recognition result. In this example, "gi" is correctly recognized.

[0101] A converted string candidate 507 is a string candidate converted from a language string candidate. In this example, "skill test" is an abbreviation for "technical mass production trial." A predicted string candidate 508 is a predicted string candidate converted from a handwritten recognition string, a language string candidate, or a converted string candidate. In this example, "Approve the skill test" and "Destination to send the minutes" are displayed.

[0102] The operation command candidate 510 is a candidate for an operation command selected based on the operation command definition data 701 to 703 in Fig. 11(a). In the example of Fig. 14, the first character of the line "》" 511 is shown to be a candidate for an operation command. In Fig. 14, there is no selection data to be selected by the handwritten object 504 "ぎ", and the character string candidate for "ぎ" "Meeting minutes" partially matches the operation command definition data 701 and 702 in Fig. 11(a), and is therefore displayed as the operation command candidate 510.

[0103] When the user selects "Load minutes template", the operation command defined in operation command definition data 701 is executed, and when the user selects "Save in minutes folder", the operation command defined in operation command definition data 702 is executed. In this way, operation command candidates are displayed when operation command definition data containing the converted character string is found, so they are not always displayed.

[0104] As shown in FIG. 14, character string candidates and operation command candidates are displayed simultaneously (together), so the user can arbitrarily select either the character string candidate or the operation command that he or she intended to input.

[0105] <Example of selecting objects> In the handwriting input device 2 of this embodiment, the user can designate a selected object by selecting a fixed object by handwriting. The selected object is to be edited or modified.

[0106] Fig. 15 is an example of a diagram for explaining an example of designation of a selected object. In Fig. 15, a handwritten object 11 is displayed with a solid black line, a handwritten object rectangular area 12 is displayed with a gray shading, a confirmed object 13 is displayed with a black line, and a selected object rectangular area 14 is displayed with a dotted line. Note that lowercase alphabets are added to the symbols when distinguishing between them. In addition, as a judgment condition for judging a confirmed object as a selected object (whether or not there is a predetermined relationship), the crossing line judgment condition 406 or the encircling line judgment condition 407 of the defined control data shown in Fig. 8 is used.

[0107] 15(a) is an example in which the user specifies two horizontally written determined objects 13a and 13b with a crossover line (handwritten object 11a). In this example, the length H1 of the short side and the length W1 of the long side of the handwritten object rectangular area 12a satisfy the crossover line determination condition 406, and the overlap rate with the determined objects 13a and 13b satisfies the crossover line determination condition 406, so both the determined objects 13a and 13b, "minutes" and "giji", are specified as selected objects.

[0108] 15B shows an example in which a horizontally written fixed object 13c is specified by an encircling line (handwritten object 11b). In this example, only the fixed object 13c called "minutes" is specified as the selected object because the overlap rate between the fixed object 13c and the handwritten object rectangular area 12c satisfies the condition of the encircling line determination condition 407.

[0109] Fig. 15(c) is an example in which multiple vertically written determined objects 13d, 13e are specified by a crossing line (handwritten object 11c). In this example, similar to Fig. 15(a), the short side length H1 and long side length W1 of the handwritten object rectangular area 12d satisfy the crossing line determination condition 406, and the overlapping rate with each of the two determined objects 13d, 13e, "minutes" and "giji", satisfy the crossing line determination condition 406, so both the determined objects 13d, 13e, "minutes" and "giji", are specified as selected objects.

[0110] Fig. 15(d) is an example in which a vertically written fixed object 13f is specified by enclosing the handwritten object 11d. In this example, just like Fig. 15(b), only the fixed object 13f called "minutes" is specified as the selected object.

[0111] <Example of display of operation command candidates> Fig. 16 shows an example of displaying operation command candidates based on the operation command definition data when there is the handwritten object shown in Fig. 13. Fig. 16(a) shows candidates for editing operation commands, and Fig. 16(b) shows candidates for modification operation commands. Fig. 16(a) also shows an example in which a selection object is specified by the handwritten object 11a in Fig. 15(a).

[0112] 16(a) and (b), the operation command candidates displayed following the first character "》" 511 are a main menu 550. The name of the last executed operation command or the first operation command name in the operation command definition data is displayed in the main menu 550. The first character "》" 511a on the first line is a candidate for an edit-related operation command, and the first character "》" 511b on the second line is a candidate for a modification-related operation command.

[0113] The ">" 512 at the end of the line indicates the presence of a sub-menu (an example of a sub-menu button). The ">" 512a on the first line displays the sub-menu (last selected) of the candidate operation commands in the editing system, and the ">" 512b on the second line displays the remaining sub-menus of the candidate operation commands in the modification system. When the user presses the ">" 512, the sub-menu 560 is displayed on its right side. All the operation commands defined in the operation command definition data are displayed in the sub-menu 560. In the display example of FIG. 16(a), the sub-menu 560 corresponding to the ">" 512a on the first line is also displayed since the main menu is displayed. It may be displayed by pressing the ">" 512a on the first line.

[0114] When the user presses any operation command name with the pen, the Command of the operation command definition data associated with the operation command name is executed by the handwritten input display control unit 23 for the selected object. That is, when "Delete" 521 is selected, "Delete" is executed, when "Move" 522 is selected, "Move" is executed, when "Rotate" 523 is selected, "Rotate" is executed, and when "Select" 524 is selected, "Select" is executed respectively.

[0115] For example, when the user presses "Delete" 521 with the pen, "Minutes" and "Proceedings" can be deleted. When "Move" 522, "Rotate" 523, or "Select" 524 is pressed, the bounding box (the circumscribed rectangle of the selected object) is displayed. With "Move" 522 and "Rotate" 523, it can be moved or rotated by dragging the pen, and with "Select" 524, other operations on the bounding box can be executed.

[0116] The strings "—" 541, "—," 542, "~" 543, "→" 544, "⇒" 545, which are string candidates other than the candidates of the operation commands, are the recognition results of the strike-through line (the handwritten object 11a). When the user intends to input a string rather than an operation command, the string candidate can be selected.

[0117] In Fig. 16(b), pressing ">" 512b on the second line displays the submenu of Fig. 16(b). The display example of Fig. 16(b) also displays a main menu 550 and a submenu 560, similar to Fig. 16(a). Based on the operation command definition data of Fig. 13, when "thick" 531 is selected, the handwriting input display control unit 23 executes "Thick", when "thin" 532 is selected, "Thin", when "bigger" 533 is selected, "Large", when "smaller" 534 is selected, "Small", and when underline 535 is selected, "Underline", for the selected object.

[0118] Note that fixed values ​​are separately defined for how thick to make the line when "Thick" 531 is selected, how thin to make the line when "Thin" 532 is selected, how large to make the line when "Larger" 533 is selected, how small to make the line when "Smaller" 534 is selected, and the line type when underline 535 is selected. Alternatively, it is better if a separate selection menu is opened when the submenu in Fig. 16(b) is selected, allowing the user to make adjustments.

[0119] When the user presses "Thicken" 531 with the pen, the handwriting input display control unit 23 thickens the lines that make up the final objects 13a and 13b, "minutes" and "giji." When the user presses "Thinner" 532 with the pen, the handwriting input display control unit 23 thins the lines that make up "minutes" and "giji." When the user presses "Larger" 533 with the pen, the handwriting input display control unit 23 enlarges the lines, when the user presses "Smaller" 534 with the pen, the handwriting input display control unit 23 makes the lines smaller, and when the user presses "Underline" 535 with the pen, the handwriting input display control unit 23 adds an underline.

[0120] FIG. 17 shows an example of display of operation command candidates based on the operation command definition data when there is a handwritten object as shown in FIG. 13. The difference from FIG. 16(a)(b) is that FIG. 17(a)(b) shows an example in which a selection object is specified by the handwritten object 11b (encircling line) in FIG. 15(b). As can be seen by comparing FIG. 16 with FIG. 17, there is no difference in the operation command candidates displayed depending on whether the handwritten object is a line or an encircling line. The handwritten input display control unit 23 displays the operation command candidates on the display unit 22 when the selection object is specified. However, the handwritten object may be recognized and the operation command candidates displayed according to the handwritten object may be changed. In this case, the recognized handwritten object (such as one or circle) is associated with the operation command definition data as shown in FIG. 13.

[0121] In FIG. 17, the character string candidates other than the operation command candidates, “○” 551, “∞” 552, “0” 553, “00” 554, and “ロ” 555, are the recognition results of the encircling line (handwritten object 11b), and if the user intends to input a character string instead of an operation command, the user can select a character string candidate.

[0122] <Operation procedure> The operation of the handwriting input device 2 will be described with reference to the above configuration and Figs. 18 to 23. Figs. 18 to 23 are sequence diagrams illustrating an example of the process in which the handwriting input device 2 displays character string candidates and operation command candidates. The process in Fig. 18 starts when the handwriting input device 2 is started (when an application is started). Note that, due to space limitations, Figs. 18 to 23 use symbols to indicate the functions in Fig. 6.

[0123] S1: First, the handwriting input display control unit 23 transmits a handwriting object start to the handwriting input storage unit 25. The handwriting input storage unit 25 secures a handwriting object area (a memory area for storing a handwritten object). The handwriting object area may be secured after the user brings the pen into contact with the handwriting input unit 21.

[0124] S2: Next, the user touches the pen to handwriting input unit 21. Handwriting input unit 21 detects pen-down and transmits the result to handwriting input display control unit 23.

[0125] S2.1: The handwritten input display control unit 23 transmits a stroke start to the handwritten input storage unit 25, and the handwritten input storage unit 25 secures a stroke area.

[0126] S3: When the user moves the pen while keeping it in contact with the handwriting input unit 21, the handwriting input unit 21 transmits the pen coordinates to the handwriting input display control unit 23.

[0127] S3.1: The handwriting input display control unit 23 transmits pen coordinate complement display data (data for interpolating discrete pen coordinates) to the display unit 22. The display unit 22 uses the pen coordinate complement display data to interpolate the pen coordinates and display a line.

[0128] S3.2: Handwriting input display control unit 23 transmits the pen coordinates and the time of reception thereof to handwriting input storage unit 25. Handwriting input storage unit 25 adds the pen coordinates to the stroke. While the user is moving the pen, handwriting input unit 21 periodically repeats transmission of the pen coordinates to handwriting input display control unit 23, so that the processes of steps S3 to S3.2 are repeated until the pen is lifted.

[0129] S4: When the user removes the pen from the handwriting input unit 21, the handwriting input unit 21 transmits a pen-up signal to the handwriting input display control unit 23.

[0130] S4.1: The handwriting input display control unit 23 transmits a stroke end signal to the handwriting input storage unit 25, and the handwriting input storage unit 25 confirms the pen coordinates of the stroke. After the pen coordinates of the stroke are confirmed, it becomes impossible to add pen coordinates to the stroke.

[0131] S4.2: Next, the handwriting input display control unit 23 transmits an overlap status acquisition between the rectangular area near the handwritten object and the stroke rectangular area based on the rectangular area near the handwritten object 403 to the handwriting input storage unit 25. The handwriting input storage unit 25 calculates the overlap status and transmits the overlap status to the handwriting input display control unit 23.

[0132] The subsequent steps S4.3 to S4.5 are executed if the hand-drawn object vicinity rectangular area and the stroke rectangular area do not overlap.

[0133] S4.3: If the handwritten object vicinity rectangular area and the stroke rectangular area do not overlap, one handwritten object is determined, and the handwritten input display control unit 23 transmits a retained data clear command to the handwriting recognition control unit 26.

[0134] S4.3.1 to 4.3.3: The handwriting recognition control unit 26 transmits a retained data clear command to the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32. The data relating to character string candidates and operation command candidates retained up to this point in the handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 are cleared. Note that at the time of clearing, the last handwritten stroke has not been added to the handwritten object.

[0135] S4.4: The handwriting input display control unit 23 transmits a handwriting object end notification to the handwriting input saving unit 25. The handwriting input saving unit 25 finalizes the handwritten object. Finalizing a handwritten object means that one handwritten object is completed (no more strokes are added).

[0136] S4.5: The handwriting input display control unit 23 transmits a handwriting object start signal to the handwriting input storage unit 25. In preparation for the start of handwriting of the next handwritten object (pen down), the handwriting input storage unit 25 secures a new handwriting object area.

[0137] S4.6: Next, the handwriting input display control unit 23 sends a stroke addition to the handwriting input storage unit 25 for the stroke completed in step S4.1. If steps S4.3 to S4.5 have been performed, the added stroke is the first stroke of the handwritten object, and the handwriting input storage unit 25 adds the stroke data to the starting handwritten object. If steps S4.3 to S4.5 have not been performed, the added stroke is added to the handwritten object that is already being handwritten.

[0138] S4.7: Next, the handwriting input display control unit 23 transmits a stroke addition to the handwriting recognition control unit 26. The handwriting recognition control unit 26 adds the stroke data to the stroke data holding area in which the character string candidates are stored.

[0139] S4.7.1: The handwriting recognition control unit 26 executes handwriting recognition on this stroke data storage area.

[0140] S4.7.2: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidates resulting from this execution to the handwriting recognition dictionary unit 27. The handwriting recognition dictionary unit 27 transmits the language character string candidates that are linguistically likely to be the handwriting recognition control unit 26.

[0141] S4.7.3: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidate and the received language character string candidate to the character string conversion control unit 28.

[0142] S4.7.3.1: The character string conversion control unit 28 transmits the handwriting recognition character string candidates and the language character string candidates to the character string conversion dictionary unit 29. The character string conversion dictionary unit 29 transmits the converted character string candidates to the character string conversion control unit 28.

[0143] S4.7.3.2: The character string conversion control unit 28 transmits the received converted character string candidate to the predictive conversion control unit 30.

[0144] S4.7.3.2.1: The predictive conversion control unit 30 transmits the received converted character string candidate to the predictive conversion dictionary unit 31. The predictive conversion dictionary unit 31 transmits the predicted character string candidate to the predictive conversion control unit 30.

[0145] S4.7.3.2.2: The predictive conversion control unit 30 transmits the received predicted character string candidate to the operation command recognition control unit 32.

[0146] S4.7.3.2.2.1: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. In this way, the operation command recognition control unit 32 can obtain the operation command candidate corresponding to the operation command definition data having a character string (String) matching the predicted character string candidate.

[0147] Thereafter, the same processing is carried out up to the transmission of operation command candidates in steps S4.7.3.3 to S4.7.5.1. S4.7.3.3: The character string conversion control unit 28 transmits the received converted character string candidate to the operation command recognition control unit 32.

[0148] S4.7.3.3.1: The operation command recognition control unit 32 transmits the received conversion character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. In this way, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) matching the conversion character string candidate.

[0149] S4.7.4: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidates and the language character string candidates to the predictive conversion control unit 30.

[0150] S4.7.4.1: The predictive conversion control unit 30 transmits the handwritten recognition character string candidate and the received language character string candidate to the predictive conversion dictionary unit 31. The predictive conversion dictionary unit 31 transmits the predicted character string candidate to the predictive conversion control unit 30.

[0151] S4.7.4.2: The predictive conversion control unit 30 transmits the received predicted character string candidate to the operation command recognition control unit 32.

[0152] S4.7.4.2.1: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. In this way, the operation command recognition control unit 32 can obtain the operation command candidate corresponding to the operation command definition data having a character string (String) matching the predicted character string candidate.

[0153] S4.7.5: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidates and the received language character string candidates to the operation command recognition control unit 32.

[0154] S4.7.5.1: The operation command recognition control unit 32 transmits the handwriting recognition character string candidate and the received language character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidates to the operation command recognition control unit 32. In this way, the operation command recognition control unit 32 can obtain operation command candidates corresponding to the operation command definition data having a character string (String) matching the language character string candidate.

[0155] S4.7.6: Next, the handwriting recognition control unit 26 transmits the stroke addition to the operation command recognition control unit 32.

[0156] S4.7.6.1: The operation command recognition control unit 32 transmits a request to acquire position information of the confirmed object to the handwriting input storage unit 25. The handwriting input storage unit 25 transmits the position information of the confirmed object to the operation command recognition control unit 32.

[0157] S4.7.6.2: To determine the selected object, the operation command recognition control unit 32 determines whether or not the position information of the stroke received from the handwriting recognition control unit 26 in the stroke addition in step S4.7.6 and the position information of the confirmed object received from the handwriting input storage unit 25 have a predetermined relationship based on the crossing line determination condition 406 and the encircling line determination condition 407, and stores the confirmed object that can be determined to be selected as the selected object. Also, in this case, since the selected object is specified, candidates for input / output system operation commands are obtained from the operation command definition unit 33.

[0158] In addition, the handwriting recognition control unit 26, the string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 each store handwriting recognition string candidates, language string candidates, conversion string candidates, predicted string candidates, operation command candidates, and data related to the selected object so that they can be acquired in subsequent steps S4.8.1.2 to 4.8.1.4.

[0159] S4.8: Immediately after transmitting the stroke addition command to the handwriting recognition control unit 26 in step S4.7, the handwriting input display control unit 23 transmits a command to start a selectable candidate display timer to the candidate display timer control unit 24. The candidate display timer control unit 24 starts this timer.

[0160] The following steps S5 to S5.2 are executed if a pen-down occurs before a certain period of time has elapsed (before the timer times out).

[0161] S5: If the user touches the pen to the handwriting input unit 21 before the timer times out, the handwriting input unit 21 transmits a pen-down signal (the same event as in step S2) to the handwriting input display control unit 23.

[0162] S5.1: The handwritten input display control unit 23 transmits a stroke start (same as step S2.1) to the handwritten input storage unit 25. The sequence thereafter is the same as that after step S2.1.

[0163] S5.2: Furthermore, the handwriting input display control unit 23 transmits a selectable candidate display timer stop command to the candidate display timer control unit 24. The candidate display timer control unit 24 stops the timer, because the timer is no longer necessary since pen-down has been detected.

[0164] Steps S4.8.1 to S7.14 are executed if the pen-down does not occur before a certain time has elapsed (before the timer times out). Therefore, the character string candidates and operation command candidates shown in FIG.

[0165] 4.8.1: If the user does not touch the pen to the handwriting input unit 21 while the selectable candidate display timer is started, the candidate display timer control unit 24 transmits a timeout to the handwriting input display control unit 23.

[0166] S4.8.1.1: The handwriting input display control unit 23 transmits a handwriting recognition / language character string candidate acquisition request to the handwriting recognition control unit 26. The handwriting recognition control unit 26 transmits the handwriting recognition / language character string candidates currently held to the handwriting input display control unit 23.

[0167] S4.8.1.2: The handwriting input display control unit 23 transmits a conversion character string candidate acquisition request to the character string conversion control unit 28. The character string conversion control unit 28 transmits the conversion character string candidates currently held to the handwriting input display control unit 23.

[0168] S4.8.1.3: The handwriting input display control unit 23 transmits a predicted character string candidate acquisition request to the predictive conversion control unit 30. The predictive conversion control unit 30 transmits the predicted character string candidates currently held to the handwriting input display control unit 23.

[0169] S4.8.1.4: The handwriting input display control unit 23 transmits the operation command candidate acquisition to the operation command recognition control unit 32. The operation command recognition control unit 32 transmits the operation command candidates and the selected object currently held to the handwriting input display control unit 23.

[0170] S4.8.1.5: Furthermore, the handwriting input display control unit 23 transmits the estimated writing direction acquisition to the handwriting input storage unit 25. The handwriting input storage unit 25 judges the estimated writing direction from the stroke addition time and the horizontal and vertical distances of the handwritten object rectangular area, and transmits the estimated writing direction to the handwriting input display control unit 23.

[0171] S4.8.1.6: The handwriting input display control unit 23 creates selectable candidate display data as shown in Figure 14 from these handwriting recognition string candidates ('gi' in Figure 14), language string candidates (for example, 'gi', although not shown in Figure 14), converted string candidates ('minutes', 'skill test' in Figure 14), predicted string candidates ('Settle the skill test', 'Destination for sending minutes' in Figure 14), operation command candidates ('Load minutes template', 'Save in minutes folder' in Figure 14), each selection probability, and estimated writing direction, and displays the selectable candidate display data as shown in Figure 14 from the string candidates and operation command candidates, by sending it to the display unit 22.

[0172] S4.8.1.7: The handwriting input display control unit 23 also transmits rectangular area display data (rectangular frames) of the handwritten object and the selected object (handwritten object rectangular area display 503 in FIG. 14) to the display unit 22.

[0173] S4.8.1.8: The handwriting input display control unit 23 transmits a selectable candidate display deletion timer start to the candidate display timer control unit 24 in order to delete the selectable candidate display data after a certain time has elapsed since its display. The candidate display timer control unit 24 starts this timer.

[0174] Steps S6 to S6.3 are executed when the user erases the selectable candidate display displayed on the display unit 22 while the selectable candidate deletion timer is started, or when a change occurs in the handwritten object (i.e., a stroke is added, deleted, moved, deformed or divided in the handwritten object), or when no candidate is selected before the timer times out.

[0175] Furthermore, steps S6 to S6.1 are executed when the candidate display is erased or a change occurs in the handwritten object.

[0176] S6: The handwriting input unit 21 transmits to the handwriting input display control unit 23 a notice of the deletion of the selectable candidate display or the occurrence of a change in the handwritten object.

[0177] S6.1: The handwriting input display control unit 23 transmits a selectable candidate deletion timer stop command. The candidate display timer control unit 24 stops the timer. This is because an operation has been performed on the handwritten object within a certain period of time, making the timer unnecessary.

[0178] S6.2: The handwriting input display control unit 23 transmits a command to erase the selectable candidate display data to the display unit 22, thereby erasing the display.

[0179] S6.3: The handwriting input display control unit 23 erases the display by transmitting a command to erase rectangular area display data of the handwritten object and the selected object to the display unit 22. Therefore, when the display of the operation command candidate is erased under a condition other than the selection of the operation command candidate, the handwritten object remains displayed as it is.

[0180] 4.8.1.8.1: On the other hand, if the selectable candidate display is not deleted or the handwritten object is not changed while the selectable candidate deletion timer is started (if the user does not perform a pen operation), the candidate display timer control unit 24 sends 4.8.1.8.1: Timeout to the handwriting input display control unit 23.

[0181] Similarly, after the selectable candidate display deletion timer times out, the handwriting input display control unit 23 executes steps S6.2 and S6.3. This is because the selectable candidate display data and the rectangular area display data of the handwritten object and the selected object may be deleted after a certain period of time has elapsed.

[0182] If the user selects a selectable candidate while the selectable candidate deletion timer is in progress, steps S7 to S7.14 are executed.

[0183] S7: When the user selects a selectable candidate while the selectable candidate deletion timer is starting, the handwriting input unit 21 transmits the character string candidate or the operation command candidate selection to the handwriting input display control unit 23.

[0184] S7.1: The handwriting input display control unit 23 transmits a selectable candidate display deletion timer stop command to the candidate display timer control unit 24. The candidate display timer control unit 24 stops this timer.

[0185] S7.2: Next, the handwriting input / display control unit 23 transmits a command to clear the stored data to the handwriting recognition control unit 26.

[0186] S7.2.1: The handwriting recognition control unit 26 sends a retained data clear command to the character string conversion control unit 28.

[0187] S7.2.2: The handwriting recognition control unit 26 transmits a command to clear the stored data to the predictive conversion control unit 30.

[0188] S7.2.3: The handwriting recognition control unit 26 transmits a command to clear the stored data to the operation command recognition control unit 32.

[0189] The handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 clear the data relating to character string candidates and operation command candidates that have been held up to that point.

[0190] S7.3: Next, the handwriting input display control unit 23 transmits a command to erase the selectable candidate display data to the display unit 22, thereby erasing the display.

[0191] S7.4: The handwriting input display control unit 23 transmits a command to erase the rectangular area display data of the handwritten object and the selected object to the display unit 22, thereby erasing the display.

[0192] S7.5: The handwriting input display control unit 23 erases the display by transmitting a command to erase the handwritten object display data and the pen coordinate complement display data transmitted in step S3.1 to the display unit 22. This is because the handwritten object, etc. becomes unnecessary since a character string candidate or an operation command candidate has been selected.

[0193] S7.6: The handwritten input display control unit 23 transmits a request to delete the handwritten object to the handwritten input storage unit 25.

[0194] If a character string candidate is selected, steps S7.7 to S7.9 are executed.

[0195] S7.7: If a character string candidate is selected, the handwriting input display control unit 23 transmits a character string object addition to the handwriting input saving unit 25.

[0196] S7.8: Furthermore, the handwriting input display control unit 23 transmits a character string object font acquisition to the handwriting input storage unit 25. The handwriting input storage unit 25 selects a predefined font from the estimated character size of the handwritten object and transmits it to the handwriting input display control unit 23.

[0197] S7.9: Next, the handwriting input display control unit 23 uses the defined font received from the handwriting input storage unit 25 to transmit character string object display data to the display unit 22 to be displayed in the same position as the handwritten object, thereby displaying the character string object.

[0198] When a candidate operation command is selected, steps S7.10 to S7.13 are executed, and further, when there is a selected object, steps S7.10 to S7.12 are executed.

[0199] S7.10: When a candidate for an operation command for a selected object is selected (when a selected object exists), the handwriting input display control unit 23 erases the display by transmitting a command to erase the selected object display data to the display unit 22. This is to erase the original selected object once.

[0200] S7.11: Next, the handwriting input display control unit 23 transmits an operation command execution for the selected object to the handwriting input storage unit 25. The handwriting input storage unit 25 transmits display data of the newly selected object (display data after editing or modification) to the handwriting input display control unit 23.

[0201] S7.12: Next, the handwriting input display control unit 23 transmits selected object display data to the display unit 22 to re-display the selected object after the operation command is executed.

[0202] If no object is selected (if an input / output operation command is selected), step S7.13 is executed.

[0203] S7.13: When an input / output operation command is selected, the handwriting input display control unit 23 executes the operation command character string (Command) of the operation command definition data corresponding to the operation command selected by the user. If the user authentication unit 34 succeeds in authenticating the user, the handwriting input display control unit 23 sets information about the successfully authenticated user to %~% of the operation command and executes it.

[0204] S7.14: For the next handwritten object, the handwritten input display control unit 23 transmits a handwritten object start to the handwritten input saving unit 25. The handwritten input saving unit 25 secures a handwritten object area. After this, the processes of steps S2 to S7.14 are repeated.

[0205] <Summary> As described above, the handwriting input device 2 of this embodiment can display candidates for any operation command by the user handwriting a handwritten object or encircling a confirmed object with a surrounding line. Therefore, any function (edit function, input / output function, pen function, etc.) can be called from the handwritten state. This makes it possible to eliminate the need for a step-by-step operation of pressing a menu button to call a desired function, and reduces the number of operation steps required from the handwriting state to calling an arbitrary function. EXAMPLES

[0206] In this embodiment, a handwriting input system in a system form in which an information processing system on a network performs processing such as handwriting recognition and returns the processing result to a handwriting input device 2 will be described.

[0207] In the description of this embodiment, components or contents of figures having the same symbols as those in the first embodiment perform similar functions, so descriptions of components that have already been described may be omitted or only the differences may be described.

[0208] 24 is an example of a system configuration diagram of a handwriting input system 100. The handwriting input system 100 has a handwriting input device 2 and an information processing system 10 that can communicate with each other via a network N.

[0209] The handwriting input device 2 is installed in a facility such as a company, and is connected to a LAN or Wi-Fi installed in the facility. The information processing system 10 is installed in, for example, a data center. The handwriting input device 2 is connected to the Internet i via a firewall 8, and the information processing system 10 is also connected to the Internet i via a high-speed LAN in the data center.

[0210] The handwriting input device 2 may be connected to the Internet i using wireless communication such as a telephone line network. In this case, the wireless communication may be 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), or the like.

[0211] The information processing system 10 has one or more information processing devices, which act as servers to provide services to the handwriting input device 2. A server is a computer or software that provides information and processing results in response to requests from clients. As will be described later, the information processing system 10 receives pen coordinates from the handwriting input device 2 and transmits information required to display the operation guide 500 shown in FIG. 14 to the handwriting input device 2.

[0212] The server-side system is sometimes called a cloud system. A cloud system is a system that uses cloud computing, which is a form of usage in which resources on a network are used without being aware of specific hardware resources. Cloud systems are not necessarily deployed on the Internet. In Figure 24, the information processing system is deployed on the Internet, but it may be deployed on a local network (called on-premise in this case).

[0213] Moreover, in one embodiment, information processing system 10 includes multiple computing devices, such as a server cluster, configured to communicate with each other over any type of communications link, including a network, shared memory, etc., to perform the processes disclosed herein.

[0214] The configuration of the handwriting input device 2 may be the same as that of the first embodiment, but in this embodiment, it is sufficient that the handwriting input device 2 has a touch panel, a display, and a communication function. The handwriting input device 2 may include multiple computing devices configured to communicate with each other.

[0215] In this embodiment, a general information processing device such as a PC or tablet can execute a web browser or a dedicated application. The web browser and the dedicated application communicate with the information processing system 10. When the web browser is operating, the user inputs or selects the URL of the information processing system 10 to connect the handwriting input device to the information processing system 10. The handwriting input device 2 executes a web application provided by the information processing system 10 in the web browser. The web application refers to software or a mechanism thereof that operates by cooperation between a program written in a programming language (e.g., JavaScript (registered trademark)) that runs on the web browser and a program on the web server side, and is executed on the web browser.

[0216] When the dedicated application runs, it connects to a pre-registered URL of the information processing system 10. Since the dedicated application has a program and a user interface, the program transmits and receives necessary information to and from the information processing system 10 and displays it on the user interface.

[0217] The communication method may be a general-purpose communication protocol such as HTTP, HTTPs, or WebSocket, or a dedicated communication protocol.

[0218] <Hardware configuration example> The hardware configuration of the handwriting input device 2 may be the same as that shown in Fig. 5. In this embodiment, an example of the hardware configuration of the information processing system 10 will be described.

[0219] Fig. 25 is a hardware configuration diagram of the information processing system 10. As shown in Fig. 25, the information processing system 10 is constructed by a computer, and includes a CPU 601, a ROM 602, a RAM 603, an HD 604, an HDD (Hard Disk Drive) controller 605, a display 606, an external device connection I / F (Interface) 608, a network I / F 609, a bus line 610, a keyboard 611, a pointing device 612, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 616.

[0220] Among these, the CPU 601 controls the operation of the entire information processing system 10. The ROM 602 stores programs such as an IPL used to drive the CPU 601. The RAM 603 is used as a work area for the CPU 601. The HD 604 stores various data such as programs. The HDD controller 605 controls reading or writing of various data from or to the HD 604 under the control of the CPU 601. The display 606 displays various information such as a cursor, a menu, a window, a character, or an image. The external device connection I / F 608 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 609 is an interface for data communication using a communication network. The bus line 610 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 601 shown in FIG. 25.

[0221] Moreover, the keyboard 611 has a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 612 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 614 controls reading and writing of various data from a DVD-RW 613 as an example of a removable recording medium. Note that the medium is not limited to DVD-RW, and may be a DVD-R or the like. The media I / F 616 controls reading and writing (storing) of data from a recording medium 615 such as a flash memory.

[0222] <About the device functions> Next, functions of the handwriting input system 100 will be described with reference to Fig. 26. Fig. 26 is an example of a functional block diagram showing functions of the handwriting input system 100 in blocks. In the description of Fig. 26, differences from Fig. 6 will be mainly described.

[0223] In this embodiment, the handwriting input device 2 has a display unit 22, a display control unit 41, a handwriting input unit 21, and a communication unit 42. Each function of the handwriting input device 2 is a function or means realized by operating any of the components shown in Fig. 5 by an instruction from the CPU 201 according to a program loaded from the SSD 204 onto the RAM 203.

[0224] The function of the handwriting input unit 21 in this embodiment may be the same as that in embodiment 1. The handwriting input unit 21 converts the user's pen input d1 into pen operation data da (pen up, pen down, or pen coordinate data) and transmits it to the display control unit 41.

[0225] The display control unit 41 controls the display of the handwriting input device 2. First, the display control unit 41 performs an interpolation calculation of coordinates between discrete values ​​of the pen coordinate data, which are discrete values, converts the pen coordinate data from pen-down to pen-up into one stroke db, and transmits the resulting stroke db to the display unit 22.

[0226] In addition, the display control unit 41 transmits the pen operation data dc to the communication unit 42, and acquires various display data dd from the communication unit 42. The display data includes information for displaying the operation guide 500 in FIG.

[0227] The communication unit 42 transmits the pen operation data dc to the information processing system 10 , receives various display data dd from the information processing system 10 , and transmits the same to the display control unit 41 .

[0228] The function of the display unit 22 may be the same as that of the first embodiment. The display unit 22 displays the stroke db and the display data de. The display unit 22 converts the stroke db and the display data de written in the video memory by the display control unit 41 into data according to the characteristics of the display 220, and transmits them to the display 220.

[0229] <<Functions of the information processing system>> The information processing system 10 includes a communication unit 43, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a character string conversion control unit 28, a character string conversion dictionary unit 29, a predictive conversion control unit 30, a predictive conversion dictionary unit 31, an operation command recognition control unit 32, and an operation command definition unit 33. Each function of the information processing system 10 is a function or means realized by any of the components shown in Fig. 25 operating in response to an instruction from the CPU 601 in accordance with a program loaded from the HD 604 onto the RAM 603.

[0230] The communication unit 43 receives pen operation data dc from the handwriting input device 2 , transmits pen operation data df to the handwriting input display control unit 23 , receives display data dd from the handwriting input display control unit 23 and transmits it to the handwriting input device 2 .

[0231] Other functions are the same as those in the first embodiment, or, if different, will not hinder the explanation of this embodiment.

[0232] As for the function of authenticating a user, the information processing system 10 performs authentication in a general manner.

[0233] <Operation procedure> The operation of the handwriting input system 100 will be described with reference to the above configuration and Figs. 27 to 33. Figs. 27 to 33 are sequence diagrams illustrating an example of processing in which the handwriting input device 2 displays character string candidates and operation command candidates. The processing in Fig. 27 starts when the handwriting input device 2 is started (a Web browser or a dedicated application is started) and communication with the information processing system 10 is established. The overall flow in Figs. 27 to 33 may be the same as that in Figs. 18 to 23.

[0234] S1: When communication is established, in order to secure a memory area of ​​the handwriting input device 2, the handwriting input display control unit 23 transmits a handwriting object start to the handwriting input storage unit 25. The handwriting input storage unit 25 secures a handwriting object area (a memory area for storing handwritten objects). The handwriting object area may be secured after the user brings the pen into contact with the handwriting input unit 21.

[0235] S2a: Next, the user touches the pen to the handwriting input unit 21. The handwriting input unit 21 detects pen-down and transmits the result to the display control unit 41.

[0236] S2b: The display control unit 41 transmits a pen-down to the communication unit 42 to notify the information processing system 10 of the pen-down.

[0237] S2c: The communication unit 42 transmits a pen-down signal to the information processing system 10.

[0238] S2d: The communication unit 43 of the information processing system 10 receives the pen-down signal and transmits it to the handwriting input display control unit 23.

[0239] S2.1: The handwritten input display control unit 23 transmits a stroke start to the handwritten input storage unit 25, and the handwritten input storage unit 25 secures a stroke area.

[0240] S3a: When the user moves the pen while keeping it in contact with the handwriting input unit 21, the handwriting input unit 21 transmits the pen coordinates to the display control unit 41.

[0241] S3b: The display control unit 41 transmits the pen coordinates to the communication unit 42 in order to notify the information processing system 10 of the pen coordinates.

[0242] S3c: The communication unit 42 transmits the pen coordinates to the information processing system 10.

[0243] S3d: The communication unit 43 of the information processing system 10 receives the pen coordinates and transmits them to the handwriting input display control unit 23.

[0244] S3.1: The display control unit 41 transmits pen coordinate complement display data (data for interpolating discrete pen coordinates) to the display unit 22. The display unit 22 uses the pen coordinate complement display data to interpolate the pen coordinates and display a line. The process of step S3.2 is the same as in the first embodiment.

[0245] S4a: When the user removes the pen from the handwriting input unit 21, the handwriting input unit 21 transmits a pen-up message to the display control unit 41.

[0246] S4b: The display control unit 41 transmits a pen-up event to the communication unit 42 in order to notify the information processing system 10 of the pen-up event.

[0247] S4c: The communication unit 42 transmits a pen-up signal to the information processing system 10.

[0248] S4d: The communication unit 43 of the information processing system 10 receives the pen-up signal and transmits it to the handwriting input display control unit 23.

[0249] The subsequent steps S4.1 to S4.3.3 and steps S4.6 to S4.7.6.2 in FIG. 29 are similar to those in the first embodiment.

[0250] S5a: If the user touches the pen to the handwriting input unit 21 before the timer times out, the handwriting input unit 21 transmits a pen-down (the same event as in step S2) to the display control unit 41. The processes in steps S5b to S5d may be the same as those in steps S2b to S2d. Also, the processes in steps S5.1 to S4.8.1.5 are the same as those in the first embodiment.

[0251] S4.8.1.6a: The handwriting input display control unit 23 creates selectable candidate display data including each character string candidate, operation command candidate, each selection probability, and estimated writing direction shown in FIG. 14, and transmits the selectable candidate display data consisting of the character string candidates and operation command candidates to the communication unit 43.

[0252] S4.8.1.6b: The communication unit 43 transmits the selectable candidate display data to the handwriting input device 2.

[0253] S4.8.1.6c: The communication unit 42 of the handwriting input device 2 receives the selectable candidate display data and transmits it to the display control unit 41.

[0254] S4.8.1.6d: The display control unit 41 has received the selectable candidate display data, and transmits it to the display unit 22 to cause it to be displayed.

[0255] S4.8.1.7a: The handwriting input display control unit 23 also transmits to the communication unit 43 rectangular area display data (rectangular frames) of the handwritten object and the selected object (handwritten object rectangular area display 503 in FIG. 14).

[0256] S4.8.1.7b: The communication unit 43 transmits the rectangular area display data to the handwriting input device 2.

[0257] S4.8.1.7c: The communication unit 42 of the handwriting input device 2 receives the rectangular area display data and transmits it to the display control unit 41.

[0258] S4.8.1.7d: The display control unit 41 has received the rectangular area display data, and transmits it to the display unit 22 to display it. The process of step S4.8.1.8 is the same as in the first embodiment.

[0259] S6a: When the user erases a selectable candidate or adds a handwritten character to the handwritten object, the handwriting input unit 21 transmits to the display control unit 41 a notification that the selectable candidate display has been erased or that the handwritten object has been changed.

[0260] S6b: The display control unit 41 transmits a notification to the communication unit 42 to notify the information processing system 10 of the occurrence of the deletion of the selectable candidate display or the change in the handwritten object.

[0261] S6c: The communication unit 42 transmits to the information processing system 10 the occurrence of the deletion of the selectable candidate display or the change in the handwritten object.

[0262] S6d: The communication unit 43 of the information processing system 10 receives the notification of deletion of the selectable candidate display or change in the handwritten object, and transmits it to the handwritten input display control unit 23. The processes of steps S6.1 and S4.8.1.8.1 are the same as those in the first embodiment.

[0263] S6.2a: The handwriting input display control unit 23 transmits a command to the communication unit 43 to erase the selectable candidate display data.

[0264] S6.2b: The communication unit 43 transmits a command to the handwriting input device 2 to erase the selectable candidate display data.

[0265] S6.2c: The communication unit 42 of the handwriting input device 2 receives the command to erase the selectable candidate display data, and transmits it to the display control unit 41.

[0266] S6.2d: The display control unit 41 has received the command to erase the selectable candidate display data, and transmits the command to the display unit 22 to erase the selectable candidates.

[0267] S6.3a: The handwriting input display control unit 23 transmits to the communication unit 43 a command to erase the rectangular area display data of the handwritten object and the selected object.

[0268] S6.3b: The communication unit 43 transmits a command to the handwriting input device 2 to erase the rectangular area display data of the handwritten object and the selected object.

[0269] S6.3c: The communication unit 42 of the handwriting input device 2 receives the rectangular area display data deletion command for the handwritten object and the selected object, and transmits the command to the display control unit 41.

[0270] S6.3d: Having received the command to erase the rectangular area display data of the handwritten object and the selected object, the display control unit 41 erases the rectangular areas of the handwritten object and the selected object by transmitting the command to the display unit 22. Therefore, when the display of the operation command candidate is erased under a condition other than the selection of the operation command candidate, the handwritten object remains displayed as it is.

[0271] If the user selects a selectable candidate while the selectable candidate deletion timer is in progress, steps S7 to S7.14 are executed.

[0272] S7a: When the user selects a selectable candidate while the selectable candidate deletion timer is starting, the handwriting input unit 21 transmits the character string candidate or the operation command candidate selection to the display control unit 41.

[0273] S7b: The display control unit 41 transmits the character string candidate or the operation command candidate selection to the communication unit 42 in order to notify the information processing system 10 of the selection.

[0274] S7c: The communication unit 42 transmits the character string candidate or the selected operation command candidate to the information processing system 10.

[0275] S7d: The communication unit 43 of the information processing system 10 receives the character string candidate or the selection of the operation command candidate, and transmits it to the handwriting input display control unit 23. The processes of steps S7.1 to S7.2.3 are the same as those in the first embodiment.

[0276] S7.3a: Next, the handwriting input display control unit 23 transmits a command to the communication unit 43 to erase the selectable candidate display data.

[0277] S7.3b: The communication unit 43 transmits a command to the handwriting input device 2 to erase the selectable candidate display data.

[0278] S7.3c: The communication unit 42 of the handwriting input device 2 receives the command to erase the selectable candidate display data, and transmits it to the display control unit 41.

[0279] S7.3d: Since the display control unit 41 has received the command to erase the selectable candidate display data, it causes the display unit 22 to erase the selectable candidates.

[0280] S7.4a: The handwriting input display control unit 23 transmits to the communication unit 43 a command to erase the rectangular area display data of the handwritten object and the selected object.

[0281] S7.4b: The communication unit 43 transmits a command to erase the rectangular area display data to the handwriting input device 2.

[0282] S7.4c: The communication unit 42 of the handwriting input device 2 receives the rectangular area display data deletion command and transmits it to the display control unit 41.

[0283] S7.4d: Since the display control unit 41 has received the command to erase the rectangular area display data, it causes the display unit 22 to erase the rectangular area.

[0284] S7.5a: The handwriting input display control unit 23 transmits a command to the communication unit 43 to erase the handwriting object display data.

[0285] S7.5b: The communication unit 43 transmits a command to erase the handwriting object display data to the handwriting input device 2.

[0286] S7.5c: The communication unit 42 of the handwriting input device 2 receives the command to erase the handwritten object display data, and transmits it to the display control unit 41.

[0287] S7.5d: Since the display control unit 41 has received the command to erase the handwritten object display data, it causes the display unit 22 to erase the handwritten object and the pen coordinate complement display data. The process of step S7.6 may be the same as in the first embodiment.

[0288] When a character string candidate is selected, steps S7.7 to S7.9 are executed. The processes in steps S7.7 and S7.8 may be the same as those in the first embodiment.

[0289] S7.9a: Next, the handwritten input display control unit 23 uses the defined font received from the handwritten input storage unit 25 to transmit to the communication unit 43 character string object display data to be displayed in the same position as the handwritten object.

[0290] S7.9b: The communication unit 43 transmits the character string object display data to the handwriting input device 2.

[0291] S7.9c: The communication unit 42 of the handwriting input device 2 receives the character string object display data, and transmits it to the display control unit 41.

[0292] S7.9d: The display control unit 41 has received the character string object display data, and thus causes the display unit 22 to display the character string object.

[0293] When a candidate operation command is selected, steps S7.10 to S7.13 are executed, and further, when there is a selected object, steps S7.10 to S7.12 are executed.

[0294] S7.10a: When a candidate for an operation command for a selected object is selected (when a selected object exists), the handwriting input display control unit 23 transmits a command to erase the display data of the selected object to the communication unit 43. This is to erase the original selected object once.

[0295] S7.10b: The communication unit 43 transmits a command to erase the display data of the selected object to the handwriting input device 2.

[0296] S7.10c: The communication unit 42 of the handwriting input device 2 receives the command to erase the selected object display data, and transmits it to the display control unit 41.

[0297] S7.10d: The display control unit 41, having received the command to erase the selected object display data, causes the display unit 22 to erase the selected object.

[0298] S7.11: Next, the handwriting input display control unit 23 transmits an operation command execution for the selected object to the handwriting input storage unit 25. The handwriting input storage unit 25 transmits display data of the newly selected object (display data after editing or modification) to the handwriting input display control unit 23.

[0299] S7.12a: Next, the handwriting input display control unit 23 transmits the selected object display data to the communication unit 43.

[0300] S7.12b: The communication unit 43 transmits the selected object display data to the handwriting input device 2.

[0301] S7.12c: The communication unit 42 of the handwriting input device 2 receives the selected object display data, and transmits it to the display control unit 41.

[0302] S7.12d: Since the display control unit 41 has received the selected object display data, it causes the display unit 22 to re-display the selected object after the operation command is executed. The processes in steps S7.13 and S7.14 may be similar to those in the first embodiment.

[0303] In this way, even in a system configuration in which the handwriting input device 2 and the information processing system 10 communicate with each other, the same effects as those of the first embodiment can be achieved. Note that the process flows in Figs. 27 to 33 are merely examples, and processes resulting from communication between the handwriting input device 2 and the information processing system 10 may be included, or processes may be omitted. In addition, the handwriting input device 2 may perform part of the processes performed by the information processing system 10. For example, the handwriting input device 2 may perform processes related to deletion.

[0304] <Example of operation guide display> The operation guide 500 in the case of the handwriting recognition system may be similar to that shown in Fig. 14. Of course, it is possible to change the design, etc. In this embodiment, an example of the operation guide displayed when English is handwritten will be described.

[0305] FIG. 34 shows an example of the operation guide 500. When the information processing system 10 recognizes a foreign language such as English, it is desirable that even the type of language can be recognized. However, in order to improve the recognition accuracy, there is some way to identify the type of language handwritten by the user. For example, the user selects the type of language from a menu displayed by the handwriting input device 2 and notifies the information processing system 10. Alternatively, the user may register user attributes including the locale (language used) in the information processing system 10, and the type of language may be identified by the user logging in. Alternatively, if the information processing system 10 is on the Internet, the country or region may be determined by the IP address of the handwriting input device 2, and the information processing system 10 may determine the language used in the country or region.

[0306] (1) In Figure 34, for example, the user has handwritten "pl (Pee-er)."

[0307] (2) The handwriting recognition control unit 26, the handwriting recognition dictionary unit 27, the character string conversion control unit 28, the character string conversion dictionary unit 29, the predictive conversion control unit 30, and the predictive conversion dictionary unit 31 are provided for English, and the handwriting recognition control unit 26 recognizes "pl" as an English word. The information processing system 10 recognizes the character code "pl" and transmits a converted character string candidate 507 that includes "pl" as a part thereof and a predicted character string candidate 508 that includes the converted character string to the handwriting input device 2. This enables the handwriting input device 2 to display words and phrases (selectable candidates) that start with "pl", such as "play", "place", "placard", "placenta", and "play ground". Although omitted in FIG. 34, an operation command may also be displayed.

[0308] (3) When the user selects "play," the operation guide 500 and the handwritten object 504 are removed, and "play" is displayed.

[0309] <Summary> Therefore, in this embodiment, handwriting recognition can be performed in the handwriting input system 100 in which the handwriting input device 2 and the information processing system 10 communicate with each other in the same manner as in the first embodiment.

[0310] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0311] For example, although the present embodiment has been described as an example of an electronic whiteboard, the present invention can be suitably applied to any information processing device having a touch panel. A device having the same function as an electronic whiteboard is also called an electronic whiteboard, an electronic information board, an interactive board, etc. Examples of information processing devices equipped with a touch panel include output devices such as PJ (Projector), digital signage, HUD (Head Up Display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, network home appliances, notebook PCs (Personal Computers), mobile phones, smartphones, tablet terminals, game consoles, PDAs (Personal Digital Assistants), digital cameras, wearable PCs, desktop PCs, etc.

[0312] In this embodiment, the coordinates of the pen are detected by detecting the coordinates of the pen tip with a touch panel, but the coordinates of the pen tip may be detected by ultrasonic waves. The pen emits light and also emits ultrasonic waves, and the handwriting input device 2 calculates the distance based on the arrival time of the ultrasonic waves. The position of the pen can be identified based on the direction and distance. The projector draws (projects) the trajectory of the pen as a stroke.

[0313] In the present embodiment, when a selected object is present, candidates for editing and modification operation commands are displayed, and when no selected object is present, candidates for input / output operation commands are displayed. However, candidates for editing and modification operation commands and candidates for input / output operation commands may be displayed simultaneously.

[0314] In addition, the configuration examples in Fig. 6, Fig. 26, etc. are divided according to main functions in order to facilitate understanding of the processing by the handwriting input device 2. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing by the handwriting input device 2 can be divided into more processing units according to the processing contents. Also, it can be divided so that one processing unit includes more processes.

[0315] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above. [Explanation of symbols]

[0316] 2 Handwriting input device 21 Handwriting input section 22 Display section 23 Handwriting input display control unit 24 Candidate display timer control unit 25 Handwriting input storage section 26 Handwriting recognition control section 27 Handwriting Recognition Dictionary Section 28 Character string conversion control section 29 String Conversion Dictionary 30 Predictive conversion control section 31 Predictive conversion dictionary section 32 Operation command recognition control section 33 Operation command definition section 34 User authentication section 35 Authentication information acquisition section [Prior art documents] [Patent documents]

[0317] [Patent Document 1] JP 2018-026185 A

Claims

1. An information processing device capable of communicating with an input device via a network, a communication unit that receives data relating to handwritten input accepted by the input device, the data being displayed on a display of the input device as a handwritten object; and a handwriting recognition control unit that recognizes the data related to the handwriting input received by the communication unit as a character string candidate; an operation command recognition control unit that acquires operation command candidates identified based on the character string candidates recognized by the handwriting recognition control unit, The communication unit is When no change in the handwritten object displayed on the display occurs for a first period of time, the input device is configured to selectably display the character string candidates and the operation command candidates on the display. transmitting to the input device the character string candidates recognized by the handwriting recognition control unit and the operation command candidates identified based on the character string candidates recognized by the handwriting recognition control unit; 23. An information processing apparatus comprising:

2. The communication unit, transmitting the character string candidates and the operation command candidates to the input device when no change has occurred in the handwritten object displayed on the display for the first period of time; 2. The information processing device according to claim 1.

3. when the operation command candidate is not selected on the input device for a second period of time, when an operation to erase the display of the operation command candidate is received, or when a change occurs in the handwritten object, erasing the displayed operation command candidate; 2 . The information processing apparatus according to claim 1 , wherein when the display of the operation command candidates is erased under a condition other than that of the operation command candidate being selected, the display of the handwritten object is maintained.

4. when the operation command candidates and the character string candidates are not selected by the input device during the second time period, or when an operation to erase the display of the operation command candidates and the character string candidates is received, erasing the display of the operation command candidates and the character string candidates; 4. The information processing apparatus according to claim 3, wherein when the display of the operation command candidate and the character string candidate is erased under conditions other than the selection of the operation command candidate and the character string candidate, the display of the handwritten object is maintained.

5. the character string candidates are letters, numbers, symbols, or graphics, The information processing apparatus according to claim 4 , wherein the communication unit transmits the operation command candidates to the input device together with characters, numbers, symbols, or figures.

6. storing a constant for replacing a variable associated with the operation command; replacing a variable associated with the operation command, which is identified based on the character string candidate recognized by the handwriting recognition control unit, with the constant; 6. The information processing apparatus according to claim 1, wherein the operation command is executed in which the variable is replaced by the constant.

7. A user authentication means for authenticating a user is provided; If the user authentication means succeeds in authentication, it sets information about the successfully authenticated user to the constant; replacing a variable associated with the operation command, which is identified based on the character string candidate recognized by the handwriting recognition control unit, with information about the user; 7. The information processing apparatus according to claim 6, wherein the operation command is executed with the variables replaced with information about the user.

8. The information processing device according to any one of claims 1 to 7, characterized in that when the position information of a confirmed object that has already been handwritten is in a predetermined relationship with the handwritten object, the operation command recognition control unit recognizes the operation command related to the confirmed object that is in a predetermined relationship with the handwritten object.

9. The information processing device according to claim 8, characterized in that the operation command recognition control unit recognizes the operation command that edits or modifies the confirmed object that has a predetermined relationship with the handwritten object.

10. When the position information of each of a plurality of determined objects already handwritten has a predetermined relationship with one of the handwritten objects, 10. The information processing apparatus according to claim 8, wherein the operation command recognition control unit recognizes the operation commands related to a plurality of the determined objects having a predetermined relationship with one of the handwritten objects.

11. The information processing device according to any one of claims 8 to 10, characterized in that when an already handwritten confirmed object and the handwritten object overlap with each other at a predetermined overlap rate or greater, the operation command recognition control unit determines that the position information of the already handwritten confirmed object and the handwritten object are in the predetermined relationship.

12. 4. The information processing device according to claim 3, wherein the condition other than that for the operation command candidate to be selected is when an operation for erasing the display of the operation command candidate is received or when a change occurs in the handwritten object.

13. A display method performed by an information processing device capable of communicating with an input device via a network, comprising: a communication process for receiving from the input device data relating to handwritten input accepted by the input device, the data being displayed on a display as a handwritten object by the input device; a handwriting recognition control process for recognizing the data related to the handwriting input received through the communication process as a character string candidate; an operation command recognition control process for acquiring operation command candidates identified based on the character string candidates recognized by the handwriting recognition control process; The communication process includes: When no change in the handwritten object displayed on the display occurs for a first period of time, the input device is configured to selectably display the character string candidates and the operation command candidates on the display. transmitting to the input device the character string candidates recognized by the handwriting recognition control process and the operation command candidates identified based on the character string candidates recognized by the handwriting recognition control process; A display method comprising:

14. An information processing device capable of communicating with an input device via a network, a communication unit that receives data relating to handwritten input accepted by the input device, the data being displayed on a display of the input device as a handwritten object; and a handwriting recognition control unit that recognizes the data related to the handwriting input received by the communication unit as a character string candidate; and acquiring operation command candidates identified based on the character string candidates recognized by the handwriting recognition control unit; The communication unit is When no change in the handwritten object displayed on the display occurs for a first period of time, the input device is configured to selectably display the character string candidates and the operation command candidates on the display. transmitting to the input device the character string candidates recognized by the handwriting recognition control unit and the operation command candidates identified based on the character string candidates recognized by the handwriting recognition control unit; A program characterized by:

15. An input system capable of communicating between an input device and an information processing device via a network, comprising: The input device is a handwriting input unit that accepts handwritten input; a first communication unit that transmits data related to the handwritten input accepted by the handwritten input unit to the information processing device; a display control unit that displays the data related to the handwritten input accepted by the handwriting input unit on a display as a handwritten object, The information processing device includes: a second communication unit that receives data related to the handwritten input transmitted from the first communication unit; a handwriting recognition control unit that recognizes the data related to the handwriting input received by the second communication unit as a character string candidate; an operation command recognition control unit that acquires operation command candidates identified based on the character string candidates recognized by the handwriting recognition control unit, The second communication unit is When no change in the handwritten object displayed on the display occurs for a first period of time, the input device is configured to selectably display the character string candidates and the operation command candidates on the display. transmitting to the input device the character string candidates recognized by the handwriting recognition control unit and the operation command candidates identified based on the character string candidates recognized by the handwriting recognition control unit; 1. An input system comprising:

Images