Input device, input method, program, input system

Abstract

To provide an input device that improves the usability of handwriting input. [Solution] The present invention provides an input device 2 comprising: a handwriting input unit that accepts handwriting input using the position of a pen or the user's finger in contact with the display; and a display unit that displays the handwriting input accepted by the handwriting input unit as a handwritten object on the display, wherein if no change occurs in the handwritten object during a first period of time, the device displays one or more operation commands based on the handwritten object.

Description

Technical Field

[0006] , , , , , ,

[0005]

[0001] The present invention relates to an input device, an input method, a program, and an input system.

Background Art

[0003] Patent Document 1 discloses an input device in which a menu for color setting, transparency setting, thickness setting, line type setting, stamp setting, and operation setting is displayed by pressing a pen button.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there has been a problem that the operability of handwriting input may not be good in a conventional input device. For example, every time the user tries to edit or modify a handwritten object, it is necessary to open the pen function menu or the editing function menu. After opening, it is further necessary to select a desired setting or function (color, thickness, line type, etc.).

[0005] An object of the present invention is to provide an input device with improved operability of handwriting input in view of the above problems.

Means for Solving the Problems

[0006] In view of the above problems, the present invention provides an input device comprising: a handwriting input unit that receives handwriting input using the position of a pen or the user's finger in contact with a display; and a display unit that displays the handwriting input received by the handwriting input unit as a handwritten object on the display, wherein if no change occurs in the handwritten object during a first period of time, the device displays one or more operation commands based on the handwritten object. [Effects of the Invention]

[0007] This allows us to provide an input device that improves the usability of handwriting input. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram shows the operation menu displayed by a conventional handwriting input device. [Figure 2] This figure shows an example of a multi-functional pen. [Figure 3] This is an example diagram illustrating the display of suggested operation commands shown by a handwriting input device. [Figure 4] This is an overall configuration diagram of an example of a handwriting input device. [Figure 5] This is a hardware configuration diagram of an example of a handwriting input device. [Figure 6] This is an example of a functional block diagram showing the functions of a handwriting input device in a block-like structure. [Figure 7] This is an example of a functional block diagram showing the user authentication functions of a handwriting input device in a block-like structure. [Figure 8] This figure shows an example of predefined control data. [Figure 9] This figure shows an example of dictionary data for the handwriting recognition dictionary section. [Figure 10] This figure shows an example of dictionary data for the string conversion dictionary section. [Figure 11] This figure shows an example of dictionary data for the predictive text dictionary section. [Figure 12]This figure shows an example of operation command definition data and system definition data held by the operation command definition unit. [Figure 13] This figure shows an example of operation command definition data when a selected object is selected by a hand-drawn object. [Figure 14] This figure shows an example of displaying candidate operation commands based on operation command definition data when no object is selected. [Figure 15] This is an example of a diagram illustrating how to specify selected objects. [Figure 16] This figure shows an example of displaying candidate operation commands based on operation command definition data when there are handwritten objects. [Figure 17] This figure shows an example of displaying candidate operation commands based on operation command definition data when there are handwritten objects. [Figure 18] This is a sequence diagram (part 1) illustrating an example of the process by which a handwriting input device displays string candidates and operation command candidates. [Figure 19] This is a sequence diagram (part 2) illustrating an example of the process by which a handwriting input device displays string candidates and operation command candidates. [Figure 20] This is a sequence diagram (part 3) illustrating an example of the process by which a handwriting input device displays string candidates and operation command candidates. [Figure 21] This is a sequence diagram (part 4) illustrating an example of the process by which a handwriting input device displays string candidates and operation command candidates. [Figure 22] This is a sequence diagram (part 5) illustrating an example of the process by which a handwriting input device displays string candidates and operation command candidates. [Figure 23] This is a sequence diagram (part 6) illustrating an example of the process by which a handwriting input device displays string candidates and operation command candidates. [Figure 24] This is an example of a system configuration diagram for a handwriting input system (Example 2). [Figure 25] This is an example of a hardware configuration diagram for an information processing system. [Figure 26] It is an example of a functional block diagram showing the functions of a handwriting input system in a block form. [Figure 27] It is an example of a sequence diagram (Part 1) for explaining the process in which a handwriting input device displays candidates for character strings and operation commands. [Figure 28] It is an example of a sequence diagram (Part 2) for explaining the process in which a handwriting input device displays candidates for character strings and operation commands. [Figure 29] It is an example of a sequence diagram (Part 3) for explaining the process in which a handwriting input device displays candidates for character strings and operation commands. [Figure 30] It is an example of a sequence diagram (Part 4) for explaining the process in which a handwriting input device displays candidates for character strings and operation commands. [Figure 31] It is an example of a sequence diagram (Part 5) for explaining the process in which a handwriting input device displays candidates for character strings and operation commands. [Figure 32] It is an example of a sequence diagram (Part 6) for explaining the process in which a handwriting input device displays candidates for character strings and operation commands. [Figure 33] It is an example of a sequence diagram (Part 7) for explaining the process in which a handwriting input device displays candidates for character strings and operation commands. [Figure 34] It is a diagram showing an example of an operation guide.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, as an example of an embodiment for carrying out the present invention, a handwriting input device 2 and a handwriting input method performed by the handwriting input device 2 will be described with reference to the drawings.

[0010] <Regarding the operation procedure of a conventional handwriting input device> For the convenience of explaining the handwriting input device of the present embodiment, first, the operation procedure of a conventional handwriting input device will be briefly described.

[0011] Figure 1 shows the operation menu displayed by a conventional handwriting input device. The entire operation panel 101 displays a pen function menu button 1021 to display the pen function menu 102, an editing function menu button 1031 to display the editing function menu, and an input / output function menu button 1041 to display the input / output function menu, etc. In the pen function menu 102, the user can select the pen color, thickness, and operating mode. In the editing function menu 103, the user can select to erase, select, cut, copy, and paste objects. In the input / output function menu 104, the user can load templates, load files, save to a file, and print, etc.

[0012] As an example of the operation procedure for conventional handwriting input devices, if a user is handwriting a straight or curved line (hereinafter referred to as a stroke), the following procedure is required to erase multiple strokes and start writing again with the pen. Unless otherwise specified, "pressing with the pen" means pressing with the pen tip. (A1) Press the editing function menu button 1031 with the pen to display the editing function menu 103. (A2) When the user presses the "Erase" button 1032 on the editing function menu 103, the handwriting input device automatically erases the editing function menu 103. (A3) When a user writes with a pen across multiple strokes that are to be erased, the strokes are erased. (A4) Press the pen function menu button 1021 with the pen.

[0013] This procedure erases the strokes to be deleted, allowing the user to continue writing strokes with the pen. Erasing strokes alone requires four steps. While the number of steps and screen transitions may vary, handwriting input devices often share a similar operating system. This type of operating system is common to computer products in general, and is fundamentally based on a system that allows the user to access the necessary functions. Since users cannot operate the device at all if they don't know how to access the functions, it's common for handwriting input devices to provide tutorial or help functions to show how to access those functions.

[0014] Similarly, when input / output such as file transfers or printing is required, the user opens the input / output function menu 104. For example, the following steps are required to load a template. (B1) When the user presses the input / output function menu button 1041 with the pen, the input / output function menu 104 opens. (B2) When the user presses the "Template Selection" button 1042 in the input / output function menu 104 with the pen, the input / output function menu closes and the template selection window opens. (B3) Use the pen to press the left or right buttons, etc., to scroll until the desired template is displayed. (B4) When you press the desired template with the pen, that template will be loaded and the template selection window will close.

[0015] Thus, conventional handwriting input devices require users to follow step-by-step instructions. In other words, because the operating system of handwriting input devices is based on an explanatory approach that guides the user to the next step, it has been difficult for users to operate them intuitively.

[0016] As shown in Figure 2, it is also being considered to incorporate functions into the pen itself in order to reduce the number of operating steps. Figure 2 shows an example of a multi-functional pen (hereinafter referred to as the active pen 110). The active pen 110 is a pen that has a built-in power supply and can send commands to a handwriting input device. In the active pen 110 of Figure 2, there is one physical switch at the tip of the pen, one at the end of the pen, and two on the side of the pen. The tip is for writing, the end of the pen is for erasing, and the side of the pen is for assigning user functions.

[0017] Users can assign one of the pen's side switches to the pen function menu 102 and the other to the editing function menu 103. While this makes it convenient for users to display the pen function menu 102 or editing function menu 103 by pressing the side button on the active pen 110 they are holding, it does not change the cumbersome process of having to press the side button to open the pen function menu 102 or editing function menu 103 every time the object being manipulated is changed.

[0018] Furthermore, using the erase switch at the end of the pen eliminates the need for steps (A1), (A2), and (A4) above, and in step (A3), the end of the pen can be used instead of the pen tip, shortening the erasing procedure to one step. However, this creates another inconvenience: the purchase and operation of a special pen called the Active Pen 110 incurs costs. In addition, the Active Pen 110 requires battery replacement or charging, and depending on usage, measures to prevent loss may also be necessary. [Examples]

[0019] <Overview> Next, the outline of the handwriting input device 2 of this embodiment will be described using Figure 3. Figure 3 is an example of a diagram illustrating an example of the display of candidate operation commands (also called candidate operation commands) displayed by the handwriting input device 2. Figure 3(a) shows candidate operation commands displayed when the user handwrites characters, and Figure 3(b) shows candidate operation commands displayed when a border is handwritten. First, Figure 3(a) will be explained.

[0020] A. When the user writes the handwritten object 11 (here, a character) "gi" with a pen, the handwriting input device 2 performs character recognition and converts it to "gi".

[0021] B. The handwriting input device 2 further converts "gi" into a Chinese character. In Fig. 3(a), as an example, it is converted to "議" (meaning "discussion"), but any Chinese character in the dictionary can be converted.

[0022] C. The handwriting input device 2 further converts "議" to a predicted string predicted from "議". In Fig. 3(a), as an example, it is converted to "議事録" (meaning "minutes"), but any string in the dictionary can be converted.

[0023] D. The handwriting input device 2 displays candidates for operation commands including "gi", "議", or "議事録". That is, as operation command candidates associated with "gi", "議", or "議事録", candidates 510 for operation commands such as "Load minutes template" and "Save to minutes folder" related to input / output functions are displayed. The candidates for operation commands are not limited to two. When a candidate 510 for an operation command is selected, the handwriting input device 2 executes file input / output according to the selected operation command.

[0024] Therefore, as a user, by writing "gi", it is possible to display candidates 510 for operation commands such as "Load minutes template" or "Save to minutes folder", and from the state of writing, the input / output functions shown in Fig. 1 can be called.

[0025] Next, Fig. 3(b) will be described.

[0026] E. The user surrounds the already determined object 13 "議事録" (minutes) with a handwritten object 11 (here, a bounding box).

[0027] F. The handwriting input device 2 determines whether the handwritten object 11 and the confirmed object 13 meet predetermined criteria (for example, whether they overlap or not).

[0028] G. If the handwritten object 11 and the finalized object 13 meet the predetermined criteria, the handwriting input device 2 displays candidate 510 of editing or modification operation commands. In Figure 3(b), candidate 510 of operation commands related to editing functions, namely "erase," "move," "rotate," and "select," are displayed. The candidate 510 of operation commands is not limited to these four.

[0029] Therefore, by enclosing the confirmed object 13 with a border, the user can display candidate command 510 for modifying the confirmed object 13, and can call up the editing function while handwriting.

[0030] Thus, the handwriting input device 2 of this embodiment allows the user to invoke input / output functions or editing functions, etc., while writing by hand. The pen function can be invoked in the same way. Therefore, the number of steps required for the user to invoke any function from the state of writing by hand can be reduced.

[0031] <About Terminology> The input method can be any means that allows handwriting by specifying coordinates on a touch panel. Examples include a pen, a person's finger or hand, or a rod-shaped object. Eye-tracking input is also possible.

[0032] A stroke is defined as a series of operations in which a user presses an input device against the display, moves it continuously, and then lifts it from the display. Stroke data is information displayed on the screen based on the trajectory of coordinates input by the input device. Stroke data may be interpolated as appropriate.

[0033] The items displayed on the screen based on stroke data are called objects. While "object" generally means a target or subject, in this embodiment it refers to the object to be displayed, etc.

[0034] A handwritten object is an object composed of one or more stroke data, and refers to a display object that a user has handwritten. The handwritten object rectangular area is the area indicated by the bounding rectangle surrounding the handwritten object.

[0035] A confirmed object is a handwritten object that has been converted into a character code (font) through character recognition and selected by the user, or a handwritten object that has been confirmed not to be converted into a character code (font).

[0036] The input device is a device that accepts information input based on the contact position of the input means with respect to the touch panel. In this embodiment, it will be described as the handwriting input device 2.

[0037] An operation command is a command provided for operating the handwriting input device 2 that instructs the execution of a specific process. In this embodiment, editing, modification, or input / output operation commands are used as examples, but all commands for operating the handwriting input device 2, such as screen inversion, page switching, and setting the operating mode, are included.

[0038] <Overall configuration of the device> The overall configuration of the handwriting input device 2 according to an embodiment of the present invention will be explained using Figure 4. Figure 4 is a diagram showing the overall configuration of the handwriting input device 2. Here, an electronic whiteboard is shown as an example of the handwriting input device 2.

[0039] As shown in Figure 4, a display 220, which is an example of a display device, is installed on the top of the handwriting input device 2. Users can input (draw) characters and other information on the display 220 using the pen 2500.

[0040] <Device Hardware Configuration> Next, the hardware configuration of the handwriting input device 2 will be explained using Figure 5. As shown in the figure, the handwriting input device 2 has the configuration of an information processing device or computer. Figure 5 is an example of a hardware configuration diagram of the handwriting input device 2. As shown in Figure 5, the handwriting input device 2 is equipped with a CPU (Central Processing Unit) 201, ROM (Read Only Memory) 202, RAM (Random Access Memory) 203, and SSD (Solid State Drive) 204.

[0041] Of these, the CPU 201 controls the operation of the entire handwriting input device 2. The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and the IPL (Initial Program Loader). The RAM 203 is used as the work area for the CPU 201. The SSD 204 stores various data, such as programs for the handwriting input device 2.

[0042] Furthermore, the handwriting input device 2 includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, and a power switch 222.

[0043] The display controller 213 controls and manages the screen display in order to output the output image to the display 220, etc. The touch sensor 216 detects when the pen 2500 or the user's hand (the pen or the user's hand serves as an input means) touches the display 220. The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 performs coordinate input and coordinate detection. One method of this coordinate input and detection is, for example, in the case of an optical system, two light-emitting and receiving devices installed at both ends of the upper part of the display 220 emit multiple infrared rays parallel to the display 220, and the light is reflected by reflective members provided around the display 220 and received by the light-receiving element as it returns along the same optical path as the light emitted. The touch sensor 216 outputs position information of the infrared rays emitted by the two light-emitting and receiving devices that are blocked by the object to the touch sensor controller 215, and the touch sensor controller 215 identifies the coordinate position, which is the contact position of the object.

[0044] The power switch 222 is a switch for turning the power of the handwriting input device 2 ON / OFF.

[0045] Furthermore, the handwriting input device 2 is equipped with a bus line 210. The bus line 210 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 201 shown in Figure 5.

[0046] The touch sensor 216 is not limited to optical sensors. Various detection methods may be used, such as a capacitive touch panel that identifies the contact position by detecting changes in capacitance, a resistive touch panel that identifies the contact position by voltage changes between two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by contact between an object and the display. The touch sensor 216 does not require an electronic pen to detect whether or not the pen tip is touching the surface. In this case, a fingertip or a pen-shaped stick can be used for touch operation. The pen 2500 does not need to be a long, slender pen shape.

[0047] <About the device's functions> Next, the functions of the handwriting input device 2 will be explained using Figure 6. Figure 6 is an example of a functional block diagram showing the functions of the handwriting input device 2 in a block-like manner. The handwriting input device 2 comprises a handwriting input unit 21, a display unit 22, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a string conversion control unit 28, a string conversion dictionary unit 29, a predictive conversion control unit 30, a predictive conversion dictionary unit 31, an operation command recognition control unit 32, and an operation command definition unit 33. Each function of the handwriting input device 2 is a function or means realized by any of the components shown in Figure 5 operating according to instructions from the CPU 201 in accordance with a program deployed from the SSD 204 onto the RAM 203.

[0048] The handwriting input unit 21 is implemented using a touch sensor 216 or the like, and accepts handwriting input from the user. The handwriting input unit 21 converts the user's pen input d1 into pen operation data d2 (pen up, pen down, or pen coordinate data) and transmits it to the handwriting input display control unit 23. The pen coordinate data is transmitted periodically as discrete values, and the coordinates between the discrete values ​​are interpolated.

[0049] The display unit 22 is implemented by a display 220 or the like, and displays handwritten objects and operation menus. The display unit 22 converts the drawing data d3 written by the handwriting input display control unit 23 to video memory into data according to the characteristics of the display 220, and transmits it to the display 220.

[0050] The handwriting input display control unit 23 performs overall control over handwriting input and display. The handwriting input display control unit 23 processes the pen operation data d2 from the handwriting input unit 21 and transmits it to the display unit 22 for display. Details of the processing of pen operation data d2 and the display of strokes will be explained in Figures 18 to 23 below.

[0051] The candidate display timer control unit 24 is a timer that controls the display of selectable candidates. It generates the timing to start or stop the timer to begin displaying selectable candidates and the timing to clear the display. The candidate display timer control unit 24 receives a timer start request d4 (or a timer stop request) from the handwriting input display control unit 23 and sends a timeout event d5 to the handwriting input display control unit 23.

[0052] The handwriting input storage unit 25 has a storage function for storing user data (handwritten objects / string objects). The handwriting input storage unit 25 receives user data d6-1 from the handwriting input display control unit 23, stores it in the handwriting input storage unit 25, receives an acquisition request d6-2 from the handwriting input display control unit 23, and transmits the user data d7 stored in the handwriting input storage unit 25. The handwriting input storage unit 25 transmits the position information d36 of the confirmed object to the operation command recognition control unit 32.

[0053] The handwriting recognition control unit 26 is a recognition engine that performs online handwriting recognition. Unlike typical OCR (Optical Character Reader), it recognizes characters (not only Japanese but also multiple languages ​​such as English), numbers, symbols (%), $, &, etc., and shapes (lines, circles, triangles, etc.) in parallel with the user's pen input. Various algorithms have been devised for recognition methods, but in this embodiment, known technologies can be used, so details are omitted.

[0054] The handwriting recognition control unit 26 receives pen operation data d8-1 from the handwriting input display control unit 23, performs handwriting recognition, and stores handwriting recognition string candidates. The handwriting recognition control unit 26 also stores language string candidates converted from handwriting recognition string candidates d12 using the handwriting recognition dictionary unit 27. Separately, when it receives an acquisition request d8-2 from the handwriting input display control unit 23, the handwriting recognition control unit 26 transmits the stored handwriting recognition string candidates and language string candidates d9 to the handwriting input display control unit 23.

[0055] The handwriting recognition dictionary unit 27 is dictionary data for language conversion in handwriting recognition. The handwriting recognition dictionary unit 27 receives the handwriting recognition string candidate d12 from the handwriting recognition control unit 26, converts it into a linguistically plausible language string candidate d13, and transmits it to the handwriting recognition control unit 26. For example, in the case of Japanese, it converts hiragana to kanji or katakana.

[0056] The string conversion control unit 28 controls the conversion of conversion string candidates into strings. A conversion string is a string that is highly likely to be generated by including a handwritten recognition string or a language string. The string conversion control unit 28 receives the handwritten recognition string and language string candidate d11 from the handwritten recognition control unit 26, converts them into conversion string candidates using the string conversion dictionary unit 29, and stores them. Separately, if it receives an acquisition request d14 from the handwritten input display control unit 23, it sends the stored conversion string candidate d15 to the handwritten input display control unit 23.

[0057] The string conversion dictionary unit 29 is dictionary data for string conversion. The string conversion dictionary unit 29 receives the handwritten recognition string and language string candidate d17 from the string conversion control unit 28 and transmits the conversion string candidate d18 to the string conversion control unit 28.

[0058] The predictive text control unit 30 receives the handwritten recognition string and language string candidate d10 from the handwritten recognition control unit 26, and the conversion string candidate d16 from the string conversion control unit 28, and converts each of them into predictive text candidates using the predictive text dictionary unit 31. A predictive text candidate is a string that is highly likely to be generated by including the handwritten recognition string, language string, or conversion string. Separately, if an acquisition request d19 is received from the handwritten input display control unit 23, the predictive text candidate d20 is sent to the handwritten input display control unit 23.

[0059] The predictive text dictionary unit 31 contains dictionary data for predictive text conversion. The predictive text dictionary unit 31 receives handwritten recognition strings, language string candidates, and conversion string candidates d21 from the predictive text control unit 30, and transmits the predictive string candidates d22 to the predictive text control unit 30.

[0060] The operation command recognition control unit 32 receives the handwritten recognition string and language string candidate d30 from the handwriting recognition control unit 26, the conversion string candidate d28 from the string conversion control unit 28, and the prediction string candidate d29 from the prediction conversion control unit 30. For each of these, it sends an operation command conversion request d26 to the operation command definition unit 33 and receives the operation command candidate d27 from the operation command definition unit 33. The operation command recognition control unit 32 stores the operation command candidate d27.

[0061] If the operation command definition unit 33 finds that the operation command conversion request d26 partially matches the operation command definition, it sends a candidate operation command d27 to the operation command recognition control unit 32.

[0062] Furthermore, the operation command recognition control unit 32 receives pen operation data d24-1 from the handwriting input display control unit 23, sends a request d23 to the handwriting input storage unit 25 to acquire location information of a previously entered and confirmed confirmed object, and stores the confirmed object specified by the pen operation data as a selected object (including location information). The operation command recognition control unit 32 identifies the location of the pen operation data d24-1 and a selected object that meets predetermined criteria. Separately, if it receives an acquisition request d24-2 from the handwriting input display control unit 23, it sends the held candidate operation command and the identified selected object d25 to the handwriting input display control unit 23.

[0063] <About User Authentication> In this embodiment, control is performed using the results of user authentication, so it is preferable that the handwriting input device 2 has a user authentication function. For this reason, the user authentication function will be explained with reference to Figure 7.

[0064] Figure 7 is a block diagram showing the user authentication functions of the handwriting input device 2. Although Figure 7 only shows the handwriting input display control unit 23 as a function related to the user authentication unit 34 (user authentication means), the user authentication result may utilize the functions shown in Figure 6.

[0065] The authentication information acquisition unit 35 acquires authentication information d31 from the user. Authentication information d31 may be the IC card number, user ID and password, or biometric information such as fingerprints. The user authentication unit 34 acquires authentication information d32 from the authentication information acquisition unit 35 and searches for authentication information d33 in the user information DB 36. If a user matching the search exists, user information d34 is acquired from the user information DB 36. User information may be any information that represents the user's attributes, such as username, password, user's computer name, department, and privileges.

[0066] The user authentication unit 34 transmits user information d35 to the handwriting input display control unit 23, which then uses the user information to execute operation commands. Operation commands using user information are explained in Figure 12.

[0067] The authentication function may be provided by the handwriting input device 2, or by an external authentication server. In this case, the handwriting input device 2 sends authentication information to the authentication server and obtains the authentication result and user information.

[0068] <About predefined control data> Next, using Figure 8, we will explain the predefined control data used by the handwriting input device 2 for various processes. Figure 8 shows an example of predefined control data. In the example in Figure 8, the control data is shown for each control item.

[0069] The selectable candidate display timer 401 defines the time until selectable candidates are displayed (an example of the first time). This is to prevent the display of selectable candidates while handwriting is in progress. In Figure 8, it means that if a pen-down does not occur within TimerValue = 500 [ms] from pen-up, selectable candidates will be displayed. The selectable candidate display timer 401 is held by the candidate display timer control unit 24. The selectable candidate display timer 401 is used when the selectable candidate display timer starts in step S4.8 of Figure 20, which will be described later.

[0070] The selectable candidate clearing timer 402 defines the time until the displayed selectable candidates are cleared (an example of a second time). This is to clear the selectable candidates if the user does not select one. In Figure 8, it means that if a selectable candidate is not selected within TimerValue = 5000 [ms] from the time the selectable candidates are displayed, the selectable candidate display will be cleared. The selectable candidate clearing timer 402 is held by the candidate display timer control unit 24. The selectable candidate clearing timer 402 is used when the selectable candidate display clearing timer starts in step S4.8.1.8 in Figure 21.

[0071] The handwritten object proximity rectangular area 403 defines a rectangular area that is considered to be the vicinity of the handwritten object. In the example in Figure 8, the handwritten object proximity rectangular area 403 is a rectangular area that is expanded horizontally by 50% (Horizontal) of the estimated character size and vertically by 80% (Vertical) of the estimated character size. In the example in Figure 8, it is specified as a percentage of the estimated character size, but it is also possible to make it a fixed length by specifying the unit as "mm", etc. The handwritten object proximity rectangular area 403 is held by the handwritten input storage unit 25. The estimated character size 405 is used in step S4.2 of Figure 19 to determine the overlap between the handwritten object proximity rectangular area and the stroke rectangular area.

[0072] The estimated writing direction / character size determination condition 404 defines constants for determining the measurement direction of writing direction and character size. In the example in Figure 8, if the difference between the time when the first stroke was added and the time when the last stroke was added to the handwritten object rectangular area is MinTime = 1000 [ms] or more, and the difference between the horizontal distance (width) and vertical distance (height) of the handwritten object rectangular area is MinDiff = 10 [mm] or more, and the horizontal distance is longer than the vertical distance, then the estimated writing direction is determined to be "horizontal" and the estimated character size is determined to be the vertical distance. If the horizontal distance is shorter than the vertical distance, then the estimated writing direction is determined to be "vertical" and the estimated character size is determined to be the horizontal distance. If none of the above conditions are met, the estimated writing direction is determined to be "horizontal" (DefaultDir="Horizontal"), and the estimated character size is determined to be the longer of the horizontal distance and the vertical distance. The estimated writing direction / character size determination condition 404 is held by the handwritten input storage unit 25. The estimated writing direction / character size determination condition 404 is used in obtaining the estimated writing direction in step S4.8.1.5 of Figure 21 and in obtaining the font of the string object in step S7.8.

[0073] The estimated character size 405 defines data for estimating the size of characters, etc. In the example in Figure 8, the estimated character size determined by the estimated writing direction / character size determination condition 404 is compared with the smaller character 405a (hereinafter referred to as the minimum font size) and the larger character 405c (hereinafter referred to as the maximum font size) of the estimated character size 405. If the estimated character size is smaller than the minimum font size, the estimated character size is determined to be the minimum font size. If the estimated character size is larger than the maximum font size, the estimated character size is determined to be the maximum font size. Otherwise, it is determined to be the character size of the medium character 405b. The estimated character size 405 is held by the handwriting input storage unit 25. The estimated character size 405 is used for obtaining the string object font in step S7.8 in Figure 23.

[0074] Specifically, the handwriting input storage unit 25 compares the estimated character size determined by the estimated writing direction / character size determination condition 404 with the FontSize of the estimated character size 405 and uses the font of the closest size. For example, if the estimated character size is 25 [mm] (FontSize for small characters) or less, it is determined as "small characters"; if the estimated character size is between 25 mm and 50 mm (FontSize for medium characters), it is determined as "medium characters"; and if the estimated character size is greater than 100 mm (FontSize for large characters), it is determined as "large characters". For "small characters" 405a, a 25 mm Mincho font (FontStyle="Mincho" FontSize="25mm") is used; for "medium characters" 405b, a 50 mm Mincho font (FontStyle="Mincho" FontSize="50mm") is used; and for "large characters" 405c, a 100 mm Gothic font (FontStyle="Gothic" FontSize="100mm") is used. If you want to increase the number of font sizes or styles, you can increase the number of options for estimated font size 405.

[0075] The crossover line determination condition 406 defines the data used to determine whether multiple objects have been selected. If the handwritten object is a single stroke, and in the example in Figure 8, the length of the long side of the handwritten object is 100 mm or more (MinLenLongSide="100mm"), the length of the short side is 50 mm or less (MaxLenShortSide="50mm"), and the overlap rate between the handwritten object in the long and short side directions is 80% or more (MinOverLapRate="80%"), then it is determined that multiple objects have been selected (selected objects). Instead of "or more," it may be determined whether the overlap rate is "greater than or equal to." The crossover line determination condition 406 is held by the operation command recognition control unit 32. The crossover line determination condition 406 is used for crossover line determination in the selection object determination in step S4.7.6.2 of Figure 20.

[0076] The border determination condition 407 defines the data used to determine whether an object is bordered or not. In the example in Figure 8, the operation command recognition control unit 32 determines a confirmed object as a selected object if the overlap rate of the long side and short side of the handwritten object is 100% or more (MinOverLapRate="100%") (greater than or equal to a predetermined overlap rate). It may also determine whether the overlap rate is "greater than or equal to" rather than "greater than or equal to". The border determination condition 407 is held by the operation command recognition control unit 32. The border determination condition 407 is used for border determination in the selection object determination in step S4.6.7.2 of Figure 20.

[0077] Note that the overlap line detection condition 406 and the enclosure line detection condition 407 can be judged in any order. For example, if the overlap line detection condition 406 is made lenient (making it easier to select overlap lines) and the enclosure line detection condition 407 is made strict (setting a value so that only enclosure lines can be selected), the operation command recognition control unit 32 should prioritize the enclosure line detection condition 407.

[0078] <Example of dictionary data> The dictionary data will be explained using Figures 9 to 11. Figure 9 is an example of the dictionary data of the handwriting recognition dictionary unit 27, Figure 10 is an example of the dictionary data of the string conversion dictionary unit 29, and Figure 11 is an example of the dictionary data of the predictive text dictionary unit 31. These dictionary data are used in steps S4.7.2 to S4.7.4 of Figure 20, respectively.

[0079] In this embodiment, the conversion result based on the dictionary data of the handwritten recognition dictionary unit 27 in FIG. 9 is called a language string candidate, the conversion result based on the dictionary data of the string conversion dictionary unit 29 in FIG. 10 is called a converted string candidate, and the conversion result based on the dictionary data of the prediction conversion dictionary unit 31 in FIG. 11 is called a predicted string candidate. For each dictionary data, the "before conversion" is the string for searching the dictionary data, the "after conversion" is the string after conversion corresponding to the searched string, and the "probability" represents the probability selected by the user. The probability is calculated from the results of the user's past selection of each string. Therefore, the probability may be calculated for each user. Although various algorithms have been devised as methods for calculating the probability, it may be calculated by an appropriate method as appropriate, and the details are omitted. In this embodiment, it is characterized in that the string candidates are displayed in descending order of selection probability from the estimated writing direction.

[0080] In the dictionary data of the handwritten recognition dictionary unit 27 in FIG. 9, it shows that the handwritten "ぎ" is converted to "議" with a probability of 0.55 and "技" with a probability of 0.4, and the handwritten "ぎし" is converted to "技士" with a probability of 0.5 and "技師" with a probability of 0.45. The same applies to other "before conversion" strings. In FIG. 9, the "before conversion" string is handwritten hiragana, but other characters than hiragana may be registered as "before conversion".

[0081] In the dictionary data of the string conversion dictionary unit 29 in FIG. 10, it shows that the string "議" is converted to "議事録" with a probability of 0.95 and the string "技" is converted to "技量試" with a probability of 0.85. The same applies to other "before conversion" strings.

[0082] In the dictionary data of the prediction conversion dictionary unit 31 in FIG. 11, it shows that the string "議事録" is converted to "議事録の送付先" with a probability of 0.65 and the string "技量試" is converted to "技量試を決裁" with a probability of 0.75. In the example of FIG. 11, all the strings before conversion are Chinese characters, but other characters than Chinese characters may be registered.

[0083] Note that the dictionary data is not language-dependent, and any strings may be registered for before conversion and after conversion.

[0084] <Operation command definition data held by the operation command definition unit> Next, the operation command definition data used by the operation command recognition control unit 32 will be explained using Figures 12 and 13. Figure 12 shows an example of operation command definition data and system definition data held by the operation command definition unit 33.

[0085] Figure 12(a) shows an example of operation command definition data. The operation command definition data in Figure 12(a) is an example of operation command definition data when there is no selected object selected by the handwriting object, and applies to all operation commands for operating the handwriting input device 2. The operation command in Figure 12(a) has an operation command name (Name), a string that partially matches the string candidate (String), and an operation command string to be executed (Command). "%~%" in the operation command string is a variable and is associated with system definition data as shown in Figure 12(b). In other words, "%~%" is replaced with the system definition data shown in Figure 12(b).

[0086] First, operation command definition data 701 indicates that the operation command name is "Load meeting minutes template", the string that partially matches the string candidate is "Meeting minutes" or "Template", and the operation command string to be executed is "ReadFile https: / / %username%:%password%@server.com / template / minutes.pdf". In this example, the operation command string to be executed contains the system definition data "%~%", and "%username%" and "%password%" are replaced by system definition data 704 and 705, respectively. Therefore, the final operation command string to be executed is "ReadFile https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf", which indicates that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf" will be read (ReadFile).

[0087] Operation command definition data 702 indicates that the operation command name is "Save to meeting minutes folder", the string that partially matches the string candidate is "Meeting minutes" or "Save", and the operation command string to be executed is "WriteFile https: / / %username%:%password%@server.com / minutes / %machinename%_%yyyy-mm-dd%.pdf". Similar to operation command definition data 701, "%username%", "%password%", and "%machinename%" in the operation command string will be replaced with system definition data 704-706, respectively. Note that "%yyyy-mm-dd%" will be replaced with the current date. For example, if the current date is September 26, 2018, it will be replaced with "2018-09-26". The final command to be executed is "WriteFile https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf", which indicates that the meeting minutes will be saved (WriteFile) to the file "https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf".

[0088] Operation command definition data 703 indicates that the operation command name is "Print", the string that partially matches the string candidate is "Print" or "Print", and the operation command string to be executed is "PrintFile https: / / %username%:%password%@server.com / print / %machinename%-"%yyyy-mm-dd%.pdf". If the operation command string is replaced in the same way as operation command definition data 702, the operation command to be executed will ultimately be "PrintFile https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf", indicating that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf" will be printed (PrintFile). In other words, the file will be sent to the server. When the user makes the printer communicate with the server and specifies the file, the printer will print the contents of the file onto the paper.

[0089] In this way, operation command definition data 701-703 can be identified from the string candidates, allowing the user to display the operation command by writing it manually. Furthermore, if user authentication is successful, the "%username%" and "%password%" in the operation command definition data will be replaced with user information, enabling file input and output associated with the user.

[0090] If user authentication is not performed (including cases where authentication fails but the user can still use handwriting input device 2), handwriting input device 2 will be replaced with the pre-configured "%username%", "%password%", etc. for handwriting input device 2. Therefore, file input and output will be possible by associating it with handwriting input device 2 even without user authentication.

[0091] Next, we will explain the operation command definition data when there are handwritten objects, that is, the editing and modification operation command definition data. Figure 13 shows an example of operation command definition data when there is a selected object selected by a handwritten object. The operation command definition data in Figure 13 has the operation command name (Name), the group name of the candidate operation command (Group), and the operation command string to be executed (Command).

[0092] Operation command definition data 707 defines the operation commands for editing (Group="Edit"), and is an example of definition data for the editing operation command names "Erase," "Move," "Rotate," and "Select." In other words, these operation commands are displayed for the selected object, and the user can select the desired operation command.

[0093] Operation command definition data 708 defines operation commands for the modifier group (Group="Decorate"), and is an example of definition data for the modifier operation command names "Bold," "Thin," "Larger," "Smaller," and "Underline." These operation commands are displayed for the selected object, and the user can select the desired operation command. In addition, color operation commands may also be displayed.

[0094] Therefore, when the user selects an object using a handwriting object, the operation command definition data 707 and 708 are identified, allowing the user to display the operation command by writing it by hand.

[0095] <Example of displaying suggested operation commands> Figure 14 shows an example of displaying candidate operation commands based on the operation command definition data in Figure 12(a) (when no object is selected). The procedure for displaying candidate operation commands is explained in detail in the sequence diagrams in Figures 18 to 23.

[0096] Figure 14 shows an example of the operation guide 500 and the selectable candidates 530 displayed by the operation guide 500. When the user writes the handwritten object 504 by hand (due to the timeout of the selectable candidate display timer), the operation guide 500 is displayed. The operation guide 500 has an operation header 520, candidates 510 for operation commands, candidates 506 for handwritten recognition strings, candidates 507 for conversion strings, candidates 508 for predicted strings, and a display 503 of the rectangular area of the handwritten object. The selectable candidates 530 are candidates 510 for operation commands, candidates 506 for handwritten recognition strings (including this if there are language string candidates 571), candidates 507 for conversion strings, and candidates 508 for predicted strings. Also, the selectable candidates 530 excluding the candidates 510 for operation commands are called string candidates �39.

[0097] The operation header <x> has buttons 501, 509, 502, and 505. The button 501 accepts the on / off switching operation of prediction conversion. In the example of Figure 14, when the user presses the button 501 displayed as "prediction", the handwritten input unit 21 accepts it and notifies the handwritten input display control unit 23 to that effect, and the display unit 22 changes the display of the button 501 to "kana". After the change, prediction conversion is not performed (the candidates 507 for conversion strings and the candidates 508 for predicted strings are not displayed), and the candidates 510 for operation commands, the candidates 506 for handwritten recognition strings, and the language string candidates (Chinese characters such as "議" after kana-kanji conversion) are displayed.

[0098] The button 502 performs page operations for candidate display. In the example of Figure 14, there are 3 pages for candidate display, and currently the first page is being displayed. The button 505 accepts the deletion of the operation guide 500. When the user presses the button 505, the handwritten input unit 21 accepts it and notifies the handwritten input display control unit 23 to that effect, and the display unit 22 deletes the display other than the handwritten object. The button 509 accepts batch display deletion. When the user presses the button 509, the handwritten input unit 21 accepts it and notifies the handwritten input display control unit 23 to that effect, and the display unit 22 deletes all the displays shown in Figure 14 including the handwritten object, enabling the user to rewrite the handwritten content from the beginning.

[0099] In FIG. 14, the handwritten object 504 is the character "ぎ" handwritten by the user. A handwritten object rectangular area display 503 surrounding the handwritten object 504 is displayed. The display procedure will be described in the sequence diagrams of FIGS. 18 to 23. In the example of FIG. 14, the handwritten object rectangular area display 503 is displayed with a dotted line frame.

[0100] In the handwritten recognition string candidates 506, the conversion string candidates 507, and the predicted string candidates 508, the respective string candidates are arranged in descending order of probability. The "ぎ" in the handwritten recognition string candidates 506 is a candidate for the recognition result. In this example, "ぎ" is correctly recognized.

[0101] The conversion string candidates 507 are string candidates converted from language string candidates. In this example, "技量試" is an abbreviation of "技術量産試作". The predicted string candidates 508 are predicted string candidates converted from the handwritten recognition strings, language string candidates, or conversion string candidates. In this example, "技量試を決裁" and "議事録の送付先" are displayed.

[0102] The candidates 510 for operation commands are candidates for operation commands selected based on the operation command definition data 701 to 703 in FIG. 11(a). In the example of FIG. 14, the leading character "》" 511 indicates that it is a candidate for the operation command. In FIG. 14, there is no selection data for the selection of the handwritten object 504 which is "ぎ", and the "議事録" which is a string candidate of "ぎ" partially matches the operation command definition data 701 and 702 in FIG. 11(a), so it is displayed as the candidate 510 for the operation command.

[0103] When the user selects "Load the minutes template", the operation command defined in the operation command definition data 701 is executed. When the user selects "Save to the minutes folder", the operation command defined in the operation command definition data 702 is executed. Thus, the candidates for operation commands are displayed when operation command definition data including the converted string is found, so they are not always displayed.

[0104] As shown in Figure 14, both string suggestions and operation command suggestions are displayed simultaneously, allowing the user to choose either the string suggestion or the operation command they intended to input.

[0105] <Example of specifying selected objects> The handwriting input device 2 of this embodiment allows the user to specify a selected object by handwriting a confirmed object. The selected object is then subject to editing or modification.

[0106] Figure 15 is an example of a diagram illustrating an example of specifying selected objects. In Figure 15, the handwritten object 11 is shown as a solid black line, the handwritten object rectangular area 12 is shaded in gray, the confirmed object 13 is shown as a black line, and the selected object rectangular area 14 is shown as a dotted line. Lowercase letters are added to the codes to distinguish between them. Furthermore, the crossover line determination condition 406 or the enclosure line determination condition 407 from the predefined control data shown in Figure 8 are used as the determination conditions (whether or not a predetermined relationship exists) for determining whether a confirmed object is a selected object.

[0107] Figure 15(a) shows an example where the user has specified two horizontally written confirmed objects 13a and 13b using a crossover line (hand-drawn object 11a). In this example, the length of the short side H1 and the length of the long side W1 of the hand-drawn object rectangular area 12a satisfy the crossover line determination condition 406, and the overlap rate with confirmed objects 13a and 13b also satisfies the crossover line determination condition 406, so both confirmed objects 13a and 13b, "meeting minutes" and "giji", are specified as selected objects.

[0108] Figure 15(b) shows an example of specifying a horizontally written confirmed object 13c with a border (hand-drawn object 11b). In this example, only the confirmed object 13c labeled "Meeting Minutes," whose overlap rate with the hand-drawn object rectangular area 12c satisfies the border judgment condition 407, is specified as the selected object.

[0109] Figure 15(c) shows an example of specifying multiple vertically written confirmed objects 13d and 13e with a crossover line (handwritten object 11c). In this example, similar to Figure 15(a), the length of the short side H1 and the length of the long side W1 of the handwritten object rectangular area 12d satisfy the conditions of the crossover line determination condition 406, and the overlap rate with the two confirmed objects 13d and 13e, "meeting minutes" and "giji", respectively, also satisfies the conditions of the crossover line determination condition 406, so both confirmed objects 13d and 13e, "meeting minutes" and "giji", are specified as selected objects.

[0110] Figure 15(d) shows an example where a vertically written fixed object 13f is specified with a border (hand-drawn object 11d). In this example, as with Figure 15(b), only the fixed object 13f titled "Meeting Minutes" is specified as the selected object.

[0111] <Example of displaying suggested operation commands> Figure 16 shows an example of displaying candidate operation commands based on operation command definition data when there is a handwritten object as shown in Figure 13. Figure 16(a) shows candidate editing operation commands, and Figure 16(b) shows candidate modification operation commands. Also, Figure 16(a) shows an example where the selected object is specified by the handwritten object 11a in Figure 15(a).

[0112] As shown in Figures 16(a) and 16(b), the candidate operation commands displayed following the bullet point "》" 511 constitute the main menu 550. The main menu 550 displays the name of the last executed operation command or the first operation command name in the operation command definition data. The bullet point "》" 511a on the first line is a candidate for an editing operation command, and the bullet point "》" 511b on the second line is a candidate for a modification operation command.

[0113] The ">" 512 at the end of the line indicates that there is a sub-menu (an example of a sub-menu button). The ">" 512a in the first line displays the sub-menu (the last selected) of the candidate operation commands of the editing system, and the ">" 512b in the second line displays the remaining sub-menus of the candidate operation commands of the modification system. When the user presses the ">" 512, the sub-menu 560 is displayed on its right side. All the operation commands defined in the operation command definition data are displayed in the sub-menu 560. In the display example of FIG. 16(a), the sub-menu 560 corresponding to the ">" 512a in the first line is also displayed since the main menu is displayed. It may be displayed by pressing the ">" 512a in the first line.

[0114] When the user presses any operation command name with the pen, the Command of the operation command definition data associated with the operation command name is executed by the handwritten input display control unit 23 for the selected object. That is, when "Delete" 521 is selected, "Delete" is executed, when "Move" 522 is selected, "Move" is executed, when "Rotate" 523 is selected, "Rotate" is executed, and when "Select" 524 is selected, "Select" is executed respectively.

[0115] For example, when the user presses "Delete" 521 with the pen, "Proceedings" and "Minutes" can be deleted. When "Move" 522, "Rotate" 523, or "Select" 524 is pressed, a bounding box (the circumscribed rectangle of the selected object) is displayed. With "Move" 522 and "Rotate" 523, it can be moved or rotated by the drag operation of the pen, and with "Select" 524, other operations on the bounding box can be executed.

[0116] The strings "—" 541, "—," 542, "~" 543, "→" 544, "⇒" 545, which are string candidates other than the candidates of the operation commands, are the recognition results of the strike-through line (the handwritten object 11a). When the user intends to input a string rather than an operation command, the string candidate can be selected.

[0117] In Figure 16(b), pressing the ">" 512b on the second line displays the submenu shown in Figure 16(b). The display example in Figure 16(b) also shows the main menu 550 and submenu 560, similar to Figure 16(a). Based on the operation command definition data in Figure 13, the handwriting input display control unit 23 executes the following commands on the selected object: "Thick" when "Thick" 531 is selected, "Thin" when "Thin" 532 is selected, "Large" when "Large" 533 is selected, "Small" when "Small" 534 is selected, and "Underline" when "Underline" 535 is selected.

[0118] Furthermore, fixed values ​​are separately defined for how thick the line becomes when "thick" 531 is selected, how thin it becomes when "thin" 532 is selected, how large it becomes when "large" 533 is selected, how small it becomes when "small" 534 is selected, and the line type when underline 535 is selected. Alternatively, it would be even better if a separate selection menu opened when the submenu in Figure 16(b) is selected, allowing the user to adjust these settings.

[0119] When the user presses "Thick" 531 with the pen, the handwriting input display control unit 23 thickens the lines that make up the confirmed objects 13a and 13b, "Meeting Minutes" and "Giji". When the user presses "Thin" 532 with the pen, the handwriting input display control unit 23 can thin the lines that make up "Meeting Minutes" and "Giji". When the user presses "Larger" 533 with the pen, the handwriting input display control unit 23 can enlarge the lines. When the user presses "Smaller" 534 with the pen, the handwriting input display control unit 23 can shrink the lines. When the user presses "Underline" 535 with the pen, the handwriting input display control unit 23 can add an underline.

[0120] Figure 17 shows an example of displaying candidate operation commands based on operation command definition data when there is a handwritten object as shown in Figure 13. The difference from Figures 16(a) and 16(b) is that Figures 17(a) and 16(b) show an example where the selected object is specified by the handwritten object 11b (border line) in Figure 15(b). As can be seen by comparing Figures 16 and 17, there is no difference in the candidate operation commands displayed depending on whether the handwritten object is a line or a border line. This is because the handwritten input display control unit 23 displays candidate operation commands on the display unit 22 when a selected object is specified. However, it is also possible to recognize the handwritten object and change the candidate operation commands displayed according to the handwritten object. In this case, operation command definition data as shown in Figure 13 is associated with the recognized handwritten object (such as a line or a circle).

[0121] In Figure 17, the string candidates other than the operation command candidates, namely "○" 551, "∞" 552, "0" 553, "00" 554, and "ロ" 555, are the result of recognizing the enclosed line (handwritten object 11b). If the user intended to input a string rather than an operation command, they can select one of the string candidates.

[0122] <Operating Procedure> The operation of the handwriting input device 2 will be explained using the above configuration and Figures 18 to 23. Figures 18 to 23 are sequence diagrams illustrating an example of the process by which the handwriting input device 2 displays string candidates and operation command candidates. The process in Figure 18 starts when the handwriting input device 2 is activated (when the application is launched). Note that, for space reasons, the functions in Figure 6 are indicated by symbols in Figures 18 to 23.

[0123] S1: First, the handwriting input display control unit 23 transmits the start of a handwriting object to the handwriting input storage unit 25. The handwriting input storage unit 25 reserves a handwriting object area (a memory area for saving handwriting objects). The handwriting object area may be reserved after the user touches the pen to the handwriting input unit 21.

[0124] S2: Next, the user touches the pen to the handwriting input unit 21. The handwriting input unit 21 detects the pen drop and transmits it to the handwriting input display control unit 23.

[0125] S2.1: The handwriting input display control unit 23 transmits a stroke start signal to the handwriting input storage unit 25, and the handwriting input storage unit 25 reserves the stroke area.

[0126] S3: When the user moves the pen while keeping it in contact with the handwriting input unit 21, the handwriting input unit 21 transmits the pen coordinates to the handwriting input display control unit 23.

[0127] S3.1: The handwriting input display control unit 23 transmits pen coordinate interpolation display data (data that interpolates discrete pen coordinates) to the display unit 22. The display unit 22 uses the pen coordinate interpolation display data to interpolate the pen coordinates and display a line.

[0128] S3.2: The handwriting input display control unit 23 transmits the pen coordinates and the time of reception to the handwriting input storage unit 25. The handwriting input storage unit 25 adds the pen coordinates to the stroke. As long as the user is moving the pen, the handwriting input unit 21 periodically repeats transmitting the pen coordinates to the handwriting input display control unit 23, so the process in steps S3 to S3.2 is repeated until the pen is lifted.

[0129] S4: When the user lifts the pen from the handwriting input unit 21, the handwriting input unit 21 sends a pen-up signal to the handwriting input display control unit 23.

[0130] S4.1: The handwriting input display control unit 23 transmits the end of the stroke to the handwriting input storage unit 25, and the handwriting input storage unit 25 confirms the pen coordinates of the stroke. Once the pen coordinates of the stroke are confirmed, it becomes impossible to add pen coordinates to the stroke thereafter.

[0131] S4.2: Next, the handwriting input display control unit 23 transmits the acquisition of the overlap status between the handwriting object proximity rectangular area and the stroke rectangular area based on the handwriting object proximity rectangular area 403 to the handwriting input storage unit 25. The handwriting input storage unit 25 calculates the overlap status and transmits the overlap status to the handwriting input display control unit 23.

[0132] The following steps S4.3 to S4.5 are executed if the handwritten object's neighboring rectangular area and the stroke rectangular area do not overlap.

[0133] S4.3: If the handwritten object proximity rectangle and the stroke rectangle do not overlap, one handwritten object is determined, and the handwritten input display control unit 23 sends a message to the handwritten recognition control unit 26 to clear the held data.

[0134] S4.3.1~4.3.3: The handwriting recognition control unit 26 sends a data clearing message to the string conversion control unit 28, the predictive text control unit 30, and the operation command recognition control unit 32, respectively. The handwriting recognition control unit 26, the string conversion control unit 28, the predictive text control unit 30, and the operation command recognition control unit 32 clear the data related to the string candidates and operation command candidates they have held up to that point. Note that at the time of clearing, the last handwritten stroke is not added to the handwriting object.

[0135] S4.4: The handwriting input display control unit 23 transmits the completion of the handwriting object to the handwriting input storage unit 25. The handwriting input storage unit 25 confirms the handwriting object. Confirmation of a handwriting object means that one handwriting object is complete (no more strokes will be added).

[0136] S4.5: The handwriting input display control unit 23 transmits the start of a handwriting object to the handwriting input storage unit 25. In preparation for the start of writing (pen down) of the next handwriting object, the handwriting input storage unit 25 reserves a new handwriting object area.

[0137] S4.6: Next, the handwriting input display control unit 23 sends a stroke addition message to the handwriting input storage unit 25 regarding the stroke that was completed in step S4.1. If steps S4.3 to S4.5 are performed, the stroke to be added is the first stroke of the handwriting object, and the handwriting input storage unit 25 adds the stroke data to the handwriting object that is currently being started. If steps S4.3 to S4.5 are not performed, the stroke to be added is added to the handwriting object that is already being written.

[0138] S4.7: Next, the handwriting input display control unit 23 transmits the addition of a stroke to the handwriting recognition control unit 26. The handwriting recognition control unit 26 adds the stroke data to the stroke data holding area where the string candidates are stored.

[0139] S4.7.1: The handwriting recognition control unit 26 performs handwriting recognition on this stroke data holding area.

[0140] S4.7.2: The handwriting recognition control unit 26 transmits the resulting handwriting recognition string candidates to the handwriting recognition dictionary unit 27. The handwriting recognition dictionary unit 27 transmits linguistically plausible language string candidates to the handwriting recognition control unit 26.

[0141] S4.7.3: The handwriting recognition control unit 26 transmits the handwriting recognition string candidates and the received language string candidates to the string conversion control unit 28.

[0142] S4.7.3.1: The string conversion control unit 28 transmits the handwritten recognition string candidates and language string candidates to the string conversion dictionary unit 29. The string conversion dictionary unit 29 transmits the conversion string candidates to the string conversion control unit 28.

[0143] S4.7.3.2: The string conversion control unit 28 transmits the received conversion string candidate to the predictive conversion control unit 30.

[0144] S4.7.3.2.1: The predictive text control unit 30 transmits the received conversion string candidates to the predictive text dictionary unit 31. The predictive text dictionary unit 31 transmits the predicted string candidates to the predictive text control unit 30.

[0145] S4.7.3.2.2: The predictive text control unit 30 transmits the received predictive string candidates to the operation command recognition control unit 32.

[0146] S4.7.3.2.2.1: The operation command recognition control unit 32 transmits the received predicted string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can obtain the operation command candidate corresponding to the operation command definition data having a string that matches the predicted string candidate.

[0147] The process continues similarly from step S4.7.3.3 to S4.7.5.1, up to the submission of candidate operation commands. S4.7.3.3: The string conversion control unit 28 transmits the received conversion string candidate to the operation command recognition control unit 32.

[0148] S4.7.3.3.1: The operation command recognition control unit 32 transmits the received conversion string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can obtain the operation command candidate corresponding to the operation command definition data having a string that matches the conversion string candidate.

[0149] S4.7.4: The handwriting recognition control unit 26 transmits the handwriting recognition string candidates and language string candidates to the predictive text conversion control unit 30.

[0150] S4.7.4.1: The predictive text control unit 30 transmits the handwritten recognition string candidates and the received language string candidates to the predictive text dictionary unit 31. The predictive text dictionary unit 31 transmits the predicted string candidates to the predictive text control unit 30.

[0151] S4.7.4.2: The predictive text control unit 30 transmits the received predictive string candidates to the operation command recognition control unit 32.

[0152] S4.7.4.2.1: The operation command recognition control unit 32 transmits the received predicted string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can obtain the operation command candidate corresponding to the operation command definition data having a string that matches the predicted string candidate.

[0153] S4.7.5: The handwriting recognition control unit 26 transmits the handwriting recognition string candidates and the received language string candidates to the operation command recognition control unit 32.

[0154] S4.7.5.1: The operation command recognition control unit 32 transmits the handwritten recognition string candidate and the received language string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can obtain the operation command candidate corresponding to the operation command definition data having a string that matches the language string candidate.

[0155] S4.7.6: Next, the handwriting recognition control unit 26 sends a stroke addition to the operation command recognition control unit 32.

[0156] S4.7.6.1: The operation command recognition control unit 32 transmits the acquired location information of the confirmed object to the handwriting input storage unit 25. The handwriting input storage unit 25 transmits the location information of the confirmed object to the operation command recognition control unit 32.

[0157] S4.7.6.2: In order to determine the selected object, the operation command recognition control unit 32 determines whether the position information of the stroke received from the handwriting recognition control unit 26 in step S4.7.6 and the position information of the confirmed object received from the handwriting input storage unit 25 have a predetermined relationship, based on the crossing line determination condition 406 and the enclosure line determination condition 407, and stores the confirmed object that can be determined to be selected as the selected object. Also, in this case, since the selected object is identified, candidate operation commands for the input / output system are obtained from the operation command definition unit 33.

[0158] Furthermore, the handwriting recognition control unit 26, the string conversion control unit 28, the predictive text conversion control unit 30, and the operation command recognition control unit 32 each store handwriting recognition string candidates, language string candidates, conversion string candidates, predictive string candidates, operation command candidates, and data related to selected objects, respectively, so that they can be obtained in subsequent steps S4.8.1.2 to 4.8.1.4.

[0159] S4.8: Immediately after sending the stroke addition message to the handwriting recognition control unit 26 in step S4.7, the handwriting input display control unit 23 sends a message to the candidate display timer control unit 24 to start the selectable candidate display timer. The candidate display timer control unit 24 starts this timer.

[0160] The following steps S5 to S5.2 are executed if a pen-down occurs before a certain period of time has elapsed (before the timer times out).

[0161] S5: If the user touches the pen to the handwriting input unit 21 before the timer times out, the handwriting input unit 21 sends a pen down message (the same event as in step S2) to the handwriting input display control unit 23.

[0162] S5.1: The handwriting input display control unit 23 transmits a stroke start signal (same as step S2.1) to the handwriting input storage unit 25. The subsequent sequence is the same as in step S2.1 and later.

[0163] S5.2: The handwriting input display control unit 23 then transmits a message to the candidate display timer control unit 24 to stop the selectable candidate display timer. The candidate display timer control unit 24 stops the timer because a pen-down has been detected and the timer is no longer needed.

[0164] Steps S4.8.1 to S7.14 are executed if a pen-down does not occur before a certain period of time has elapsed (before the timer times out). Therefore, the string suggestions and operation command suggestions shown in Figure 14 are displayed.

[0165] 4.8.1: If the user does not touch the handwriting input unit 21 with a pen while the selectable candidate display timer is running, the candidate display timer control unit 24 sends a timeout message to the handwriting input display control unit 23.

[0166] S4.8.1.1: The handwriting input display control unit 23 transmits the acquisition of handwriting recognition / language string candidates to the handwriting recognition control unit 26. The handwriting recognition control unit 26 transmits the currently held handwriting recognition / language string candidates to the handwriting input display control unit 23.

[0167] S4.8.1.2: The handwriting input display control unit 23 sends the acquisition of conversion string candidates to the string conversion control unit 28. The string conversion control unit 28 sends the currently held conversion string candidates to the handwriting input display control unit 23.

[0168] S4.8.1.3: The handwriting input display control unit 23 transmits the acquisition of predicted string candidates to the predictive conversion control unit 30. The predictive conversion control unit 30 transmits the currently held predicted string candidates to the handwriting input display control unit 23.

[0169] S4.8.1.4: The handwriting input display control unit 23 transmits the acquisition of candidate operation commands to the operation command recognition control unit 32. The operation command recognition control unit 32 transmits the currently held candidate operation commands and selected objects to the handwriting input display control unit 23.

[0170] S4.8.1.5: Further, the handwritten input display control unit 23 transmits the estimation writing direction acquisition to the handwritten input storage unit 25. The handwritten input storage unit 25 determines from the stroke addition time, horizontal distance, and vertical distance of the handwritten object rectangular area, and transmits the estimated writing direction to the handwritten input display control unit 23.

[0171] S4.8.1.6: The handwritten input display control unit 23 creates selectable candidate display data as shown in FIG. 14 from these handwritten recognition character string candidates (in FIG. 14, "gi"), language string candidates (although not shown in FIG. 14, for example, "gi"), conversion string candidates (in FIG. 14, "minutes of meeting", "skill test"), prediction string candidates (in FIG. 14, "settle the skill test", "send destination of the minutes of meeting"), candidate operation commands (in FIG. 14, "load the minutes of meeting template", "save to the minutes of meeting folder"), each selection probability, and the estimated writing direction, and transmits the selectable candidate display data consisting of the character string candidates and the candidate operation commands to the display unit 22 for display.

[0172] S4.8.1.7: Also, the handwritten input display control unit 23 transmits the rectangular area display data (rectangular frame) of the handwritten object and the selected object (in FIG. 14, the handwritten object rectangular area display 503) to the display unit 22.

[0173] S4.8.1.8: The handwritten input display control unit 23 transmits the start of the selectable candidate display deletion timer to the candidate display timer control unit 24 in order to delete the selectable candidate display data after a certain time from the display. The candidate display timer control unit 24 starts this timer.

[0174] Steps S6 to S6.3 are executed when, during the start of the selectable candidate deletion timer, the user deletes the selectable candidate display displayed on the display unit 22, a change in the handwritten object occurs (that is, when the stroke of the handwritten object is added, deleted, moved, deformed, or divided), or when no candidate is selected until the timeout.

[0175] Furthermore, steps S6 to S6.1 are executed when the candidate display is cleared or when a change occurs in the handwritten object.

[0176] S6: The handwriting input unit 21 transmits to the handwriting input display control unit 23 that the selectable candidate display has been cleared or that a change has occurred in the handwriting object.

[0177] S6.1: The handwriting input display control unit 23 transmits a message to stop the selectable candidate deletion timer. The candidate display timer control unit 24 stops the timer because an operation has been performed on the handwriting object within a certain time period, making the timer unnecessary.

[0178] S6.2: The handwriting input display control unit 23 sends a message to the display unit 22 to erase the selectable candidate display data, thereby erasing the display.

[0179] S6.3: The handwriting input display control unit 23 clears the display by sending a message to the display unit 22 indicating that the rectangular area display data for the handwritten object and the selected object has been cleared. Therefore, if the display of the operation command candidate is cleared under conditions other than when an operation command candidate is selected, the handwritten object remains displayed.

[0180] 4.8.1.8.1: On the other hand, if no selection candidate display is erased or a change in the handwritten object occurs while the selectable candidate erase timer is running (i.e., the user does not perform any pen operations), the candidate display timer control unit 24 sends a timeout message to the handwritten input display control unit 23.

[0181] Similarly, after the timer for clearing the selectable candidate display times out, the handwriting input display control unit 23 executes steps S6.2 and S6.3. This is because the selectable candidate display data, the handwritten object, and the rectangular area display data for the selected object may be cleared after a certain period of time has elapsed.

[0182] If the user selects a candidate while the selectable candidate clear timer is running, steps S7 to S7.14 are executed.

[0183] S7: If the user selects a candidate while the selectable candidate clearing timer is running, the handwriting input unit 21 transmits the selection of a string candidate or operation command candidate to the handwriting input display control unit 23.

[0184] S7.1: The handwriting input display control unit 23 transmits a message to the candidate display timer control unit 24 to stop the selectable candidate display clear timer. The candidate display timer control unit 24 stops this timer.

[0185] S7.2: Next, the handwriting input display control unit 23 sends a message to the handwriting recognition control unit 26 to clear the retained data.

[0186] S7.2.1: The handwriting recognition control unit 26 sends a message to the string conversion control unit 28 to clear the retained data.

[0187] S7.2.2: The handwriting recognition control unit 26 transmits a message to the predictive conversion control unit 30 to clear the retained data.

[0188] S7.2.3: The handwriting recognition control unit 26 sends a message to the operation command recognition control unit 32 to clear the retained data.

[0189] The handwriting recognition control unit 26, the string conversion control unit 28, the predictive text control unit 30, and the operation command recognition control unit 32 clear the data related to the string candidates and operation command candidates that they have been holding up to that point.

[0190] S7.3: Next, the handwriting input display control unit 23 clears the display by sending a message to the display unit 22 to erase the selectable candidate display data.

[0191] S7.4: The handwriting input display control unit 23 erases the display by sending a message to the display unit 22 to erase the rectangular area display data of the handwritten object and the selected object.

[0192] S7.5: The handwriting input display control unit 23 erases the display by sending the erased handwriting object display data and the erased pen coordinate interpolation display data sent in step S3.1 to the display unit 22. This is because a string candidate or operation command candidate has been selected, making the handwriting object etc. unnecessary.

[0193] S7.6: The handwriting input display control unit 23 transmits a message to the handwriting input storage unit 25 indicating the deletion of a handwriting object.

[0194] If a string candidate is selected, steps S7.7 to S7.9 are executed.

[0195] S7.7: If a string candidate is selected, the handwriting input display control unit 23 sends a message to the handwriting input storage unit 25 to add a string object.

[0196] S7.8: The handwriting input display control unit 23 then transmits the acquisition of the string object font to the handwriting input storage unit 25. The handwriting input storage unit 25 selects a predefined font from the estimated character size of the handwriting object and transmits it to the handwriting input display control unit 23.

[0197] S7.9: Next, the handwriting input display control unit 23 uses the predefined font received from the handwriting input storage unit 25 to send string object display data to the display unit 22, which will be displayed at the same position as the handwritten object, thereby displaying it.

[0198] If a candidate for an operation command is selected, steps S7.10 to S7.13 are executed. Furthermore, if there are selected objects, steps S7.10 to S7.12 are executed.

[0199] S7.10: If a candidate for an operation command for the selected object is selected (if the selected object exists), the handwriting input display control unit 23 sends a message to the display unit 22 to erase the selected object display data, thereby erasing the display. This is to erase the original selected object.

[0200] S7.11: Next, the handwriting input display control unit 23 sends an operation command to the selected object to the handwriting input storage unit 25. The handwriting input storage unit 25 sends the display data of the new selected object (display data after editing or modification) to the handwriting input display control unit 23.

[0201] S7.12: Next, the handwriting input display control unit 23 transmits the selected object display data to the display unit 22, thereby redisplaying the selected object after the operation command has been executed.

[0202] If no object is selected (i.e., an input / output operation command is selected), step S7.13 is executed.

[0203] S7.13: When an input / output system operation command is selected, the handwriting input display control unit 23 executes the operation command string (Command) from the operation command definition data corresponding to the operation command selected by the user. If the user authentication unit 34 successfully authenticates the user, the handwriting input display control unit 23 sets the information about the successfully authenticated user in the %~% of the operation command and executes it.

[0204] S7.14: For the next handwritten object, the handwriting input display control unit 23 transmits a handwritten object start signal to the handwriting input storage unit 25. The handwriting input storage unit 25 reserves a handwritten object area. From here on, the process from steps S2 to S7.14 is repeated.

[0205] <Summary> As described above, the handwriting input device 2 of this embodiment can display a list of possible operation commands by allowing the user to handwrite an object or draw a border around a confirmed object. Therefore, any function (editing function, input / output function, or pen function, etc.) can be called up from the handwriting state. This eliminates the need for step-by-step operations such as pressing a menu button to call up the desired function, and reduces the number of steps required for the user to call up a desired function from the handwriting state. [Examples]

[0206] In this embodiment, we describe a handwriting input system in which an information processing system on a network performs processing such as handwriting recognition and returns the processing result to the handwriting input device 2.

[0207] In this description, components or figures that use the same reference numerals as in Example 1 perform the same function, so the explanation of components that have already been explained may be omitted, or only the differences may be explained.

[0208] Figure 24 is an example of a system configuration diagram of the handwriting input system 100. The handwriting input system 100 includes a handwriting input device 2 and an information processing system 10 that can communicate via a network N.

[0209] The handwriting input device 2 is located in a facility such as an office and is connected to a LAN or Wi-Fi network installed within the facility. The information processing system 10 is located, for example, in a data center. The handwriting input device 2 is connected to the internet i via a firewall 8, and the information processing system 10 is also connected to the internet i via a high-speed LAN within the data center.

[0210] The handwriting input device 2 may connect to the Internet i using wireless communication such as a telephone network. In this case, the wireless communication may be 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), etc.

[0211] The information processing system 10 has one or more information processing devices, one or more of which act as servers to provide services to the handwriting input device 2. A server is a computer or software that performs the function of providing information and processing results in response to requests from a client. As described later, the information processing system 10 receives pen coordinates from the handwriting input device 2 and transmits to the handwriting input device 2 the information necessary to display the operation guide 500 shown in Figure 14.

[0212] The server-side system is sometimes referred to as a cloud system. A cloud system is a system that utilizes cloud computing, which is a usage model in which network resources are used without being aware of specific hardware resources. Cloud systems are not necessarily located on the internet. In Figure 24, the information processing system is located on the internet, but it may also be located on a local network (in this case, called on-premises).

[0213] Furthermore, in one embodiment, the information processing system 10 includes a plurality of computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein.

[0214] The configuration of the handwriting input device 2 may be the same as in Embodiment 1, but in this embodiment, it is sufficient to have a touch panel, a display, and communication functionality. The handwriting input device 2 may include multiple computing devices configured to communicate with each other.

[0215] In this embodiment, a general information processing device such as a PC or tablet can run a web browser or a dedicated application. The web browser or dedicated application communicates with the information processing system 10. When the web browser is running, the user enters or selects the URL of the information processing system 10 to connect the handwriting input device to the information processing system 10. The handwriting input device 2 runs the web application provided by the information processing system 10 in the web browser. A web application refers to software or a system that operates on a web browser through the cooperation of a program written in a programming language (for example, JavaScript®) that runs on the web browser and a program on the web server side, and is executed on the web browser.

[0216] When the dedicated application is running, it connects to a pre-registered URL of the information processing system 10. Since the dedicated application has a program and a user interface, the program sends and receives the necessary information with the information processing system 10 and displays it on the user interface.

[0217] The communication method can be a general-purpose communication protocol such as HTTP, HTTPS, or WebSocket, or a dedicated communication protocol can be used.

[0218] <Example Hardware Configuration> The hardware configuration of the handwriting input device 2 may be the same as in Figure 5. In this embodiment, an example of the hardware configuration of the information processing system 10 will be described.

[0219] Figure 25 is a hardware configuration diagram of the information processing system 10. As shown in Figure 25, the information processing system 10 is built by a computer and includes a CPU 601, ROM 602, RAM 603, HD 604, HDD (Hard Disk Drive) controller 605, display 606, external device connection I / F (Interface) 608, network I / F 609, bus line 610, keyboard 611, pointing device 612, DVD-RW (Digital Versatile Disk Rewritable) drive 514, and media I / F 616.

[0220] Of these, the CPU 601 controls the operation of the entire information processing system 10. The ROM 602 stores programs used to drive the CPU 601, such as IPL. The RAM 603 is used as the work area for the CPU 601. The HD 604 stores various data such as programs. The HDD controller 605 controls the reading or writing of various data to the HD 604 according to the control of the CPU 601. The display 606 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 608 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 609 is an interface for data communication using a communication network. The bus line 610 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 601 shown in Figure 25.

[0221] The keyboard 611 also has multiple keys for inputting characters, numbers, and various instructions. The pointing device 612 is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor. The DVD-RW drive 614 controls the reading or writing of various data to the DVD-RW 613, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may also be DVD-R, etc. The media interface 616 controls the reading or writing (storage) of data to the recording medium 615, such as flash memory.

[0222] <About the device's functions> Next, we will explain the functions of the handwriting input system 100 using Figure 26. Figure 26 is an example of a functional block diagram that shows the functions of the handwriting input system 100 in block form. In the explanation of Figure 26, we will mainly explain the differences from Figure 6.

[0223] In this embodiment, the handwriting input device 2 includes a display unit 22, a display control unit 41, a handwriting input unit 21, and a communication unit 42. Each function of the handwriting input device 2 is a function or means realized by any of the components shown in Figure 5 operating according to instructions from the CPU 201 in accordance with a program deployed from the SSD 204 onto the RAM 203.

[0224] The function of the handwriting input unit 21 in this embodiment may be the same as in Embodiment 1. The handwriting input unit 21 converts the user's pen input d1 into pen operation data da (pen up, pen down, or pen coordinate data) and transmits it to the display control unit 41.

[0225] The display control unit 41 controls the display of the handwriting input device 2. First, the display control unit 41 interpolates the coordinates between discrete values ​​of the pen coordinate data, which are discrete values, and converts the pen coordinate data from pen down to pen up into a single stroke db, which is then transmitted to the display unit 22.

[0226] Furthermore, the display control unit 41 transmits pen operation data dc to the communication unit 42 and obtains various display data dd from the communication unit 42. The display data includes information for displaying the operation guide 500 shown in Figure 14. The display control unit 41 transmits display data de to the display unit 22.

[0227] The communication unit 42 transmits pen operation data dc to the information processing system 10, receives various display data dd from the information processing system 10, and transmits it to the display control unit 41.

[0228] The functions of the display unit 22 may be the same as in Embodiment 1. The display unit 22 displays stroke db and display data de. The display unit 22 converts the stroke db and display data de written to the video memory by the display control unit 41 into data according to the characteristics of the display 220 and transmits it to the display 220.

[0229] <<Functions of the Information Processing System>> The information processing system 10 includes a communication unit 43, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a string conversion control unit 28, a string conversion dictionary unit 29, a predictive conversion control unit 30, a predictive conversion dictionary unit 31, an operation command recognition control unit 32, and an operation command definition unit 33. Each function of the information processing system 10 is a function or means realized by any of the components shown in Figure 25 operating according to instructions from the CPU 601 in accordance with a program deployed from HD 604 onto RAM 603.

[0230] The communication unit 43 receives pen operation data dc from the handwriting input device 2, transmits pen operation data df to the handwriting input display control unit 23, receives display data dd from the handwriting input display control unit 23, and transmits it to the handwriting input device 2.

[0231] Other functions are the same as in Example 1, or any differences will not hinder the explanation of this embodiment.

[0232] Furthermore, the information processing system 10 shall authenticate users using a standard method.

[0233] <Operating Procedure> The operation of the handwriting input system 100 will be explained using the above configuration and Figures 27 to 33. Figures 27 to 33 are sequence diagrams illustrating an example of the process by which the handwriting input device 2 displays string candidates and operation command candidates. The process in Figure 27 starts when the handwriting input device 2 is activated (a web browser or dedicated application is launched) and communication with the information processing system 10 is established. The overall flow of Figures 27 to 33 is the same as that of Figures 18 to 23.

[0234] S1: Once communication is established, the handwriting input display control unit 23 transmits a handwriting object start signal to the handwriting input storage unit 25 in order to reserve a memory area for the handwriting input device 2. The handwriting input storage unit 25 reserves a handwriting object area (a memory area for saving handwriting objects). The handwriting object area may be reserved after the user touches the pen to the handwriting input unit 21.

[0235] S2a: Next, the user touches the pen to the handwriting input unit 21. The handwriting input unit 21 detects the pen drop and transmits it to the display control unit 41.

[0236] S2b: The display control unit 41 transmits a pen-down message to the communication unit 42 in order to notify the information processing system 10 of the pen-down.

[0237] S2c: The communication unit 42 transmits a pendown to the information processing system 10.

[0238] S2d: The communication unit 43 of the information processing system 10 receives the pen down signal and transmits it to the handwriting input display control unit 23.

[0239] S2.1: The handwriting input display control unit 23 transmits a stroke start signal to the handwriting input storage unit 25, and the handwriting input storage unit 25 reserves the stroke area.

[0240] S3a: When the user moves the pen while keeping it in contact with the handwriting input unit 21, the handwriting input unit 21 transmits the pen coordinates to the display control unit 41.

[0241] S3b: The display control unit 41 transmits the pen coordinates to the communication unit 42 in order to notify the information processing system 10 of the pen coordinates.

[0242] S3c: The communication unit 42 transmits the pen coordinates to the information processing system 10.

[0243] S3d: The communication unit 43 of the information processing system 10 receives the pen coordinates and transmits them to the handwriting input display control unit 23.

[0244] S3.1: The display control unit 41 transmits pen coordinate interpolation display data (data that interpolates discrete pen coordinates) to the display unit 22. The display unit 22 interpolates the pen coordinates using the pen coordinate interpolation display data and displays the line. The process in step S3.2 is the same as in Embodiment 1.

[0245] S4a: When the user lifts the pen from the handwriting input unit 21, the handwriting input unit 21 sends a "pen up" message to the display control unit 41.

[0246] S4b: The display control unit 41 transmits a pen-up to the communication unit 42 in order to notify the information processing system 10 of the pen-up.

[0247] S4c: Communication unit 42 transmits the pen-up to information processing system 10.

[0248] S4d: The communication unit 43 of the information processing system 10 receives the pen up and transmits it to the handwriting input display control unit 23.

[0249] The subsequent steps S4.1 to S4.3.3, and steps S4.6 to S4.7.6.2 in Figure 29, are the same as in Example 1.

[0250] S5a: If the user touches the handwriting input unit 21 with the pen before the timer times out, the handwriting input unit 21 sends a pen down message (the same event as in step S2) to the display control unit 41. The processing in steps S5b to S5d may be the same as in steps S2b to S2d. Also, the processing in steps S5.1 to S4.8.1.5 will be the same as in Embodiment 1.

[0251] S4.8.1.6a: The handwriting input display control unit 23 creates selectable candidate display data including each string candidate, operation command candidate, selection probability, and estimated writing direction shown in Figure 14, and transmits the selectable candidate display data consisting of string candidates and operation command candidates to the communication unit 43.

[0252] S4.8.1.6b: The communication unit 43 transmits selectable candidate display data to the handwriting input device 2.

[0253] S4.8.1.6c: The communication unit 42 of the handwriting input device 2 receives selectable candidate display data and transmits it to the display control unit 41.

[0254] S4.8.1.6d: The display control unit 41 has received the selectable candidate display data and transmits it to the display unit 22 to display it.

[0255] S4.8.1.7a: The handwriting input display control unit 23 also transmits rectangular area display data (rectangular frame) (handwriting object rectangular area display 503 in Figure 14) of the handwritten object and the selected object to the communication unit 43.

[0256] S4.8.1.7b: The communication unit 43 transmits rectangular area display data to the handwriting input device 2.

[0257] S4.8.1.7c: The communication unit 42 of the handwriting input device 2 receives rectangular area display data and transmits it to the display control unit 41.

[0258] S4.8.1.7d: Since the control unit 41 has received the rectangular area display data, it is transmitted to the display unit 22 for display. The process of step S4.8.1.8 is the same as that in the first embodiment.

[0259] S6a: When the user deletes the selectable candidates or adds handwritten content to the handwritten object, the handwriting input unit 21 transmits the occurrence of the deletion of the selectable candidate display or the change of the handwritten object to the display control unit 41.

[0260] S6b: The display control unit 41 transmits it to the communication unit 42 in order to notify the information processing system 10 of the occurrence of the deletion of the selectable candidate display or the change of the handwritten object. <​​​​​​​​​​​​​​​​​​​​​​​​​​S6.3a: The handwriting input display control unit 23 transmits to the communication unit 43 that the rectangular area display data for the handwritten object and the selected object has been erased.

[0268] S6.3b: The communication unit 43 transmits to the handwriting input device 2 the deletion of the rectangular area display data for the handwritten object and the selected object.

[0269] S6.3c: The communication unit 42 of the handwriting input device 2 receives the deletion of the rectangular area display data for the handwritten object and the selected object, and transmits it to the display control unit 41.

[0270] S6.3d: The display control unit 41 receives the notification that the rectangular area display data for the handwritten object and the selected object has been erased, and transmits it to the display unit 22 to erase the rectangular area for the handwritten object and the selected object. Therefore, if the display of the operation command candidate is erased under conditions other than when an operation command candidate is selected, the handwritten object will remain displayed.

[0271] If the user selects a candidate while the selectable candidate clear timer is running, steps S7 to S7.14 are executed.

[0272] S7a: If the user selects a candidate while the selectable candidate clearing timer is running, the handwriting input unit 21 transmits the selection of a string candidate or operation command candidate to the display control unit 41.

[0273] S7b: The display control unit 41 transmits to the communication unit 42 the selection of a string candidate or operation command candidate to the information processing system 10.

[0274] S7c: The communication unit 42 transmits a selection of string candidates or operation command candidates to the information processing system 10.

[0275] S7d: The communication unit 43 of the information processing system 10 receives the selection of a string candidate or operation command candidate and transmits it to the handwriting input display control unit 23. The processing in steps S7.1 to S7.2.3 is the same as in Embodiment 1.

[0276] S7.3a: Next, the handwritten input display control unit 23 transmits the erasure of selectable candidate display data to the communication unit 43.

[0277] S7.3b: The communication unit 43 transmits the erasure of selectable candidate display data to the handwritten input device 2.

[0278] S7.3c: The communication unit 42 of the handwritten input device 2 receives the erasure of selectable candidate display data and transmits it to the display control unit 41.

[0279] S7.3d: Since the display control unit 41 has received the erasure of selectable candidate display data, it causes the display unit 22 to erase the selectable candidates.

[0280] S7.4a: The handwritten input display control unit 23 transmits the erasure of the rectangular area display data of the handwritten object and the selected object to the communication unit 43.

[0281] S7.4b: The communication unit 43 transmits the erasure of the rectangular area display data to the handwritten input device 2.

[0282] S7.4c: The communication unit 42 of the handwritten input device 2 receives the erasure of the rectangular area display data and transmits it to the display control unit 41.

[0283] S7.4d: Since the display control unit 41 has received the erasure of the rectangular area display data, it causes the display unit 22 to erase the rectangular area.

[0284] S7.5a: The handwritten input display control unit 23 transmits the erasure of the handwritten object display data to the communication unit 43.

[0285] S7.5b: The communication unit 43 transmits the erasure of the handwritten object display data to the handwritten input device 2.

[0286] S7.5c: The communication unit 42 of the handwritten input device 2 receives the erasure of the handwritten object display data and transmits it to the display control unit 41.

[0287] S7.5d: The display control unit 41 receives a notification that the handwritten object display data has been deleted, and instructs the display unit 22 to delete the handwritten object and the pen coordinate interpolation display data. The process in step S7.6 may be the same as in Example 1.

[0288] If a string candidate is selected, steps S7.7 to S7.9 are executed. The processing in steps S7.7 and S7.8 may be the same as in Example 1.

[0289] S7.9a: Next, the handwriting input display control unit 23 uses the predefined font received from the handwriting input storage unit 25 to send string object display data to the communication unit 43 to be displayed in the same position as the handwritten object.

[0290] S7.9b: The communication unit 43 transmits the string object display data to the handwriting input device 2.

[0291] S7.9c: The communication unit 42 of the handwriting input device 2 receives string object display data and transmits it to the display control unit 41.

[0292] S7.9d: The display control unit 41 has received the string object display data and will display the string object on the display unit 22.

[0293] If a candidate for an operation command is selected, steps S7.10 to S7.13 are executed. Furthermore, if there are selected objects, steps S7.10 to S7.12 are executed.

[0294] S7.10a: If a candidate for an operation command for the selected object is selected (if the selected object exists), the handwriting input display control unit 23 sends a message to the communication unit 43 to erase the selected object display data. This is to erase the original selected object.

[0295] S7.10b: The communication unit 43 transmits a message to the handwriting input device 2 to erase the selected object display data.

[0296] S7.10c: The communication unit 42 of the handwriting input device 2 receives the deletion of the selected object display data and transmits it to the display control unit 41.

[0297] S7.10d: The display control unit 41 has received the notification that the selected object display data has been deleted, so it instructs the display unit 22 to delete the selected object.

[0298] S7.11: Next, the handwriting input display control unit 23 sends an operation command to the selected object to the handwriting input storage unit 25. The handwriting input storage unit 25 sends the display data of the new selected object (display data after editing or modification) to the handwriting input display control unit 23.

[0299] S7.12a: Next, the handwriting input display control unit 23 transmits the selected object display data to the communication unit 43.

[0300] S7.12b: The communication unit 43 transmits the selected object display data to the handwriting input device 2.

[0301] S7.12c: The communication unit 42 of the handwriting input device 2 receives the selected object display data and transmits it to the display control unit 41.

[0302] S7.12d: The display control unit 41 has received the selected object display data and instructs the display unit 22 to redisplay the selected object after the operation command has been executed. The processing in steps S7.13 and S7.14 may be the same as in Embodiment 1.

[0303] Thus, even in a system configuration where the handwriting input device 2 and the information processing system 10 communicate, the same effects as in Embodiment 1 can be achieved. Note that the processing flow in Figures 27 to 33 is just an example, and may include or omit processing that occurs as a result of communication between the handwriting input device 2 and the information processing system 10. Furthermore, the handwriting input device 2 may perform some of the processing that the information processing system 10 performs. For example, the handwriting input device 2 may perform the processing related to deletion.

[0304] <Example of operation guide display> The operation guide 500 for the handwriting recognition system can be the same as that shown in Figure 14. Of course, the design and other elements can be changed. This example describes an example of the operation guide displayed when English is handwritten.

[0305] Figure 34 shows an example of the operation guide 500. When the information processing system 10 recognizes a foreign language such as English, it would be desirable for it to also recognize the type of language. However, in order to improve recognition accuracy, there is a way to identify the type of language that the user has handwritten. For example, the user can select the type of language from a menu displayed by the handwriting input device 2 and notify the information processing system 10. Alternatively, the user can register user attributes, including the locale (language used), with the information processing system 10, and the type of language can be identified when the user logs in. Or, if the information processing system 10 is on the internet, the country or region can be determined by the IP address of the handwriting input device 2, and the information processing system 10 can determine the language used in that country or region.

[0306] (1) In Figure 34, for example, the user wrote "pl" by hand.

[0307] (2) The handwriting recognition control unit 26, handwriting recognition dictionary unit 27, string conversion control unit 28, string conversion dictionary unit 29, predictive text control unit 30, and predictive text dictionary unit 31 are all equipped to handle English, and the handwriting recognition control unit 26 recognizes "pl" as an English word. The information processing system 10 recognizes the character code "pl" and sends a conversion string candidate 507 that partially contains "pl" and a predictive string candidate 508 that contains the conversion string to the handwriting input device 2. As a result, the handwriting input device 2 can display words and phrases (selectable candidates) that start with "pl," such as "play," "place," "placard," "placenta," and "play ground." Although not shown in Figure 34, operation commands may also be displayed.

[0308] (3) When the user selects "play", the operation guide 500 and the handwritten object 504 are erased and "play" is displayed.

[0309] <Summary> Therefore, in this embodiment, handwriting recognition can be performed in the handwriting input system 100, which communicates with the handwriting input device 2 and the information processing system 10, in the same manner as in Embodiment 1.

[0310] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0311] For example, although an electronic blackboard was described as an example in this embodiment, any information processing device having a touch panel can be suitably applied. Devices with similar functions to an electronic blackboard are also called electronic whiteboards, electronic information boards, interactive boards, etc. Examples of information processing devices equipped with a touch panel include projectors (PJ), output devices such as digital signage, head-up display (HUD) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, networked home appliances, personal computers (notebook PCs), mobile phones, smartphones, tablet devices, game consoles, personal digital assistants (PDAs), digital cameras, wearable PCs, or desktop PCs.

[0312] In this embodiment, the pen's coordinates were detected by detecting the pen tip's coordinates on a touch panel, but the pen tip's coordinates may also be detected by ultrasound. The pen emits ultrasound along with light emission, and the handwriting input device 2 calculates the distance based on the arrival time of the ultrasound. The pen's position can be determined by its direction and distance. The projector draws (projects) the pen's trajectory as a stroke.

[0313] Furthermore, in this embodiment, candidate editing and modification operation commands are displayed when there is a selected object, and candidate input / output operation commands are displayed when there is no selected object. However, candidate editing and modification operation commands and candidate input / output operation commands may be displayed simultaneously.

[0314] Furthermore, the configuration examples in Figures 6 and 26 are divided according to their main functions to facilitate understanding of the processing performed by the handwriting input device 2. The present invention is not limited by the way the processing units are divided or their names. The processing of the handwriting input device 2 can be further divided into many more processing units depending on the processing content. Also, one processing unit can be divided to include even more processing.

[0315] Furthermore, each function of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above. [Explanation of Symbols]

[0316] 2. Handwriting input device 21 Handwriting input section 22 Display section 23 Handwriting Input Display Control Unit 24 Candidate display timer control unit 25 Handwriting input storage section 26 Handwriting Recognition Control Unit 27 Handwriting Recognition Dictionary Section 28 String Conversion Control Unit 29 String Conversion Dictionary Section 30 Predictive text control unit 31. Predictive Text Dictionary Section 32 Operation Command Recognition Control Unit 33 Operation Command Definition Section 34. User Authentication Section 35. Authentication Information Acquisition Section [Prior art documents] [Patent Documents]

[0317] [Patent Document 1] Japanese Patent Publication No. 2018-026185

Claims

1. A handwriting input unit that accepts handwriting input using the position of a pen or the user's finger in contact with the display, The system comprises a display unit that displays the handwritten input received by the handwritten input unit as a handwritten object on the display, An input device characterized in that, if no change occurs in the handwritten object during the first time period, it displays one or more operation commands based on the handwritten object.

2. If no change occurs in the handwritten object during the first time period, the handwritten object rectangular region containing the handwritten object is displayed, and one or more candidate operation commands that recognize the handwritten object as an operation command are displayed. If the aforementioned operation command candidate is not selected during the second time, if an operation to clear the display of the aforementioned operation command candidate is received, or if a change occurs in the handwritten object, the displayed operation command candidate is cleared. The input device according to claim 1, characterized in that if the display of the operation command candidate is cleared under conditions other than the selection of the operation command candidate, the display of the handwritten object is maintained.

3. The system displays one or more string candidates that recognize the handwritten object as a string. If neither the operation command candidate nor the string candidate is selected during the second time, or if an operation to clear the display of the operation command candidate and the string candidate is received, the display of the operation command candidate and the string candidate is cleared. The input device according to claim 2, characterized in that if the display of the operation command candidate and the string candidate is erased under conditions other than the selection of the operation command candidate and the string candidate, the display of the handwritten object is maintained.

4. The aforementioned string candidates are displayed as characters, numbers, symbols, or shapes. The input device according to claim 3, characterized in that it displays the operation command together with letters, numbers, symbols, or figures.

5. It stores constants that replace the variables associated with the aforementioned operation commands, Replace the variable associated with the operation command recognized from the handwritten object with the constant, The input device according to any one of claims 1 to 4, characterized in that it executes the operation command in which the variable is replaced by the constant.

6. It has a user authentication means for authenticating users, If the user authentication means succeeds in authentication, the information regarding the successfully authenticated user is set to the constant. Replace the variables associated with the recognized operation command with information about the user, The input device according to claim 5, characterized in that it executes the operation command in which the aforementioned variable is replaced with information about the user.

7. The input device according to any one of claims 1 to 6, characterized in that, when the position information of a confirmed object that has already been handwritten and the handwritten object are in a predetermined relationship, the device recognizes the operation command relating to the confirmed object that is in a predetermined relationship with the handwritten object.

8. The input device according to claim 7, characterized in that it recognizes the operation command for editing or modifying the confirmed object which has a predetermined relationship with the handwritten object.

9. If the positional information of each of the multiple already handwritten confirmed objects is in a predetermined relationship with one of the handwritten objects, The input device according to claim 7 or 8, characterized in that it recognizes the operation command relating to a plurality of confirmed objects that have a predetermined relationship with one of the handwritten objects.

10. The input device according to any one of claims 7 to 9, characterized in that, if a confirmed object that has already been handwritten and the handwritten object overlap by a predetermined overlap rate or greater than or equal to a predetermined overlap rate, the device determines that the position information of the confirmed object that has already been handwritten and the handwritten object are in the predetermined relationship.

11. When displaying multiple operation commands, one or more operation commands and submenu buttons are displayed. The input device according to any one of claims 1 to 10, characterized in that it displays the remaining operation commands in response to the operation of the submenu button.

12. The input device according to claim 11, characterized in that the operation command displayed together with the submenu button is the operation command last selected by the user.

13. The input device according to claim 2, characterized in that conditions other than the selection of the operation command candidate are when an operation to clear the display of the operation command candidate is received, or when a change occurs in the handwritten object.

14. The handwriting input unit accepts handwriting input using the position of a pen or the user's finger in contact with the display. The display unit displays the handwritten input received by the handwritten input unit as a handwritten object on the display, If no change occurs in the handwritten object during the first time period, the step is to display one or more operation commands based on the handwritten object. An input method characterized by having the following features.

15. Information processing equipment, A handwriting input unit that accepts handwriting input using the position of a pen or the user's finger in contact with the display, The handwriting input received by the handwriting input unit is configured to function as a display unit that displays the handwriting object on the display, A program that, if no change occurs in the handwritten object during the first time period, displays one or more operation commands based on the handwritten object.

16. An input system in which an input device and an information processing system communicate, The aforementioned input device is A handwriting input unit that accepts handwriting input using the position of a pen or the user's finger in contact with the display, A display unit that displays the handwritten input received by the handwritten input unit as a handwritten object on the display, The system includes a communication unit that transmits the position of the pen or the user's finger received by the handwriting input unit to the information processing system and receives candidate operation commands from the information processing system, The aforementioned information processing system is An input system characterized in that, if no notification is received during the first period indicating that a change has occurred in the handwritten object, one or more candidate operation commands are sent to the input device based on the handwritten object.

Images