Display device, display method, and program

Abstract

To provide a display device which can conceal handwritten display information.SOLUTION: A display device 2 which displays handwritten data input by input means includes: conversion means 26 which converts handwritten data to a character string; and display control means 23 which conceals the character string converted by the conversion means when a first time passes after displaying the character string.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a display device, a display method, and a program. [Background technology]

[0002] Display devices that display data handwritten on a touch panel using an input means such as a pen or a finger are known. Display devices equipped with relatively large touch panels are placed in places such as conference rooms and used by multiple users as electronic whiteboards.

[0003] A technique for managing confidential information using handwriting data for such display devices is known (see, for example, Patent Document 1). Patent Document 1 discloses a method in which a user registers handwriting data in advance and links the handwriting data with confidential information that the user wants to input. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional display devices have a problem in that they cannot conceal information displayed by handwriting. For example, there are cases where multiple users share a display device. If personal information such as a name is handwritten and left displayed, there is a high possibility that the personal information will be leaked to a third party. On the other hand, if personal information is not displayed at all, it will be difficult to view, which will hinder business operations. For example, there is a possibility that incident information at a fire scene, or patient information at a disaster or emergency medical site, may be leaked by a third party.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide a display device that can conceal information displayed by handwriting input. [Means for solving the problem]

[0006] In view of the above problems, the present invention is a display device that displays handwritten data input by an input means, characterized in that it comprises a conversion means that converts the handwritten data into a character string, and a display control means that conceals the character string converted by the conversion means after a first time has elapsed since it was displayed. [Effects of the Invention]

[0007] It is possible to provide a display device that can conceal information displayed by handwriting input. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 10 is a diagram illustrating a comparative example of a method for inputting confidential information. [Figure 2] 10A and 10B are diagrams illustrating an example of input of confidential information according to the present embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a perspective view of a pen. [Figure 4] FIG. 1 is a diagram showing an example of an overall configuration diagram of a display device. [Figure 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device. [Figure 6] FIG. 2 is a diagram illustrating functions of a display device and a pen. [Figure 7] FIG. 10 is a diagram illustrating an example of predefined control data. [Figure 8] FIG. 4 is a diagram showing an example of dictionary data in a handwriting recognition dictionary unit. [Figure 9] FIG. 10 is a diagram illustrating an example of dictionary data in a character string conversion dictionary unit. [Figure 10] FIG. 10 is a diagram illustrating an example of dictionary data in a predictive conversion dictionary unit. [Figure 11] 10A and 10B are diagrams illustrating an example of operation command definition data and system definition data held by an operation command definition unit. [Figure 12] FIG. 10 is a diagram showing an example of file list display data displayed on a display unit. [Figure 13] 10A and 10B are diagrams showing an example of operation command definition data in the case where a selected object is selected by a handwritten object; [Figure 14] 10 is a diagram showing an example of user-defined data held in an operation command definition unit; FIG. [Figure 15] 10 is a diagram showing an example of handwritten signature data stored in a handwritten signature data storage unit. FIG. [Figure 16] 10 is a diagram showing an example of handwritten input saved data saved in a handwritten input saving unit; FIG. [Figure 17] 10A and 10B are diagrams illustrating pen ID control data stored in a pen ID control data storage unit. [Figure 18] FIG. 10 is a diagram illustrating an example of color definition data. [Figure 19] 19A and 19B are diagrams showing an example of data displayed in black and white highlighting and data displayed in a color display method using the color definition data of FIG. 18. [Figure 20] FIG. 10 is a diagram illustrating an example of pen color selection button definition data. [Figure 21] FIG. 10 is a diagram showing an example of pen color selection buttons displayed on a display. [Figure 22] FIG. 10 is a diagram showing a display example of a handwriting input area. [Figure 23] FIG. 10 is a diagram showing an example of definition data of a handwriting input area. [Figure 24] FIG. 2 is a diagram illustrating an example of the configuration of a handwriting input area. [Figure 25] FIG. 10 is a diagram showing whether each control of handwriting recognition, selectable candidate display, stroke display, dialog control, and UI operation is enabled or disabled for each value that Recognition can take. [Figure 26] FIG. 10 is a diagram illustrating an example of a mask display area. [Figure 27] FIG. 10 is a diagram showing an example of definition data of a mask display area. [Figure 28] 10A and 10B are diagrams illustrating an example of an operation guide and selectable candidates displayed by the operation guide. [Figure 29] 10A and 10B are diagrams illustrating an example of designating a selected object; [Figure 30] 10A and 10B are diagrams showing an example of display of operation command candidates based on operation command definition data when a handwritten object is present; [Figure 31] 10A and 10B are diagrams showing an example of display of operation command candidates based on operation command definition data when a handwritten object is present; [Figure 32] FIG. 10 is a diagram illustrating a method for inputting 90-degree angle information. [Figure 33] FIG. 10 is a diagram illustrating a method for registering handwritten signature data. [Figure 34] FIG. 10 is a diagram showing an example of an operation guide that is displayed when a user handwrites "Suzuki," which is handwritten signature data that has been registered. [Figure 35] FIG. 10 is a diagram illustrating a method for changing user-defined data. [Figure 36] FIG. 2 is an example of a top view of a display device placed flat. [Figure 37] FIG. 2 is a diagram illustrating an example of a pen operation panel. [Figure 38] FIG. 10 is a diagram illustrating an example of a page operation panel. [Figure 39] FIG. 10 is a diagram illustrating an example of a page navigation operation panel. [Figure 40] FIG. 10 is a diagram showing an example of operation button definition data of the lower user operation menu (part 1). [Figure 41] FIG. 10 is a diagram showing an example of operation button definition data of the lower user operation menu (part 2). [Figure 42] FIG. 10 is a diagram showing an example of operation button definition data of the upper user operation menu (part 1). [Figure 43] FIG. 10 is a diagram showing an example of operation button definition data of the upper user operation menu (part 2). [Figure 44] FIG. 10 is a diagram illustrating an example of page navigation data. [Figure 45] FIG. 10 is a sequence diagram (part 1) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 46] FIG. 10 is a sequence diagram (part 2) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 47]FIG. 10 is a sequence diagram (part 3) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 48] FIG. 10 is a sequence diagram (part 4) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 49] FIG. 5 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 5). [Figure 50] FIG. 10 is a sequence diagram (part 6) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 51] FIG. 10 is a sequence diagram (part 7) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 52] FIG. 8 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 8). [Figure 53] FIG. 9 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 9). [Figure 54] 10A and 10B are diagrams illustrating a method in which a display device controls display of data depending on an area containing pen coordinates. [Figure 55] 10A and 10B are diagrams illustrating a method for a display device to control display of data in accordance with a mask display area that includes pen coordinates. [Figure 56] 10 is an example of a flowchart illustrating a flow of mask display and mask cancellation of a mask display area based on a mask display timer (1) and a mask display timer (2). [Figure 57] FIG. 10 is a diagram illustrating another configuration example of the display device. [Figure 58] FIG. 10 is a diagram illustrating another configuration example of the display device. [Figure 59] FIG. 10 is a diagram illustrating another configuration example of the display device. [Figure 60] FIG. 10 is a diagram illustrating another configuration example of the display device. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A display device and a display method performed by the display device will be described below as an example of an embodiment of the present invention with reference to the accompanying drawings. [Example]

[0010] <Comparative example of a display device> For convenience of explanation of the display device of this embodiment, a comparative example to the display device of this embodiment will be first explained.

[0011] FIG. 1 is a diagram illustrating an LCD protection privacy filter 102 or an anonymous database. FIG. 1(a) shows a PC (Personal Computer) 101 and an LCD protection privacy filter 102 attached to the PC 101. FIG. 1(b) shows the viewing angle of the LCD protection privacy filter 102. A privacy filter is a filter that allows light to pass easily in a direction perpendicular to the display, but makes it difficult for light to pass through in other directions, making the display appear dark. As shown in FIG. 1(b), the viewing angle of the LCD protection privacy filter 102 is approximately 60 degrees.

[0012] In the case of a personal PC 101, the user's eyes are in a vertical direction, so the screen becomes dark and difficult to see from the eyes of others, preventing peeping. However, in a device for group use that is used by multiple users simultaneously, a wider viewing angle is required, and the LCD protection privacy filter 102 cannot be used.

[0013] Figure 1(c) shows one record in the anonymous database. Personal information (hereafter referred to as confidential information), including email address 103, name 104, and kana name 105, is anonymized and displayed, and if you have administrator privileges, you can view the information before it is anonymized. With an anonymous database, confidential information is protected even without the LCD privacy filter 102, but since only users with administrator privileges can view the confidential information, it cannot be used for group purposes.

[0014] FIG. 1(d) shows an example of a user using the display device 2 in a flat position. Because this display device 2 has a large screen, multiple users can use it simultaneously. If the display device is installed in an open space without partitions, rather than in a conference room separated by walls, there is a risk that confidential information displayed on the screen may be spied on by third parties. Because this case is also for group use, the LCD privacy filter 102 and an anonymized database cannot be used.

[0015] Furthermore, on typical login screens where a user name and password are displayed, each character entered by the user from the keyboard is masked, making it impossible for the user to see the entire password. Also, even if a user tries to enter a password by hand, it is difficult to guess which characters constitute one character unless the display device converts them, making it difficult to mask each character.

[0016] As described above, in the comparative example, when confidential information is displayed on a display device for group use, it is difficult to achieve both protection and ease of use.

[0017] <Method for inputting confidential information in the display device of this embodiment> Therefore, the display device of this embodiment makes it possible to protect confidential information while making it easy to use, as shown in Fig. 2. Fig. 2 is a diagram illustrating an example of displaying confidential information. The display device 2 is divided into areas in advance, and the display mode of handwritten information can be controlled for each area.

[0018] In Figure 2(a), there are areas for entering name and gender. Name is highly confidential, but gender is not. The user handwrites the name in area 109 and the gender in area 111. Figure 2(b) shows the state after the name and gender have been handwritten. Candidate strings, which will be described later, are displayed for each, and the user can select from these to display in areas 109 and 111. In Figure 2(c), the name is displayed as "Yamada" and the gender as "Male."

[0019] After displaying "Yamada" in the predetermined name area 109, the display device 2 switches to a masked display of the character string after a certain time has passed. Therefore, the user can see the name they entered until the certain time has passed, as shown in FIG. 2(d). However, after the certain time has passed, the name area 109 switches to a masked display, as shown in FIG. 2(e). Therefore, the display device 2 can make it difficult for third parties to see confidential information.

[0020] Furthermore, even after a certain period of time has passed, the character string in the gender area 111 is not masked, allowing the user to check their gender at any time.

[0021] In this way, the display device 2 of this embodiment can be used for business purposes because it displays the entire confidential information for a short period of time immediately after the confidential information is displayed. Furthermore, the confidential information is masked after a certain period of time has passed, so it can be kept secret from third parties.

[0022] <Terminology> An input means can be any means that allows handwriting by specifying coordinates on a touch panel. Examples include a pen, a human finger or hand, or a rod-shaped object. A stroke is a series of operations in which a user presses the input means against the display, moves it continuously, and then lifts it from the display. Stroke data is information that is displayed on the display based on the trajectory of coordinates input by the input means. Stroke data may be interpolated as appropriate. Data handwritten using strokes is called stroke data. Handwritten data has one or more strokes, and the displayed object displayed on the display based on the stroke data is called an object.

[0023] A character string is one or more character codes converted from handwritten data. Characters include numbers, alphabets, and symbols.

[0024] Confidential information is information that should not be made known to unspecified persons. It may also be called secret information, personal information, or confidential information. Confidential information is entered into a predetermined area. However, the information entered by the user may be any type and is not limited to confidential information. In this embodiment, the information entered into the predetermined area is confidential information.

[0025] Concealment means keeping something secret and hidden. It is sufficient if the information is difficult for the user to read with the naked eye. If it is difficult to identify the content of the information, at least part of it may be concealed. In this embodiment, this is explained using the term mask display. Mask display means covering information, for example, replacing at least part of it with symbols such as "●" or "*". It may also be hidden (space).

[0026] Black and white device: A device that can only output black or white, or grayscale. Examples include electronic paper and monochrome printers. It can also be said to be a device that does not support color.

[0027] Color-compatible devices are devices that can output not only black and white but also color. These include devices with LCD or OLED displays, and color printers. Color-compatible devices are also black and white compatible devices.

[0028] Black and white emphasis device: A device that can only output black or white, or grayscale, but that displays handwritten data with black and white emphasis based on color information associated with the handwritten data. An example of such a device is the display device of this embodiment. Note that the display device is capable of handwritten input, and is sometimes called an input device or a handwriting input device.

[0029] Handwritten data refers to data that has been input by handwriting on a touch panel. It does not matter whether the data remains handwritten or is converted into text data after input. Data obtained from an external device also retains its handwritten origin. Handwritten data may include text data converted through character recognition, as well as data converted based on user operation, such as stamps displayed as fixed characters such as "Done" and "Confidential," shapes such as circles and stars, and straight lines.

[0030] Data not of handwritten origin refers to data other than data entered by handwriting on a touch panel, such as image data, handwritten data entered on a device that does not support black and white enhancement, or text data.

[0031] <Example of pen appearance> FIG. 3 shows an example of a perspective view of the pen 2500. FIG. 3 shows an example of a multi-function pen 2500. The pen 2500 has a built-in power source and can send commands to the display device 2, which is called an active pen (a pen without a built-in power source is called a passive pen). The pen 2500 in FIG. 3 has one physical switch at the tip, one at the end, and two on the side of the pen; the tip is for writing, the end is for erasing, and the side is for assigning user functions. In this embodiment, the pen also has non-volatile memory that stores a pen ID that is unique to other pens.

[0032] Furthermore, if the pen has a switch, it is possible to reduce the number of steps the user has to take to operate the display device 2. Pens with switches mainly refer to active pens, but since passive pens that do not have a built-in power supply can generate power using only an LC circuit in the electromagnetic induction method, this includes not only active pens but also passive pens that use the electromagnetic induction method. Pens with switches that use optical, infrared, and capacitive methods other than electromagnetic induction are also active pens.

[0033] The hardware configuration of the pen 2500 is assumed to be similar to that of a general control system equipped with a communication function and a microcomputer. The pen 2500 may be of an electromagnetic induction type or an active electrostatic coupling type. It may also have functions such as pressure detection, tilt detection, and a hover function (displaying the cursor before the pen touches the screen).

[0034] <Overall configuration of the device> The overall configuration of the display device 2 according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing the overall configuration of the display device 2. Fig. 4(a) shows, as an example of the display device 2, a display device 2 used as a horizontally long electronic whiteboard hung on a wall.

[0035] As shown in Fig. 4(a), a display 220 as an example of a display device is installed above the display device 2. A user U can handwrite (also referred to as input or drawing) characters and the like on the display 220 using a pen 2500.

[0036] FIG. 4(b) shows a display device 2 used as a vertically long electronic whiteboard hung on a wall.

[0037] 4(c) shows the display device 2 placed flat on a desk 230. The display device 2 is about 1 cm thick, so there is no need to adjust the height of the desk even if it is placed flat on a regular desk. In addition, it can be easily moved.

[0038] The tilt sensor automatically detects how the device is being used.

[0039] <Device hardware configuration> Next, the hardware configuration of the display device 2 will be described with reference to Fig. 5. As shown in the figure, the display device 2 has the configuration of an information processing device or a computer. Fig. 5 is an example of a hardware configuration diagram of the display device 2. As shown in Fig. 5, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an SSD (Solid State Drive) 204.

[0040] Of these, the CPU 201 controls the overall operation of the display device 2. The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201. The SSD 204 stores various data such as the OS and programs for the display device 2. Note that this program may be an application program that runs on an information processing device equipped with a general-purpose OS (Windows (registered trademark), Mac OS (registered trademark), Android (registered trademark), iOS (registered trademark), etc.). In other words, the display device 2 may be an information processing device.

[0041] The display device 2 also includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, a power switch 227, a tilt sensor 217, a serial interface 218, a speaker 219, a microphone 221, a wireless communication device 222, an infrared I / F 223, a power control circuit 224, an AC adapter 225, and a battery 226.

[0042] The display controller 213 controls and manages the screen display in order to output an output image to the display 220, etc. The touch sensor 216 detects that the pen 2500, the user's hand, etc. (the pen and the user's hand are input means) has come into contact with the display 220. The touch sensor 216 also receives a pen ID.

[0043] The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 inputs and detects coordinates. The method for inputting and detecting coordinates will be described below. For example, in the case of an optical type, two light emitting and receiving devices installed at both ends of the upper side of the display 220 emit a plurality of infrared rays parallel to the display 220. The infrared rays are reflected by a reflecting member installed around the display 220. The light receiving element receives the light returning along the same optical path as the emitted light.

[0044] The touch sensor 216 outputs position information of the infrared rays emitted by the two light receiving and emitting devices that are blocked by the object to the touch sensor controller 215, and the touch sensor controller 215 identifies the coordinate position of the contact position of the object. The touch sensor controller 215 also has a communication unit 215a and can communicate wirelessly with the pen 2500. For example, when communication is based on a standard such as Bluetooth (registered trademark), a commercially available pen can be used. If one or more pens 2500 are registered in the communication unit 215a in advance, the user can communicate with the display device 2 without having to perform connection settings to connect the pen 2500 to the display device 2.

[0045] The power switch 227 is a switch for switching ON / OFF the power of the display device 2. The tilt sensor 217 is a sensor for detecting the tilt angle of the display device 2. It is mainly used to detect whether the display device 2 is being used in the installation state shown in FIG. 4(a), FIG. 4(b), or FIG. 4(c), and can automatically change the thickness of characters, etc. depending on the installation state.

[0046] The serial interface 218 is an external communication interface such as a USB or LAN interface. It is used for inputting information from the outside. The speaker 219 is used for audio output, and the microphone 221 is used for audio input. The wireless communication device 222 communicates with a terminal carried by the user, and relays a connection to the Internet, for example. The wireless communication device 222 communicates using Wi-Fi, Bluetooth (registered trademark), or the like, but any communication standard is acceptable. The wireless communication device 222 forms an access point, and when the user sets the SSID (Service Set Identifier) ​​and password they have obtained in the terminal they carry, they can connect to the access point.

[0047] Preferably, the wireless communication device 222 is provided with two access points. a. Access point → Internet b. Access point → Internal network → Internet Access point a is for users outside the company, and users cannot access the company network but can use the Internet. Access point b is for users inside the company, and users can use the company network and the Internet.

[0048] The infrared I / F 223 detects the display device 2 placed adjacently. By utilizing the linearity of infrared rays, only the display device 2 placed adjacently can be detected. It is preferable to provide one infrared I / F 223 on each side, so that it is possible to detect in which direction another display device 2 is placed relative to the display device 2. Previously handwritten information (handwritten information on another page, with the area of ​​one display 220 being one page) can be displayed on the adjacent display device 2.

[0049] The power supply control circuit 224 controls an AC adapter 225 and a battery 226, which are power sources for the display device 2. The AC adapter 225 converts AC common to commercial power sources into DC.

[0050] If the display 220 is so-called electronic paper, it consumes little or no power to maintain an image after it has been drawn, and so can be driven by a battery 226. This makes it possible to use the display device 2 for purposes such as digital signage even in places where it is difficult to connect a power source, such as outdoors.

[0051] The display device 2 further includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201 shown in FIG.

[0052] The touch sensor 216 is not limited to an optical type, and various detection means may be used, such as a capacitive touch panel that identifies the contact position by detecting changes in capacitance, a resistive touch panel that identifies the contact position by detecting voltage changes across two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by an object touching the display unit. The touch sensor 216 may be of a type that does not require an electronic pen to detect whether or not the pen tip is touching. In this case, a fingertip or a pen-shaped stick can be used to perform a touch operation. The pen 2500 does not have to be a long, thin pen-shaped panel.

[0053] <About the device functions> Next, the functions of the display device 2 and the pen 2500 will be described with reference to FIG. 6. FIG. 6(a) is an example of a functional block diagram showing the functions of the display device 2 in blocks. The display device 2 includes a handwriting input unit 21, a display unit 22, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a character string conversion control unit 28, a character string conversion dictionary unit 29, a predictive conversion control unit 30, a predictive conversion dictionary unit 31, an operation command recognition control unit 32, an operation command definition unit 33, a pen ID control data storage unit 36, a file transmission / reception control unit 37, a handwritten signature authentication control unit 38, a handwritten signature data storage unit 39, and a mask display timer control unit 40. Each function of the display device 2 is realized by any of the components shown in FIG. 5 operating in response to an instruction from the CPU 201 in accordance with a program loaded from the SSD 204 onto the RAM 203.

[0054] The handwriting input unit 21 is realized by a touch sensor 216 or the like, accepts handwriting input by the user, and receives a pen ID from the pen 2500. The handwriting input unit 21 converts the user's pen input d1 into pen operation data d2 (pen up, pen down, or pen coordinate data) with the pen ID, and transmits it to the handwriting input display control unit 23. The pen coordinate data is periodically transmitted as discrete values, and coordinates between the discrete values ​​are interpolated.

[0055] The display unit 22 is realized by a display 220 or the like, and displays handwritten objects, operation menus, etc. The display unit 22 converts the drawing data d3 written to the video memory by the handwriting input display control unit 23 into data according to the characteristics of the display 220, and transmits the data to the display 220.

[0056] The handwriting input display control unit 23 performs overall control over handwriting input and display. The handwriting input display control unit 23 processes pen operation data d2 from the handwriting input unit 21 and transmits it to the display unit 22 for display. Details of the processing of the pen operation data d2 and the display of strokes will be explained later with reference to Figs. 45 to 53. The handwriting input display control unit 23 holds operation button definition data, which will be described later, detects an operation menu (button) operated based on the pen operation data d2 from the handwriting input unit 21, and performs control according to the operation menu.

[0057] The candidate display timer control unit 24 is a timer for controlling the display of selectable candidates. It starts or stops the timer to generate the timing for starting the display of selectable candidates and the timing for erasing the display. The selectable candidates are handwriting recognition character string / language character string candidates, conversion character string candidates, character string / predictive conversion candidates, and operation command candidates that are selectably displayed in an operation guide (see FIG. 28 ) described later. The candidate display timer control unit 24 receives a timer start request d4 (which may also be a timer stop request) from the handwriting input display control unit 23, and transmits a timeout event d5 to the handwriting input display control unit 23.

[0058] The handwriting input saving unit 25 has a storage function for saving user data (handwritten objects / character string objects). The handwriting input saving unit 25 receives user data d6-1 from the handwriting input display control unit 23 and saves it in the handwriting input saving unit 25, and receives an acquisition request d6-2 from the handwriting input display control unit 23 and transmits the user data d7 saved in the handwriting input saving unit 25. The handwriting input saving unit 25 transmits position information d36 of a confirmed object (a character string object that has already been recognized or a handwritten object that has been confirmed not to be converted) to the operation command recognition control unit 32.

[0059] The handwriting recognition control unit 26 is a recognition engine that performs online handwriting recognition. Unlike a typical OCR (Optical Character Reader), it recognizes characters (not only Japanese but also multiple languages ​​such as English), numbers, symbols (%, $, &, etc.), shapes (lines, circles, triangles, etc.) in parallel with the user's pen operation. Various algorithms have been devised for recognition methods, but this embodiment uses known techniques and will not be described in detail.

[0060] The handwriting recognition control unit 26 receives the pen operation data d8-1 from the handwriting input display control unit 23, performs handwriting recognition, and stores handwriting-recognized character string candidates. The handwriting recognition control unit 26 also stores language character string candidates converted from the handwriting-recognized character string candidates d12 using the handwriting recognition dictionary unit 27. Separately, when an acquisition request d8-2 is received from the handwriting input display control unit 23, the handwriting recognition control unit 26 transmits the stored handwriting-recognized character string candidates and language character string candidates d9 to the handwriting input display control unit 23.

[0061] The handwriting recognition dictionary unit 27 stores dictionary data for language conversion in handwriting recognition. The handwriting recognition dictionary unit 27 receives handwriting recognition character string candidates d12 from the handwriting recognition control unit 26, converts them into linguistically probable language character string candidates d13, and transmits them to the handwriting recognition control unit 26. For example, in the case of Japanese, hiragana characters are converted into kanji characters or katakana characters.

[0062] The character string conversion control unit 28 controls the conversion of conversion character string candidates into character strings. A conversion character string is a character string that is likely to be generated by including a handwriting recognition character string or a language character string. The character string conversion control unit 28 receives handwriting recognition character strings and language character string candidates d11 from the handwriting recognition control unit 26, converts them into conversion character string candidates using the character string conversion dictionary unit 29, and stores them. Separately, when an acquisition request d14 is received from the handwriting input display control unit 23, the character string conversion control unit 28 transmits the stored conversion character string candidates d15 to the handwriting input display control unit 23.

[0063] The character string conversion dictionary unit 29 stores dictionary data for character string conversion. The character string conversion dictionary unit 29 receives handwriting-recognized character strings and language character string candidates d17 from the character string conversion control unit 28, and transmits converted character string candidates d18 to the character string conversion control unit 28.

[0064] The predictive conversion control unit 30 receives the handwriting-recognized character string and language character string candidate d10 from the handwriting recognition control unit 26 and receives the converted character string candidate d16 from the character string conversion control unit 28, and converts each of them into a predicted character string candidate using the predictive conversion dictionary unit 31. A predicted converted character string is a character string that is likely to be generated by including a handwriting-recognized character string, a language string, or a converted character string. If an acquisition request d19 is separately received from the handwriting input display control unit 23, the predictive conversion control unit 30 transmits the predicted character string candidate d20 to the handwriting input display control unit 23.

[0065] The predictive conversion dictionary unit 31 stores dictionary data for predictive conversion. The predictive conversion dictionary unit 31 receives handwritten recognition character strings, language character string candidates, and conversion character string candidates d21 from the predictive conversion control unit 30, and transmits predicted character string candidates d22 to the predictive conversion control unit 30.

[0066] The operation command recognition control unit 32 receives the handwriting recognition character string and language character string candidates d30 from the handwriting recognition control unit 26, receives the converted character string candidates d28 from the character string conversion control unit 28, and receives the predicted character string candidates d29 from the predictive conversion control unit 30. Then, for each, it transmits an operation command conversion request d26 to the operation command definition unit 33 and receives the operation command candidates d27 from the operation command definition unit 33. The operation command recognition control unit 32 holds the operation command candidates d27.

[0067] If the operation command conversion request d26 partially matches the operation command definition, the operation command definition unit 33 sends operation command candidates d27 to the operation command recognition control unit 32.

[0068] Furthermore, the operation command recognition control unit 32 receives pen operation data d24-1 from the handwriting input display control unit 23 and transmits a request d23 to acquire position information of a confirmed object that was previously input and confirmed to the handwriting input storage unit 25. The operation command recognition control unit 32 stores the confirmed object designated by the pen operation data as a selected object (including position information). The operation command recognition control unit 32 identifies a selected object that satisfies the position of the pen operation data d24-1 and a predetermined criterion. Separately, when an acquisition request d24-2 is received from the handwriting input display control unit 23, the operation command recognition control unit 32 transmits a selected object d25 identified as a candidate for the stored operation command to the handwriting input display control unit 23.

[0069] The pen ID control data storage unit 36 ​​holds pen ID control data (which may be called storage means). Before the handwriting input display control unit 23 transmits display data to the display unit 22, the pen ID control data storage unit 36 ​​transmits pen ID control data d41 to the handwriting input display control unit 23. The handwriting input display control unit 23 draws the display data under the operating conditions stored in association with the pen ID. Furthermore, before the handwriting recognition control unit 26 performs handwriting recognition, the pen ID control data storage unit 36 ​​transmits angle information d44 of the pen ID control data to the handwriting recognition control unit 26. The handwriting recognition control unit 26 rotates the stroke using the angle information stored in association with the pen ID and performs handwriting recognition.

[0070] Furthermore, after the handwriting recognition control unit 26 recognizes a straight line for setting angle information when the user writes characters or the like by hand, the handwriting recognition control unit 26 transmits angle information d43 of the pen ID control data to the pen ID control data storage unit 36. The handwriting recognition control unit 26 stores the angle information d43 in the pen ID control data storage unit 36 ​​in association with the pen ID. Furthermore, after the handwriting input display control unit 23 executes an operation command for setting angle information, the handwriting input display control unit 23 transmits pen ID control data d42 to the pen ID control data storage unit 36. The handwriting input display control unit 23 stores the execution result of the operation command (angle information set by the user) in association with the pen ID. Thereafter, the stroke of that pen ID is rotated using the set angle information, and then handwriting recognition is performed.

[0071] Furthermore, the handwriting recognition control unit 26 transmits stroke data d49 rotated clockwise using the angle information of the pen ID control data to the handwritten signature authentication control unit 38. This makes it possible to authenticate a handwritten signature regardless of the user's operating position (the direction from which the handwriting is made relative to the display device 2).

[0072] The handwritten signature data storage unit 39 holds handwritten signature data. When the handwritten signature data storage unit 39 receives a handwritten signature data acquisition request d45 from the handwritten signature authentication control unit 38, it transmits handwritten signature data d46 to the handwritten signature authentication control unit 38. The format of the handwritten signature data depends on the handwritten signature authentication algorithm of the handwritten signature authentication control unit 38. The data in the handwritten signature data storage unit 39 will be described with reference to FIG. 15.

[0073] When handwritten signature authentication control unit 38 receives clockwise rotated stroke data d49 from handwriting recognition control unit 26, it sends a handwritten signature data acquisition request d45 to handwritten signature data storage unit 39. Handwritten signature data storage unit 39 sends handwritten signature data d46 to handwritten signature authentication control unit 38.

[0074] The handwritten signature authentication control unit 38 authenticates the user based on the handwritten signature data. Various algorithms have been devised for authenticating users based on handwritten signature data, but this embodiment uses a technology that can achieve a recognition rate that is practically acceptable. The handwritten signature authentication control unit 38 generates a feature vector whose elements include, for example, the coordinates, writing pressure, and stroke writing time that make up the handwritten signature data. The handwritten signature authentication control unit 38 appropriately weights the elements and compares the feature vector of the registered handwritten signature data with a feature vector such as the user name handwritten by the user when signing in. If the degree of match is equal to or greater than a threshold, authentication is determined to be successful; if it is less than the threshold, authentication is determined to be unsuccessful.

[0075] The handwritten signature authentication control unit 38 stores the handwritten signature authentication result, which is the comparison result between the stroke data d49 and the handwritten signature data d46, and when an acquisition request d48 is separately received from the handwritten input display control unit 23, the handwritten signature authentication control unit 38 transmits the stored handwritten signature authentication result d47 to the handwritten input display control unit 23. The handwritten signature authentication result indicates whether or not the stroke data d49 and the handwritten signature data d46 can be considered to match, and if they do match, includes a SignatureId (described later) that is associated with the matching handwritten signature data d46.

[0076] If the handwriting recognition result of handwriting recognition control unit 26 matches the operation command instructing the execution of handwritten signature registration, handwriting recognition control unit 26 obtains data d52 input into a handwritten signature registration form (a box into which handwritten signature data is input, as described below) from handwriting input storage unit 25. Handwriting recognition control unit 26 transmits handwritten signature data d50 from data d52 to handwritten signature authentication control unit 38. Handwritten signature authentication control unit 38 transmits the received handwritten signature data d50 to handwritten signature data storage unit 39 for registration.

[0077] If the handwriting recognition result of the handwriting recognition control unit 26 is an instruction to cancel the handwritten signature or to execute registration, the handwriting recognition control unit 26 sends a request d51 to delete the handwritten signature registration form to the handwriting input storage unit 25 and deletes the handwritten signature registration form from the handwriting input storage unit 25.

[0078] When the handwriting recognition result of handwriting recognition control unit 26 is an instruction to change the user-defined data, handwriting recognition control unit 26 acquires data d53 entered in the user-defined data change form from handwriting input storage unit 25, and sends change value d54 of data d53 to operation command definition unit 33 to change the user-defined data. User-defined data will be described with reference to FIG. 14.

[0079] If the handwriting recognition result of the handwriting recognition control unit 26 is an instruction to cancel or register the user-defined data change form, the handwriting recognition control unit 26 sends a request d55 to delete the user-defined data change form to the handwriting input storage unit 25. The handwriting recognition control unit 26 deletes the user-defined data change form from the handwriting input storage unit 25.

[0080] The file transmission / reception control unit 37 saves and acquires files of handwritten data or data not originating from handwritten data in a storage medium, communicates with external devices (print requests, display requests, etc.), etc. The file transmission / reception control unit 37 receives a request d64 to send or receive a file from the handwritten input display control unit 23. When saving or printing a file, the handwritten input display control unit 23 sends a file transmission request to the file transmission / reception control unit 37, and the file transmission / reception control unit 37 sends a request to acquire handwritten input saved data d61 to the handwritten input save unit 25. In the case of data that is not handwritten, the handwritten input storage unit 25 transmits the color information held in the data as is. In the case of data originating from handwriting, if the destination is a color-compatible device (for example, a color printer), the handwriting input storage unit 25 converts the data into color and transmits the converted handwriting input data d62 to the file transmission / reception control unit 37. In the case of data originating from handwriting, if the destination is a black-and-white compatible device (e.g., a monochrome printer), the handwriting input saving unit 25 converts the data to black-and-white and sends the handwriting input saved data d62 to the file transmission / reception control unit 37. Since the black-and-white compatible device may be capable of converting the data to grayscale, the data may be converted to color before being sent. If the destination is a device that supports black and white emphasis, the handwritten input saved data, which will be described later, is sent to the file transmission / reception control unit 37. Furthermore, when the handwriting input saving unit 25 saves handwritten data in a file, it converts the handwritten input saved data d62 into color in accordance with the file format and attaches the handwritten input saved data as metadata for the file. When saving data that is not handwritten in a file, it converts the handwritten input saved data d62 into color in accordance with the file format.

[0081] Whether a device is color-compatible or monochrome-compatible is stored, for example, in an MIB (Management Information Base) held by the network device, and the file transmission / reception control unit 37 can determine this by acquiring the MIB. Similarly, whether a device is monochrome-emphasis compatible can also be determined from the model name and the like published by the MIB.

[0082] When reading a file, the handwriting input display control unit 23 sends a file list acquisition request d65 to the file transmission / reception control unit 37, and the file transmission / reception control unit 37 sends the file list acquisition request to the external device. The file transmission / reception control unit 37 acquires a file list d63 and sends it to the handwriting input display control unit 23. The handwriting input display control unit 23 displays the file list on the display unit 22, and the handwriting input unit 21 sends the display position of the selected file to the handwriting input display control unit 23. The handwriting input display control unit 23 sends a file reception request d66 to the file transmission / reception control unit 37. The file transmission / reception control unit 37 acquires a file from the external device and sends this file d67 to the handwriting input storage unit 25. The handwriting input storage unit 25 analyzes the metadata of the file to determine whether the data is handwritten, and if it is handwritten, extracts handwritten input storage data (black and white highlight / color convertible data, described below). The handwritten data is converted to black and white highlight display, and if it is not handwritten data, it is sent to the handwriting input display control unit 23 without conversion (displayed in grayscale). The handwriting input display control unit 23 transmits display data of the handwritten object to the display unit 22 .

[0083] The mask display timer control unit 40 controls the mask display timer of the mask display area based on the definition data of the mask display area (see FIG. 7). The mask display timer is managed for each divided mask display area. The operation of one mask display area will be described below as an example. When the handwriting input display control unit 23 transmits the handwriting-recognized character string object display data to the display unit 22, it transmits the mask display timer (1) defined in the definition data of FIG. 7 to the mask display timer control unit 40 (d71). The mask display timer control unit 40 starts the mask display timer. When the handwriting input unit 21 detects pen-down on the mask display area, the mask display timer control unit 40 resumes the mask display timer. This extends the mask display timer while input is being made into the mask display area. When the mask display timer times out, the mask display timer control unit 40 transmits a timeout to the handwriting input display control unit 23 (d72). The handwriting input display control unit 23 erases the character string object and transmits mask display data of the character string object to the display unit 22. The display unit 22 switches the character string object to a masked display (masking confidential information to hide it). Furthermore, when the handwriting input unit 21 detects a pen event on the mask display area during mask display, the handwriting input display control unit 23 transmits a gesture recognition request (tap, double tap, etc.) to the handwriting recognition control unit 26. The handwriting recognition control unit 26 transmits the gesture recognition result to the handwriting input display control unit 23. If the gesture received from the handwriting recognition control unit 26 is a mask display release gesture, the handwriting input display control unit 23 erases the mask display data of the character string object and transmits character string object display data to the display unit 22. The display unit 22 displays the character string object (displays confidential information) and transmits a mask display timer (2) defined in the definition data in FIG. 7 to the mask display timer control unit 40. The mask display timer control unit 40 starts the mask display timer.

[0084] This gesture is handwritten data that determines whether the user has input to redisplay confidential information. The determination content differs from the gesture that the display device 2 uses to recognize angle information from a straight line, which will be described later.

[0085] 6(b) is a functional block diagram showing the functions of the pen 2500. The pen 2500 has a pen event transmission unit 41. The pen event transmission unit 41 attaches a pen ID to event data of pen up, pen down, and pen coordinates and transmits them to the display device 2.

[0086] <About predefined control data> Next, the predefined control data used by the display device 2 for various processes will be described with reference to Fig. 7. Fig. 7 shows an example of the predefined control data. In the example of Fig. 7, control data is shown for each control item.

[0087] The selectable candidate display timer 401 defines the time until the selectable candidates are displayed. This is because the selectable candidates are not displayed while handwriting. In FIG. 7, this means that the selectable candidates are displayed if the pen does not come down within TimerValue=500 [ms] after the pen is up. The selectable candidate display timer 401 is held by the candidate display timer control unit 24. The selectable candidate display timer 401 is used when the selectable candidate display timer starts in step S18-2 of FIG. 47, which will be described later.

[0088] The selectable candidate deletion timer 402 defines the time until the displayed selectable candidates are deleted. This is to delete the selectable candidates if the user does not select a selectable candidate. In FIG. 7, this means that if a selectable candidate is not selected within TimerValue=5000 [ms] from the display of the selectable candidates, the selectable candidate display data is deleted. The selectable candidate deletion timer 402 is held by the candidate display timer control unit 24. The selectable candidate deletion timer 402 is used when the selectable candidate display deletion timer starts in step S64 of FIG. 49.

[0089] The handwritten object neighborhood rectangular area 403 defines a rectangular area considered to be near the handwritten object. In the example of FIG. 7, the handwritten object neighborhood rectangular area 403 is a rectangular area obtained by enlarging the handwritten object rectangular area by 50% (Horizontal) of the estimated character size in the horizontal direction and 80% (Vertical) of the estimated character size in the vertical direction. In the example of FIG. 7, the rectangular area is a percentage of the estimated character size (specified as a %), but it can also be a fixed length by using units such as "mm". The handwritten object neighborhood rectangular area 403 is stored in the handwritten input storage unit 25. The font data 405 is used to determine the overlap state between the handwritten object neighborhood rectangular area and the stroke rectangular area in step S10 of FIG. 46.

[0090] The estimated writing direction / character size determination condition 404 defines constants for determining the writing direction and the measurement direction of the character size. In the example of FIG. 7, it is determined that the difference between the time when the first stroke was added to the rectangular area of ​​the handwritten object and the time when the last stroke was added is MinTime=1000 [ms] or more, and the difference between the horizontal distance (width) and vertical distance (height) of the rectangular area of ​​the handwritten object is MinDiff=10 [mm] or more. If the horizontal distance is longer than the vertical distance, it means that the estimated writing direction is determined to be "horizontal writing" and the estimated character size is determined to be the vertical distance. If the horizontal distance is shorter than the vertical distance, it means that the estimated writing direction is determined to be "vertical writing" and the estimated character size is determined to be the horizontal distance. If these conditions are not met, it is determined that the estimated writing direction is "horizontal writing" (DefaultDir="Horizontal") and the estimated character size is determined to be the longer of the horizontal distance and the vertical distance. The estimated writing direction / character size determination condition 404 is held in the handwriting input storage unit 25. The estimated writing direction / character size judgment condition 404 is used to obtain the estimated writing direction in step S59 of FIG. 49 and to obtain the character string object font in step S81 of FIG.

[0091] Font data 405 defines data for estimating the size of characters, etc. In the example of FIG. 7, this means that the estimated character size determined by estimated writing direction / character size determination condition 404 is compared with small characters 405a (hereinafter referred to as minimum font size) and large characters 405c (hereinafter referred to as maximum font size) of font data 405. If the estimated character size is smaller than the minimum font size, the estimated character size is determined to be the minimum font size. If the estimated character size is larger than the maximum font size, the estimated character size is determined to be the maximum font size. Otherwise, the character size is determined to be that of medium characters 405b. Font data 405 is held by handwriting input storage unit 25. Font data 405 is used to obtain the character string object font in step S81 of FIG. 51.

[0092] Specifically, the handwriting input saving unit 25 compares the estimated character size determined by the estimated writing direction / character size determination condition 404 with the FontSize of the font data 405, and uses the font with the closest size. For example, if the estimated character size is 25 mm (FontSize of small characters) or less, the handwriting input saving unit 25 determines it as a "small character." If the estimated character size is more than 25 mm and less than 50 mm (FontSize of medium characters), the handwriting input saving unit 25 determines it as a "medium character." If the estimated character size is more than 100 mm (FontSize of large characters), the handwriting input saving unit 25 determines it as a "large character." A 25 mm Mincho font (FontStyle="Mincho" FontSize="25 mm") is used for the "small characters" 405a. A 50 mm Mincho font (FontStyle="Mincho" FontSize="50 mm") is used for the "medium characters" 405b. A 100mm Gothic font (FontStyle="Gothic" FontSize="100mm") is used for "large characters" 405c. To meet the need for more types of font sizes or styles, the display device provider can simply increase the types of font data 405.

[0093] The crossing line determination condition 406 defines data used to determine whether multiple objects have been selected. The handwritten object is a single stroke. In the example of FIG. 7, if there is an object whose long side length is 100 mm or more (MinLenLongSide="100mm"), whose short side length is 50 mm or less (MaxLenShortSide="50mm"), and whose overlap rate with the handwritten object in the long side direction and short side direction is 80% or more (MinOverLapRate="80%"), it is determined that multiple objects have been selected (selected objects). The crossing line determination condition 406 is held by the operation command recognition control unit 32. The crossing line determination condition 406 is used in the crossing line determination for determining the selected object in step S50 of FIG. 48.

[0094] The encircling line determination condition 407 defines data used to determine whether an object is an encircling line. In the example of Fig. 7, the operation command recognition control unit 32 determines a fixed object, whose overlap rate in the long side direction and short side direction of the handwritten object is 100% or more (MinOverLapRate="100%"), to be a selected object. The encircling line determination condition 407 is held by the operation command recognition control unit 32. The encircling line determination condition 407 is used in the encircling line determination for determining the selected object in step S50 of Fig. 48.

[0095] Note that either the cross-line determination condition 406 or the enclosing line determination condition 407 may be given priority in the determination. For example, if the cross-line determination condition 406 is set to a relaxed value (to make it easier to select a cross-line) and the enclosing line determination condition 407 is set to a strict value (to make it possible to select only an enclosing line), the operation command recognition control unit 32 may give priority to the enclosing line determination condition 407 in the determination.

[0096] The mask display timer (1) 408 is a timer that starts from when the display unit 22 displays a character string object until it switches to masked display. It is used to start the mask display timer in step S83 of FIG. 52. The mask display timer (1) 408 indicates that if there is no pen-down in the masked display area for 5 seconds (TimerValueForMaskedDisplay1="5 sec"), it will switch to masked display (an example of a first time period). The mask display timer (1) 408 starts immediately after the first character string object is input, and in this example, it is set to a short time that assumes the input time until input of information related to this character string object (for example, one area's worth) is completed.

[0097] The mask display timer (2) 409 is a timer that is used to count down the time until the display unit 22 switches back to the mask display after the mask display has been canceled. The mask display timer (2) 409 is used to start the mask display timer in step S94 of FIG. 52. The mask display timer (2) 409 indicates that if there is no pen-down on the mask display area for 30 seconds (TimerValueForMaskedDisplay2="30 sec") after the mask display is canceled, the display will be switched back to the mask display (an example of a second time period). The mask display timer (2) 409 starts immediately after the first character string object is input and the display has been switched to the mask display at least once. This is a longer time that assumes the time it will take the user to finish comparing or verifying information different from the character string object. For this reason, TimerValueForMaskedDisplay1 and TimerValueForMaskedDisplay2 are different.

[0098] The mask display timer (1) 408 and the mask display timer (2) 409 can be arbitrarily set by the user from a setting screen or the like displayed on the display device 2. For example, the mask display timer (1) 408 may be greater than the mask display timer (2) 409, or the mask display timer (1) 408 or the mask display timer (2) 409 may be disabled (infinite).

[0099] The mask display cancel gesture 410 defines a gesture by which the user cancels the mask display. The mask display cancel gesture 410 is used to determine the gesture recognition result returned in response to the gesture recognition request in steps S90 and S91 of FIG. 52. The mask display cancel gesture 410 is a tap (GestureForCancelMaskedDisplay="Tap"), i.e., when the user taps (presses and immediately releases with a pen) within the masked display area, the mask display is canceled. The gesture determination method can be performed in the same way as character recognition, such as when the recognition result is "," = tap (dot) and "゛" = double tap (voiced consonant). In other words, when predetermined handwritten data is input by the input means, the handwriting input display control unit 23 displays the concealed character string again.

[0100] If the user taps the screen incorrectly, the user can change the gesture to a more complex one. For example, the user can select a command to change the gesture by handwriting the operation command displayed in the operation guide 500. Then, a gesture input field is displayed, allowing the user to register the desired gesture.

[0101] <Example of dictionary data> The dictionary data will be described with reference to Fig. 8 to Fig. 10. Fig. 8 is an example of dictionary data in handwriting recognition dictionary unit 27, Fig. 9 is an example of dictionary data in character string conversion dictionary unit 29, and Fig. 10 is an example of dictionary data in predictive conversion dictionary unit 31. These dictionary data are used in steps S33 to S41 in Fig. 48, respectively.

[0102] In this embodiment, the conversion results using the dictionary data of the handwriting recognition dictionary unit 27 in FIG. 8 are called language string candidates, the conversion results using the dictionary data of the string conversion dictionary unit 29 in FIG. 9 are called converted string candidates, and the conversion results using the dictionary data of the predictive conversion dictionary unit 31 in FIG. 10 are called predicted string candidates. For each dictionary data, "before conversion" represents the string to be searched for in the dictionary data, "after conversion" represents the converted string corresponding to the string to be searched for, and "probability" represents the probability of selection by the user. The probability is calculated from the results of users selecting each string in the past. Therefore, the probability may be calculated for each user. Various algorithms have been devised for calculating probability, but the calculation may be performed using an appropriate method as appropriate, and details will be omitted. This embodiment is characterized by displaying string candidates in descending order of selection probability based on the estimated writing direction.

[0103] The dictionary data of the handwriting recognition dictionary unit 27 in Figure 8 shows that handwritten "gi" is converted to "gi" with a probability of 0.55 and to "gi" with a probability of 0.4, and handwritten "gishi" is converted to "gishi" with a probability of 0.5 and to "gishi" with a probability of 0.45. The same applies to other "before conversion" character strings. In Figure 8, the "before conversion" character string is handwritten hiragana, but characters other than hiragana may also be registered as "before conversion."

[0104] 9 indicates that the character string "" is converted to "" with a probability of 0.95, and the character string "" is converted to "" with a probability of 0.85. The same applies to other "before conversion" character strings.

[0105] The dictionary data of the predictive conversion dictionary unit 31 in Fig. 10 indicates that the character string "minutes" is converted to "addressee of the minutes" with a probability of 0.65, and the character string "skill test" is converted to "approval of skill test" with a probability of 0.75. In the example of Fig. 10, all character strings before conversion are kanji, but characters other than kanji may also be registered.

[0106] The dictionary data is not language-dependent, and any character strings may be registered before and after conversion.

[0107] <Operation command definition data held by the operation command definition section> Next, the operation command definition data used by the operation command recognition control unit 32 will be described with reference to Fig. 11. Fig. 11 shows an example of the operation command definition data and system definition data held by the operation command definition unit 33.

[0108] Fig. 11(a) shows an example of operation command definition data. The operation command definition data in Fig. 11(a) is an example of operation command definition data when no selected object is selected by a handwritten object, and covers all operation commands that can be executed by the display device 2. The operation command in Fig. 11(a) has an operation command name (Name), a string (String) that partially matches a string candidate, and an operation command string (Command) to be executed. "%~%" in the operation command string is a variable, and is associated with system definition data as shown in Fig. 11(b). In other words, "%~%" is replaced with the system definition data shown in Fig. 11(b).

[0109] First, the operation command definition data 701 indicates that the operation command name is "Read minutes template," the string that partially matches the string candidate is "minutes" or "template," and the operation command string to be executed is "ReadFile https: / / %username%:%password%@server.com / template / minutes.pdf." In this example, the operation command string to be executed contains the system definition data "%~%," and "%username%" and "%password%" are replaced by the system definition data 704 and 705, respectively. Therefore, the operation command string to be ultimately executed is the string "ReadFile https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf," which indicates that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf" is to be read (ReadFile).

[0110] The operation command definition data 702 indicates that the operation command name is "Save to minutes folder," the strings that partially match the string candidate are "minutes" or "save," and the operation command string to be executed is "WriteFile https: / / %username%:%password%@server.com / minutes / %machinename%_%yyyy-mm-dd%.pdf." As with the operation command definition data 701, "%username%," "%password%," and "%machinename%" in the operation command string are replaced with system definition data 704 to 706, respectively. Note that "%yyyy-mm-dd%" indicates that it should be replaced with the current date. For example, if the current date is September 26, 2018, it should be replaced with "2018-09-26." The final operation command to be executed is "WriteFile https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf", which indicates that the minutes should be saved (WriteFile) in the file "https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf".

[0111] The operation command definition data 703 indicates that the operation command name is "Print", the string that partially matches the string candidate is "Print" or "Print", and the operation command string to be executed is "PrintFile https: / / %username%:%password%@server.com / print / %machinename%-"%yyyy-mm-dd%.pdf". When the operation command string is replaced in the same way as in the operation command definition data 702, the operation command to be executed finally becomes "PrintFile https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf", which indicates that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf" will be printed (PrintFile). In other words, the file is sent to the server. When the user makes the printer communicate with the server and specifies a file, the printer prints the contents of the file on paper.

[0112] In this way, the operation command definition data 701 to 703 can be identified from the character string candidates, and the operation command can be displayed by the user's handwriting. Also, if the user authentication is successful, "%username%", "%password%", etc. in the operation command definition data are replaced with the user information, making it possible to input and output files associated with the user.

[0113] If user authentication is not performed (including the case where authentication fails but the user can still use the display device 2), the display device 2 replaces the username and password with the pre-set username and password of the display device 2. Therefore, file input and output can be performed in association with the display device 2 without user authentication.

[0114] Operation command definition data 709, 710, 711, 720, 721, 722, 723, 724, 725, and 726 are operation commands for changing the pen color. Pen color refers to the color of handwritten data entered with the pen being used by the user. The operation command names of operation command definition data 709, 710, 711, 720, 721, 722, 723, 724, 725, and 726 are "black pen," "red pen," "blue pen," "green pen," "magenta pen," "cyan pen," "fine black pen," "fine red pen," "fine blue pen," and "fine green pen." For example, in the case of "black pen" and "fine black pen," if the character strings that partially match the character string candidates are "kuro" (black) and "pen" (pen), and the user writes "kuro" (black), only "black pen" and "fine black pen" are displayed as operation command candidates. On the other hand, "pen" corresponds to a character string (String) that partially matches a character string candidate such as "red pen," so if the user writes "pen," the operation command candidates displayed are "black pen" to "cyan pen" and "thin black pen" to "thin green pen." When these operation commands are executed, as shown in Fig. 17, the control data associated with the pen ID of the pen 2500 used by the user for operation is updated, and the pen color of this pen ID is set to ColorId of the pen ID control data (for example, Black in "Command="ChangePen Black"" becomes ColorId).

[0115] In the operation command definition data 719, the operation command name is "Read File," the character strings (String) that partially match the character string candidates are "File," "Read," and "Read," and the operation command to be executed is "ReadFile https: / / %username%:%password%@server.com / files / ." When substituted with the system definition data in the same way as the operation command definition data 702, the operation command becomes "ReadFile https: / / taro.tokkyo:x2PDHTyS@server.com / files / ," which indicates that the file at this address (folder) is to be read (ReadFile). When this operation command is executed, the display device 2 displays a file selection window such as that shown in FIG. 12. The file selection window in FIG. 12 displays thumbnails of the first page of the files stored at this address (folder).

[0116] FIG. 12 is an example of file list display data displayed on the display unit 22 in step S108 (file list display data) in FIG. 53. FIG. 12(a) is created from the file list information acquired by the file list information acquisition in step S105 in FIG. 53. The file list information has a list of file addresses, and the display unit 22 extracts file names and thumbnails from each file address and displays them in ascending order of file name. In FIG. 12, four thumbnails and file names are displayed, and left and right arrow icons 99 are used to display the previous and next thumbnails and file names. When a displayed thumbnail or file name is pressed with a pen, the display unit 22 clears the screen shown in FIG. 12, and the file transmission / reception control unit 37 receives the file from the file address pressed with the pen.

[0117] When the close button 98 in the upper right corner is pressed with a pen, the display unit 22 erases the screen shown in FIG. 12, and execution of the operation command definition data 719 is canceled. When the PDF file shown in FIGS. 12(a) and 12(d) is selected, the file transmission / reception control unit 37 receives the PDF file and stores it in the handwriting input storage unit 25. The file transmission / reception control unit 37 analyzes the PDF file. In the case of handwritten text data (with metadata), the file transmission / reception control unit 37 stores the data as handwritten input storage data (black and white emphasis and color convertible data) such as handwritten input storage data 801 to 805 shown in FIG. 16. Black and white emphasis and color convertible data is data that can be both black and white emphasis and converted to color (mainly data derived from handwriting). Text data not derived from handwriting is converted to handwritten input storage data such as handwritten input storage data 806 shown in FIG. 16, stored, and displayed on the display 220.

[0118] When the PDF file of Figure 12(b)(c) is selected, the file transmission / reception control unit 37 receives the image file, stores it in the handwriting input storage unit 25 as handwriting input storage data such as handwriting input storage data 807 in Figure 16, and displays it on the display 220.

[0119] Returning to FIG. 11, the explanation will be given. The operation command definition data 712 is an operation command for aligning the direction of text data in a certain direction. The operation command name of the operation command definition data 712 is "align text direction", the character string that partially matches the character string candidate is "text", "orientation", or "direction", and the operation command character string is "AlignTextDirection". Text data written by the user in a direction other than upside-down is in various directions, making it difficult to read the entire data from a single direction. When the user executes the operation command definition data 712, the display device 2 aligns the handwritten and recognized character string in the same direction (for example, upside-down). "Aligning" in this case means rotating the text data by the angle information.

[0120] The operation command definition data 713 indicates that the operation command name is "Register handwritten signature", the character strings that partially match the character string candidate are "Sign" and "Register", and the operation command character string is "RegistSignature". When the RegistSignature command is executed, a handwritten signature registration form is added to the handwritten input storage unit 25 and is displayed on the screen. An example of the handwritten signature registration form will be described later (see FIG. 33).

[0121] The operation command definition data 714 indicates that the operation command name is "Sign in by handwriting," the character string partially positioned as the character string candidate is "%signature%," and the operation command is "Signin." Here, "%signature%" is a reserved word in the system definition data, and indicates that the registered handwritten signature data matches the stroke data such as the user name. In other words, if there is a match, the operation command 512 based on the operation command definition data 714 is displayed in the operation guide 500 (see FIG. 34).

[0122] When the Signin command is executed, the AccountId of the user who has the matching SignatureId of the handwritten signature data is saved in the pen ID control data of the pen 2500 that handwritten the stroke data such as the user name. This associates the pen ID with the AccountId, and the display device 2 can use the user-defined data identified by this AccountId (see FIG. 17(a)).

[0123] The operation command definition data 715 indicates that the operation command name is "Sign out by handwriting", the character strings that partially match the character string candidates are "Sign" or "Out", and the operation command is "Signout". When the Signout command is executed, the AccountId is deleted from the pen ID control data of the pen 2500 that operated the handwriting signout. This removes the association between the pen ID and the AccountId, allowing any user to use the pen 2500.

[0124] The operation command definition data 716 indicates that the operation command name is "Change settings," the character strings that partially match the character string candidates are "Settings" or "Change," and the operation command is "ConfigSettings." When the ConfigSettings command is executed, a user-defined data change form is added to the handwritten input storage unit 25 and displayed on the screen. The user-defined data change form will be described later (see FIG. 35).

[0125] The operation command definition data 741 indicates that the operation command name is “Enter confidential information,” the character string that partially matches the character string candidate is “Lock” or “Confidential information,” and the operation command is “RecognitionControl.” When the RecognitionControl command is executed, the form of the input window 110 is added to the handwritten input storage unit 25, and the input window 110 is displayed on the screen (see FIG. 22).

[0126] Next, we will explain the operation command definition data when there is a handwritten object. Fig. 13 shows an example of operation command definition data when there is a selected object selected by a handwritten object. The operation command definition data in Fig. 13 has an operation command name (Name), a group name (Group) of operation command candidates, and an operation command string to be executed (Command).

[0127] The operation command definition data 707 defines operation commands for editing (Group="Edit"), and is an example of definition data for the names of operation commands for editing: "erase," "move," "rotate," and "select." In other words, these operation commands are displayed for the selected object, allowing the user to select the desired operation command.

[0128] The operation command definition data 708 defines operation commands for the decoration system (Group="Decorate"), and is an example of definition data for the names of operation commands for the decoration system: "bold", "thin", "larger", "smaller", and "underline". These operation commands are displayed for the selected object, allowing the user to select the desired operation command. In addition, color operation commands may also be displayed.

[0129] Therefore, when the user selects a selection object with a handwritten object, the operation command definition data 707 and 708 are specified, and the user can display the operation command by handwriting.

[0130] The operation command definition data 731 is an operation command without grouping (Group="None"), and is an example of data with the operation command name "Set to page name." Display examples of this operation command are shown in Figs. 30 and 31. When the operation command definition data 731 is selected, the display device 2 executes "SetPageName" on the selected object, and sets the character string of the selected object as the page name of the currently displayed page. This page name is displayed in real time on the page navigation.

[0131] <User-defined data> Next, user-defined data will be described with reference to Fig. 14. Fig. 14 shows an example of user-defined data held by operation command definition unit 33. The user-defined data in Fig. 14 is an example of definition data for one user. In user-defined data 717, AccountId is user identification information that is automatically assigned a number for each user, AccountUsername and AccountPassword are a user name and password, SignatureId is handwritten signature data identification information that is automatically assigned a number when handwritten signature data is registered, and username, password, and machinename are character strings that are set in operation command definition data 701 to 703 instead of system definition data 704 to 706, respectively. This makes it possible to execute operation commands using user-defined data.

[0132] When a user signs in by handwriting a user name or the like, the pen ID and AccountId are associated with the pen ID control data (see FIG. 17(a)). A character string in the user-defined data having an AccountId associated with the pen ID of the pen 2500 used by the user is used when an operation command is executed. After the user signs out, the character string in the system-defined data is used when an operation command is executed, even if the pen 2500 used to sign in is used.

[0133] User-defined data 718 is data used in the user-defined data change form. Name is the item name of AccountUsername, AccountPassword, username, password, or machinename in the user-defined data 717, and Data is the changed value of AccountUsername, AccountPassword, username, password, or machinename. In this example, the data for "Name" is "%AccountName%", the data for "Password" is "%AccountPassword%", the data for "Folder User Name" is "%username%", the data for "Folder Password" is "%password", and the data for "Folder File Name" is "%machinename", which correspond to each item in the user-defined data 717. These items entered in the user-defined data change form are reflected in the user-defined data 717.

[0134] It is also possible for the user to register color definition data, which will be described later, in user-defined data 717, and the user can input color definition data that he or she has defined himself or herself.

[0135] <Handwritten signature data> Next, the handwritten signature data will be described with reference to Fig. 15. Fig. 15 shows an example of handwritten signature data stored in the handwritten signature data storage unit 39. The handwritten signature data has Data representing the handwritten signature associated with SignatureId. SignatureId is identification information that is automatically assigned when the handwritten signature data is registered, and Data is data calculated by the handwritten signature authentication algorithm of the handwritten signature authentication control unit 38 from the stroke data received from the handwritten signature authentication control unit 38.

[0136] <Handwritten input data saved by the handwritten input saving unit> Next, the handwritten input saved data will be described with reference to Fig. 16. Fig. 16 shows an example of handwritten input saved data saved in the handwritten input saving unit 25. One line in Fig. 16 represents one stroke, one sentence (text), or one image.

[0137] When Type is Stroke, one piece of saved handwriting input data has the following items: DataId, Type, PenId, ColorId, Angle, StartPoint, StartTime, EndPoint, EndTime, Point, and Pressure. DataId is stroke identification information. Type is the type of saved handwriting input data. Types include Stroke, Group, Text, and Image. The type of saved handwriting input data 801 and 802 is Stroke, the type of saved handwriting input data 803 is Group, the type of saved handwriting input data 804, 805, and 806 is Text, and the type of saved handwriting input data 807 is Image.

[0138] Group means grouping other strokes, and handwriting input saved data with a type of Group specifies the stroke to be grouped by DataId. PenId, ColorId, and Angle are transcriptions of the pen ID control data described below. StartPoint is the coordinate of the start point of the stroke, and StartTime is the time at which the stroke started. EndPoint is the coordinate of the end point of the stroke, and EndTime is the time at which the stroke ended. Point is the coordinate sequence from the start point to the end point, and Pressure is the pen pressure from the start point to the end point. As shown in Angle, handwriting input saved data 804 and 805 were rotated clockwise by 180 degrees and 270 degrees, respectively, before being recognized as handwriting.

[0139] Furthermore, handwritten input saved data 804, 805, and 806 whose Type is Text have FontName, FontSize, and Text, where FontName is the font name of the text, FontSize is the character size, and Text is the content of the text (character code).

[0140] If there is no FontName in the pen ID control data, the handwriting input saving unit 25 copies the FontStyle of the font data 405 of the defined control data in Fig. 7 to the saved handwriting input data. The FontSize is copied from the font data 405 of the defined control data in Fig. 7.

[0141] The handwritten input save data 801-805 may be attached as metadata as it is when the file is saved as handwritten data. Therefore, the handwritten input save data 801-805 can be acquired by the display device 2 when the file is read. When the display device 2 transmits the handwritten input save data to an external device, the handwritten input save data is converted to color for color-compatible devices, and may be transmitted without or after conversion to color for monochrome-compatible devices. When transmitting to a monochrome emphasis-compatible device, the handwritten input save data is simply transmitted. In this way, the handwritten input save data 801-805 supports both monochrome emphasis and color conversion.

[0142] While handwritten input saved data 801 to 805 are data derived from handwriting, handwritten input saved data 806 and 807 are not. These are files loaded using a file load command. This is determined by whether or not a color value defined by the color definition data (described later) is registered in ColorId. For example, the ColorId of handwritten input saved data 801 is "Black," while the ColorId of handwritten input saved data 806 is "#e6001200." This is expressed as # and eight hexadecimal digits, with each pair of digits representing the values ​​of R (red), G (green), B (blue), and A (transparency).

[0143] In this way, in text data originating from handwriting, ColorId contains a string of characters that represents color information, but in text data that is not originating from handwriting, ColorId is "#color value." In this way, whether or not text data is originating from handwriting can be determined by focusing on ColorId.

[0144] The handwritten input saving unit 25 saves the color of the text data of the file read by the file transmission / reception control unit 37 as ColorId="#e6001200". Therefore, the handwritten input saved data 806 does not support black and white emphasis, but only supports color conversion.

[0145] The handwriting input display control unit 23 determines whether the data is handwritten by using ColorId, and if it is not handwritten, calculates and displays a grayscale value from RGBA. If the display device 2 is a color-compatible device, it displays the RGBA as is.

[0146] The handwritten input saved data 807 is also an image file read from a storage medium by a file read command. If the Type is Image, the handwritten input saved data 807 has a FileId and a FileName. The FileId is a management number within the handwritten input saved data, and the FileName is the original file name. For example, if the file transmission / reception control unit 37 reads the file "color_chart.pdf" in FIG. 12(a), the RGBCMY text in the center is saved in the handwritten input saved unit 25 in the format of the handwritten input saved data 806. The ring-shaped color portion is saved in the handwritten input saved unit 25 in the format of the handwritten input saved data 807. Note that handwritten input saved data with a Type of Image is obviously not of handwritten origin. Therefore, it is also possible to determine whether the data is of handwritten origin based on the Type.

[0147] In Figure 16, the value of ColorId is used to determine whether the data is handwritten, but whether it is handwritten or not can also be recorded using a dedicated flag or other means. Also, although this is limited to handwritten data with a Type of stroke, it is also possible to determine whether it is handwritten or not from its shape using machine learning. In this case, the learning device uses deep learning or other methods to learn the correspondence between training data that indicates whether it is handwritten or not and the character shape, and outputs whether the input character shape is handwritten or not.

[0148] In addition, data derived from handwriting may include not only text data converted from handwritten data through character recognition, but also data converted based on user operations, such as stamps displayed as fixed characters such as "Done" and "Confidential," shapes such as circles and stars, and straight lines.

[0149] The handwritten input saved data is used in step S7 of FIG. 45 (pen coordinates and reception time), step S102 of FIG. 53 (acquisition of handwritten input saved data), step S110 of FIG. 53 (save file), and so on.

[0150] <Pen ID control data stored in the pen ID control data storage unit> Next, the pen ID control data will be described with reference to Fig. 17. Fig. 17 is a diagram illustrating pen ID control data 901 to 904 stored in the pen ID control data storage unit 36. The pen ID control data controls the color of handwritten data, etc. Fig. 17 shows four pens 2500. Four is just an example, and one or more pens may be used.

[0151] One line in Figure 17(a) shows the pen ID control data for one pen. Figure 17(b) is a diagram explaining angle information when a user writes by hand on the display device 2. The angle information corresponds to the operation position, and can be said to be the angle of the direction in which the user is present, the angle of the direction in which the pen is used, or the angle related to the rotation of characters handwritten by the user. With a predetermined direction of the display device 2 (for example, the vertical direction) as 0 degrees (reference), the angle information for each user is 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees, and 315 degrees counterclockwise.

[0152] Note that the angle information is not automatically determined by the user's operation position, but is set by each user. There are two methods for setting the angle: by inputting a straight line, which will be described later, and by selecting from an operation menu. The resolution of the angle information that can be specified by inputting a straight line (45 degrees in Figure 17) is just one example, and a smaller resolution such as 5 to 30 degrees may also be used. However, it is believed that users will be able to read characters that are rotated at around 45 degrees.

[0153] The pen ID control data 901 to 904 can have PenId, ColorId, Angle, FontName (font designation), handwriting recognition candidate off state (RecommendMenuOff), and AccountId. Items other than PenId are optional. AccountId is associated with the pen used by the user to sign in. PenId is identification information stored inside the pen 2500. ColorId indicates the color selected by the user using this pen 2500 (can be changed by the user as desired). ColorId is set with the color ID of the color definition data described below. The color definition data defines the details of each color (thickness, etc.).

[0154] Angle is stroke angle information set for this pen 2500 (can be changed as desired by the user). In the example of Figure 17(a), the angle information for each pen is 0 degrees, 90 degrees, 180 degrees, and 270 degrees counterclockwise. AccountId is user identification information. By associating AccountId with the pen ID, it is possible to identify the AccountId associated with the pen ID of the pen 2500 used by the user, and to execute operation commands using user-defined data.

[0155] FontName is added when the user presses the pen color selection button in Fig. 21. When FontName is added, when the character string object font is obtained in step S81 in Fig. 51, FontName is obtained instead of FontStyle in the font data 405 of the defined control data in Fig. 7. The character string object is displayed in the font of this FontName.

[0156] Pen ID control data 901 is pen ID control data with a PenId of 1, with a color of black, angle information of 0 degrees, AccountId=1, no FontName, and no RecommendMenuOff. The user with AccountId=1 is the user in user-defined data 717 in FIG. 14. This indicates that this user signed in by handwriting their username and other information with the pen with PenId=1, and selected black. Pen ID control data with no AccountId indicates a signed-out state (not associated with a user). Similarly, pen ID control data 902 is pen ID control data with a PenId of 2, with a color of red, angle of 90 degrees, no AccountId, FontName of pop font, and no RecommendMenuOff.

[0157] The pen ID control data 903 has a handwriting recognition candidate off state (RecommendMenuOff) set. The handwriting recognition candidate off state (RecommendMenuOff) is control data for controlling the operation guide 500 (described later) so that it does not display the character string candidates 539 (displaying only operation commands) or so that it does not even display the operation command candidates 510. In the latter case, the operation guide 500 itself is not displayed. The handwriting recognition candidate off state (RecommendMenuOff) is set by pressing a handwriting recognition candidate on / off button (see FIG. 37 ) (described later). When a user inputs handwritten data that does not require character recognition, such as a graphic, some users may find the operation guide 500 distracting. In this embodiment, the handwriting recognition candidate off state can be controlled, so the operation guide 500 can be hidden and the processing load can be reduced when the display device 2 does not recognize characters. The pen ID control data 903 has a PenID of 3, a color of blue, an angle of 180 degrees, no AccountId, and no FontName.

[0158] The pen ID control data 904 is pen ID control data with a PenId of 4, a color of green, an angle of 270 degrees, no AccountId, no FontName, and no RecommendMenuOff.

[0159] If RecommendMenuOff="True" is added to the pen ID control data, and control is adopted to not display operation commands, the operation commands will not be executed either. Therefore, to cancel RecommendMenuOff="True", a button operation such as that shown in Figure 37 is required.

[0160] Furthermore, the pen ID control data of the pen 2500 when the user presses the pen color selection button in FIG. 21 or the button in FIG. 37 is updated with the ColorId, Angle, FontName, and RecommendMenuOff defined for that button.

[0161] The pen ID control data is used in step S5 of Figure 45 (obtaining pen ID control data), step S20 of Figure 47 (saving angle information of pen ID control data), step S21 of Figure 47 (obtaining angle information of pen ID control data), and step S60 of Figure 49 (obtaining pen ID control data).

[0162] <Color definition data> Figure 18 shows an example of color definition data. One line of color definition data defines one color. The color definition data defines the black and white highlighting for ColorId on a black and white device (how to display the black and white pattern, width, and edge) and the color conversion method (how to display color information (Color) and width on a color device). Color information is represented by # and eight hexadecimal digits, with every two digits representing #R (red), G (green), B (blue), and A (transparency), and px represents the pixel width. Note that color definition data only applies to data originating from handwriting.

[0163] Color definition data 1001 is an example of a line type or color definition for a ColorId of "Black." On a monochrome-enhanced device that cannot display color, the color is represented by the line type. Pattern represents the content of the stroke or text border, Edged represents the presence or absence of a border, Color represents RGBA color information, and Width represents the line width. Color definition data 1001 indicates that handwritten input saved data is displayed as solid black 5 pixels wide + borderless on a monochrome-enhanced device, and as black with 0% transparency + 5 pixels wide on a color-enabled device. Similarly, color definition data 1002 is an example of a line type or color definition for a ColorId of "Red." A monochrome-enhanced device displays as solid black 3 pixels wide + a black border 1 pixel wide from the outside + a white border 2 pixels wide, and a color-enabled device displays as red with 0% transparency from the color chart + 5 pixels wide. Color definition data 1003 is an example of a line type or color definition for a ColorId of "Blue." A black and white emphasis-compatible device displays solid white 4 pixels wide + a black border 1 pixel wide, while a color-compatible device displays it with 0% blue transparency on the color chart + 5 pixel width. Color definition data 1004 is an example of a line type or color definition for a "Green" ColorId. A black and white emphasis-compatible device displays it with a black dot pattern 4 pixels wide + a black border 1 pixel wide, while a color-compatible device displays it with 0% green transparency on the color chart + 5 pixel width. Color definition data 1005 is an example of a line type or color definition for a "Magenta" ColorId. A black and white emphasis-compatible device displays it with a black upper right diagonal line pattern with 5 pixel transparent white + borderless, while a color-compatible device displays it with 60% magenta transparency on the color chart. Color definition data 1006 is an example of a line type or color definition for a "Cyan" ColorId. A black and white enhancement device displays a black horizontal line pattern with a width of 5 pixels and transparent white, while a color device displays a cyan color chart with 60% transparency.

[0164] Color definition data 1007 defines a line type or color with ColorId "ThinBlack." A black and white emphasis-compatible device displays it as solid black 3 pixels wide + no border, while a color-compatible device displays it as black with 0% transparency + 3 pixels wide. Color definition data 1008 defines a line type or color with ColorId "ThinRed." A black and white emphasis-compatible device displays it as solid black 1 pixel wide + a black border 1 pixel wide from the outside + a white border 1 pixel wide, while a color-compatible device displays it as red with 0% transparency on the color chart + 3 pixels wide. Color definition data 1009 defines a line type or color with ColorId "ThinBlue." A black and white emphasis-compatible device displays it as solid white 2 pixels wide + a black border 1 pixel wide, while a color-compatible device displays it as blue with 0% transparency on the color chart + 3 pixels wide. Color definition data 1010 defines a line type or color with ColorId "ThinGreen." A black and white emphasis compatible device displays a black dot pattern 2 pixels wide + 1 pixel black border, and a color compatible device displays it with 0% transparency of green on the color chart + 3 pixels wide.

[0165] In this way, the color definition data includes black and white emphasis and color conversion data. The color definition data is held in the handwritten input storage unit 25 and is used to convert the handwritten input storage data.

[0166] By defining color definition data as system-defined data and user-defined data and adding a form to change the color definition data to the user-defined data, users can change the color definition to their personal preferences after signing in.

[0167] Figure 19 shows data displayed in black and white highlighting and in color display mode using the color definition data of Figure 18. Figure 19(a) shows the black and white highlighting displayed by a black and white highlighting compatible device, while Figure 19(b) shows the color conversion and display by a color compatible device. For ease of drawing, Figure 19(b) is also displayed in black and white (hatched).

[0168] Code C1 is handwritten text in "black," "red," "blue," "green," "magenta," and "cyan," code C2 is handwritten strokes in "black," "red," "green," "magenta," and "cyan," code C3 is a spring-like single-stroke (handwritten), code C4 is a donut-shaped color chart (image), and code C5 is RGBCMY inside the color chart (text not handwritten). Beneath code C3 is handwritten black text that says "Transparent."

[0169] The color chart of code C4 and the text RGBCMY of code C5 are data read from a PDF file or the like and are not handwritten. Therefore, in terms of handwritten input data stored by the handwritten input storage unit 25 shown in FIG. 16, the stroke (handwritten) of code C2 is stored as handwritten input data 801 and 802 in FIG. 16. The handwritten text of code C1 is stored as handwritten input data 804 and 805 in FIG. 16. The non-handwritten text of code C5 is stored as handwritten input data 806 in FIG. 16. The image of code C4 is stored as handwritten input data such as handwritten input data 807 in FIG. 16. The color information of each handwritten input data is defined by ColorId.

[0170] As can be seen by comparing Figures 19(a) and (b), the red of the text "R" inside the color chart of code C10 in Figure 19(b) is not handwritten, so it is displayed in grayscale in Figure 19(a). In contrast, the text "red" of code C6 is handwritten, so it is displayed in black and white highlighting in Figure 19(a).

[0171] Also, magenta and cyan are transparent colors, and in Figure 19(b) "Transparent" is visible through the color, and in Figure 19(a) magenta and cyan also appear transparent because a transparent white pattern is used. In this way, a device that supports black and white emphasis can display colors that you want to stand out in black and white, and on a color device, they can be converted to color and displayed.

[0172] The color definition data is held by the handwriting input display control unit 23 and is used in step S6 of FIG. 45 (coordinate complement display data), step S82 of FIG. 51 (character string object display data), and step S114 of FIG. 53 (object display data).

[0173] <Pen color selection button definition data> Next, a method for a user to select a pen color will be described with reference to Figures 20 and 21. Figure 20 is an example of pen color selection button definition data. The pen color selection button definition data is data that associates a pen button displayed on the display 220 with a ColorId. The pen color selection button definition data has a pen button ID (PenButtonId), an icon file (Icon), a color ID (ColorId), and a font name (FontName).

[0174] FIG. 21 shows an example of pen color selection buttons 81 to 86 displayed on the display. The pen color selection buttons 81 to 86 are displayed using the icon file described above. The pen color selection buttons 81 to 86 are displayed in order from left to right in ascending order of pen button ID. When the user presses a pen color selection button 81 to 86 with the pen 2500, ColorId and FontName are added to the pen ID control data. Any handwritten object or character string object entered with that pen 2500 thereafter is displayed using ColorId and FontName.

[0175] Pen color selection button definition data 1101 is definition data for the pen color selection button 81 displayed at the far left. When the user presses the pen color selection button 81 with the pen 2500, the stroke data handwritten with that pen 2500 will have a ColorId of "Black" and the font of the text after handwriting recognition will be Mincho. Pen color selection button definition data 1102 is for the pen color selection button 82 displayed second from the left, the ColorId of the stroke data will be "Red" and the font of the text after handwriting recognition will be Pop. Similarly, pen color selection button definition data 1103 to 1106 define the button display position, ColorId, and text font of the pen color selection buttons 83 to 86.

[0176] Figure 21(a) shows an example of a text font display that is highlighted in black and white based on the pen color selection button definition data. The text font display example is for illustrative purposes only, but may be displayed when the user hovers the pen 2500. Figure 21(b) shows an example of a text font display that is displayed in color on a color-capable device based on the pen color selection button definition data (it is actually in color, but is displayed in black and white for drawing purposes).

[0177] As shown in FIG. 21(a), by changing the font in addition to the black and white emphasis, it is possible to achieve a more effective black and white emphasis display.

[0178] If the pen color selection button definition data is also defined in system-defined data and user-defined data, and a form for changing the pen color selection button definition data is added to the user-defined data, the pen color selection button can be changed to suit individual preferences after handwritten signature authentication. Therefore, the correspondence between ColorId and font is just one example.

[0179] The pen color selection button is held by the handwriting input display control unit 23 and is used in step S6 of FIG. 45 (coordinate complement display data), step S82 of FIG. 51 (character string object display data), and step S114 of FIG. 53 (object display data).

[0180] <Example of definition data for handwriting input area> Fig. 22 is a diagram showing a display example of the handwriting input area. The user handwrites "lock" or "confidential information" and selects it from the operation guide 500. This causes the display device 2 to execute the operation command "Recognition Control" of the operation command definition data 741 in Fig. 11(a), and the input window 110 in Fig. 22(a) is displayed.

[0181] It is assumed that confidential information will be entered into the handwriting input area 2706, and here we will explain the case where the user handwrites a user code consisting of numbers only. A user code is a type of identification information that identifies a user. While the user ID used for an account is an email address or the like, a user code simply identifies a user using numbers only.

[0182] Authentication using handwritten signature data requires an administrator to link the handwritten signature data to an account, which increases costs. In contrast, since the user code is already linked to the account, user authentication can be performed without increasing costs. In other words, in this embodiment, authentication using the user code is possible instead of authentication using handwritten signature data.

[0183] 22(a) shows the handwriting input area 2706 in which no handwritten data is displayed (when stroke display is disabled), and FIG. 22(b) shows the handwriting input area 2706 in which handwritten data is displayed (when stroke display is enabled).

[0184] Immediately after the user displays the input window 110, the display device 2 is in a stroke display disabled state. The input window 110 has a guide message 2702, a close button 2703, a mask display 2704, a display / hide switching button 2705, a handwriting input area 2706, and a confirm button 2707. The guide message 2702 is a message that says, "Please enter your user code." The close button 2703 is a button that the user clicks to stop the operation on the input window 110 and close the input window 110 . The mask display 2704 is the recognition result of the handwritten data, but it is displayed as a mask. The mask display is not limited to "●", but can also be "※", "*", hatching patterns, etc. The number of "●" matches the number of characters, so the user can know the number of characters they have entered. The number of "●" may be different from the number of characters, which makes it difficult to guess the character string. The display / hide switch button 2705 is a button for switching between the mask display 2704 and the handwriting recognition result 2709, and also for switching between displaying and hiding strokes in the handwriting input area 2706. The handwriting input area 2706 is an area where the user inputs the numbers of the user code by handwriting. The confirm button 2707 is a button that the user presses to confirm the recognized handwritten user code.

[0185] When the user presses the display / hide switch button 2705, the state changes to when stroke display is enabled, as shown in FIG. 22(b). User code 2710 handwritten by the user is displayed in the handwriting input area 2706. The mask display 2704 is displayed as the recognition result 2709 of the handwritten data. The recognition result 2709 can be a character string such as Japanese, numbers, English, or symbols, but is limited to numbers in this embodiment.

[0186] When stroke display is enabled, the display device 2 displays the strokes that the user has handwritten up to that point and the handwriting recognition result 2709. After that, the display device 2 displays the strokes that the user has added and the character strings that are the recognition result of the handwriting recognition, including the added handwritten strokes.

[0187] When the user presses the display / hide switch button 2705 in the state of FIG. 22(b), the display device 2 hides the user code 2710 in the handwriting input area 2706 and displays the mask display 2704 again.

[0188] There may be multiple handwriting input areas 2706. For example, a handwriting input area 2706 may be provided for each character.

[0189] Fig. 23 shows an example of definition data for a handwriting input area. Fig. 24 is a diagram for explaining the configuration of a handwriting input area. In Fig. 23 and Fig. 24, three areas are defined. (i) The area outside the input window 110 (referred to as area A in the sequence diagram described later, but not including areas B and C below). Note that area A is not limited to the periphery of the input window 110 and is a general handwriting area.

[0190] (x11, y11) to (x12, y12). This is the area defined by the definition data 2804. (ii) An area outside the handwriting input area of ​​the input window 110 (referred to as area B in the sequence diagram described later. However, this does not include the next area C.) (x21, y21) to (x22, y22). This is the area defined by the definition data 2805. (iii) Handwriting input area (referred to as area C in the sequence diagram below) (x31, y31) to (x32, y32). This is the area defined by the definition data 2806 or 2807.

[0191] (i) Area A outside the input window 110 is defined by definition data 2804 in Fig. 23. (ii) Area B of the input window 110 other than the handwriting input area 2706 is defined by definition data 2805 in Fig. 23. (iii) Area C which is the handwriting input area 2706 is defined by definition data 2806 and 2807 in Fig. 23. Definition data 2806 is definition data when handwriting data display is disabled, and definition data 2807 is definition data when handwriting data display is enabled.

[0192] The attributes of the definition data will be described below. RegionId is an identifier for the region. RegionId is used when the handwriting recognition control unit 26 refers to the definition data. It is not used in this example of definition data. Depth is the overlapping order in the depth direction. The larger the Depth value, the closer the area is to the front. When a pen-down event occurs, the handwriting recognition control unit 26 executes control according to the definition data with the largest Depth among the Positions including the pen coordinates. Position is the coordinate range of the area. Position has the start and end coordinates of the rectangular area. Recognition is the handwriting recognition mode. Details are explained in Figure 25. RecognitionDic is the name of the handwriting recognition dictionary file used in the area. The dictionary files in the definition data 2804 are recognition.doc, conversion.dic, and prediction.dic. These correspond to a general-purpose handwriting recognition dictionary (handwriting recognition dictionary section 27 in FIG. 6), a character string conversion dictionary (character string conversion dictionary section 29 in FIG. 6), and a predictive conversion dictionary (predictive conversion dictionary section 31 in FIG. 6), respectively. In other words, the same dictionary files are used as for normal handwriting recognition. The dictionary file "number.dic" in the definition data 2806 and 2807 is a handwriting recognition dictionary file dedicated to numbers. "number.dic" is a dictionary file that returns a recognition result of one of the numbers 0 to 9 regardless of what the user writes by hand. In this embodiment, the user cannot see the handwritten data, so the handwriting of the handwritten data is more likely to be distorted than when the user writes while looking at it. For this reason, higher recognition accuracy is required than when the user writes while looking at it. Therefore, in this embodiment, the dictionary used in the handwriting input area is limited to number input. Number input alone provides the highest recognition rate, eliminating the need to display selectable candidates 530 (described later). The reason why the definition data 2806 and 2807 do not specify a character string conversion dictionary or a predictive conversion dictionary is that a handwriting recognition dictionary dedicated to numbers is sufficient for practical use. However, if practical recognition accuracy can be obtained using a handwriting recognition dictionary, a character string conversion dictionary, and a predictive conversion dictionary, these dictionaries may be used in the definition data 2806 and 2807.

[0193] FIG. 25 shows whether each of the following controls is enabled or disabled for each possible value of Recognition: handwriting recognition, selectable candidate display, stroke display, dialog control, and UI operation.

[0194] When Recognition is set to Enable, the handwriting recognition control unit 26 performs handwriting recognition and displays the operation guide 500 (selectable candidates 530) described later. This indicates that handwritten data is displayed and that the display / hide of the recognition results of the handwritten data is not controlled (dialog control). Recognition set to Enable indicates that UI operations are disabled. UI operations indicate whether the display device 2 accepts operations other than stroke input, such as button operations on the input window 110. Therefore, Enable is the normal input mode (recognizing strokes and displaying selectable candidates 530, and accepting input of the selectable candidate once a candidate is selected).

[0195] "Disable" disables all operations except the UI, and the user cannot input handwritten data. The user can only operate the menu on the operation panel, etc.

[0196] ForceForDialog recognizes handwritten strokes but does not display selectable candidates. It also displays handwritten strokes and enables dialog control. Dialog control means that the display unit 22 displays the recognition results of handwritten data in a position and font specified by the input window 110. That is, the display / hide switch button 2705 switches between displaying the mask display 2704 and the recognition results 2709 of the handwritten data. The area where the recognition results 2709 of the handwritten data are displayed can be outside the ForceForDialog area. Furthermore, because stroke display is enabled, the user can see the handwritten strokes. Therefore, ForceForDialog corresponds to the handwriting input area 2706 in FIG. 22(b). The ForceForDialog area displays handwritten data but does not display selectable candidates for the character string obtained by recognizing the handwritten data, and instead accepts the input of the character string with the highest recognition probability.

[0197] ForceForDialogWithoutStroke disables stroke display. Dialog control is enabled, so a mask display 2704 is displayed by clicking the display / hide switch button 2705. Other aspects are the same as ForceForDialog. Therefore, ForceForDialogWithoutStroke corresponds to the handwriting input area 2706 in FIG. 22(a), where handwritten data is not displayed. The ForceForDialogWithoutStroke area accepts character string input without displaying either the handwritten data or the converted character string.

[0198] The definition data 2806 and the definition data 2807 have the same RegionId because they indicate the same region (2706 and 2710 in FIG. 22). When the user opens the input window 110 in FIG. 22(a), the definition data 2806 is added, and when the display / hide switching button 2705 is pressed, the handwriting recognition control unit 26 changes the definition data 2806 to the definition data 2807. As a result, the handwritten data is displayed in the handwriting input region 2706.

[0199] Control using Recognition shown in FIG. 25 enables, for example, the following control. When handwritten data is input with the pen 2500 into an area that is not a predetermined area, the display device 2 displays selectable candidates for character strings converted from the handwritten data. The display device 2 then displays a character string selected from the selectable candidates in an area that is not a predetermined area, and accepts input of the character string. On the other hand, when handwritten data is input with the pen 2500 into a predetermined area, the display device 2 does not display the selectable candidates. The display device 2 then accepts input of the character string converted from the handwritten data.

[0200] Force is an area where the display of selectable candidates in Enable is disabled. In other words, Force is used in an area where the display device 2 automatically selects the candidate with the highest probability of recognition. For example, Force is used when the range of characters or character strings to be entered is fixed (gender, age, area name, etc.). Force is not used in the definition data 2804 to 2807, but is used in the definition data of the mask display area in Fig. 27.

[0201] In this way, the display device 2 stores information that defines whether to enable or disable recognition of handwritten data, whether to display or hide selectable candidates for character strings obtained by recognizing handwritten data, and whether to display or hide handwritten data, in association with each area. The display device 2 switches between recognizing handwritten data, displaying or hiding selectable candidates, and displaying or hiding handwritten data, based on the information and the area into which handwritten data is input.

[0202] In the definition data of FIG. 25, the display and non-display of strokes and the like are associated with areas, but the display device 2 may control the display and non-display of strokes and the like over time. For example, immediately after the display device 2 is started up, the confidential information input mode is turned on, or the user turns on the confidential information input mode from a menu or the like. The display device 2 operates in ForceForDialogWithoutStroke mode for a certain period of time (regardless of the area) from the time the confidential information input mode is turned on. In this case, it is not necessary to display the input window 110, but it is preferable to display the display / hide switch button 2705 so that the user can check the character string they have handwritten. After a certain period of time has passed, the display returns to the same state as area A.

[0203] Alternatively, the display of strokes and the like may be switched on and off by a user operation, rather than by time. From the time the user turns on the confidential information input mode from a menu or the like, until the time the user turns it off, the display device 2 operates in the ForceForDialogWithoutStroke mode.

[0204] <Mask display area and mask display area definition data> Fig. 26 is an example of a mask display area. Fig. 27 shows an example of definition data for a mask display area. Figs. 26 and 27 show mask display areas that are set when the display device 2 is used for reception work, for example. These are merely examples, and the size, number, or position of the areas can be set by the user as desired.

[0205] The row headings (1, 2, 3, 4...), column headings ("Name", "Phonetic Reading", "Gender", "Age"), and lines are fixed and are displayed on the background layer when the display device 2 is turned on. The user writes by hand on the layer in front of the background layer.

[0206] The user listens to the visitors and inputs their name, phonetic reading, gender, and age into each column by handwriting. Since the name and phonetic reading are personal information (confidential information), it is preferable that the display device 2 switch to a masked display.

[0207] In Figure 26, the screen is divided into a total of 11 areas, as shown below. Area 2901 is an area that does not accept handwritten input (one location). Area 2902 is an area where a name is input by handwriting (four locations). Based on definition data 2907 to 2910 in FIG. 27, selectable candidates 530 (described later) are displayed in area 2902, and a character string selected by the user from selectable candidates 530 is displayed. The character string selected by the user is displayed in a masked state after a certain period of time has passed. The handwritten data may be displayed as is (without character recognition). The handwriting input display control unit 23 may not mask some characters (for example, the beginning or end) and display the remaining characters in a masked state. Furthermore, the masked display may be "*" or "*" instead of "●", or may be a hatching pattern, etc. The number of "●" may or may not match the number of characters. The masked display may be such that the contents of area 2902 are concealed. Area 2903 is an area where furigana is handwritten (four locations). It is assumed that the user will only input hiragana in area 2903. Based on definition data 2911 to 2914 in FIG. 27, selectable candidates 530 are not displayed in area 2903, and the hiragana with the highest probability is forcibly displayed. The furigana in area 2903 may be automatically displayed as the recognition result of the hiragana input by the user in area 2902. Area 2904 is an area where gender is handwritten (all in one place because no mask is displayed). It is assumed that the user will only input gender in area 2904. Based on the definition data 2915 in FIG. 27, the selectable candidates 530 are not displayed in area 2904, and the one with the higher probability, "male" or "female," is forcibly displayed. Area 2905 is an area where age is handwritten (same as gender, in one area). It is assumed that the user will only enter age (number) in area 2905. Based on the definition data 2916 in FIG. 27, selectable candidates 530 are not displayed in area 2905, and the number with the highest probability is forcibly displayed.

[0208] Fig. 27 shows definition data for each area. Differences from the definition data in Fig. 23 will be explained below. Region, Depth, Recognition, Recognition, RecognitionDic, ConversionDic, and PredictionDic have the same meanings as in Fig. 23. MaskedDisplay indicates the area in which handwritten input data is displayed by the display device 2 in a masked manner. In the definition data in Fig. 27, MaskedDisplay="True" is defined in the areas for handwritten input of names in definition data 2907 to 2910 and in the areas for handwritten input of hiragana in definition data 2911 to 2914. When MaskedDisplay="True" is defined, the area becomes a masked display area, and character strings are displayed in a masked manner.

[0209] Each of the areas 2902 and 2903 in FIG. 26 is divided into four areas, and each area is displayed with a mask according to the operation status.

[0210] The definition data 2906 indicates that the area is one in which processing related to handwritten input is not performed because Recognition="Disable" is set.

[0211] Definition data 2907 to 2910 indicate that this is an area where the most versatile handwriting input processing is performed because Recognition="Enable". Definition data 2907 to 2910 define the name area. A dictionary dedicated to names (RecognitionDic="recognition_name.dic", ConversionDic="conversion_name.dic", PredictionDic="prediction_name.dic") is specified in the dictionary file, allowing users to efficiently input names by hand.

[0212] The definition data 2911 to 2914 indicate that the area is one in which the candidate with the highest probability of recognition is automatically displayed without the display device 2 displaying the selectable candidates 530, which are candidates for the recognition result, because Recognition="Force". In other words, the effort required for the user to select a character string from the selectable candidates 530 can be reduced. The definition data 2911 to 2914 define the furigana area. The dictionary file for the definition data 2911 to 2914 specifies a dictionary dedicated to hiragana (Recognition="hiragana.dic"), so the user can efficiently input furigana by hand.

[0213] The definition data 2915 indicates that this is an area in which the candidate with the highest probability of recognition is automatically selected without the display device 2 displaying the selectable candidates 530, which are candidates for the recognition result, because Recognition="Force". The definition data 2915 defines the gender area. The dictionary file of the definition data 2915 specifies a dictionary dedicated to gender (RecognitionDic="gender.dic"), which indicates that this is an area in which gender can be efficiently input by hand. In other words, this allows for efficient input in situations where the user must input information in a limited time, such as at a disaster site.

[0214] The definition data 2916, because Recognition="Force", indicates that this is an area in which the display device 2 automatically selects the candidate with the highest probability of recognition without displaying the selectable candidates 530, which are candidates for the recognition result. The definition data 2916 defines the age area. The dictionary file of the definition data 2916 specifies a dictionary dedicated to numbers (RecognitionDic="number.dic"), indicating that this is an area in which age can be efficiently input by hand. In other words, this allows for efficient input in situations where the user must input age in a limited time, such as at a disaster site.

[0215] In this way, the display device 2 of this embodiment allows the user to enable or disable handwriting recognition control (control for character recognition of handwritten strokes and character string conversion or predictive conversion), selectable candidate display control (control for allowing the user to select candidates resulting from handwriting recognition), and the type of dictionary related to handwriting recognition for each area, depending on the type of information handwritten in the area. Therefore, when the range of data to be handwritten is fixed, more efficient handwriting input and character string display are possible.

[0216] In FIG. 27, Recognition="Force" is set for hiragana, gender, and age. However, if the range of data to be handwritten, such as addresses and area names, is predetermined, the user can set Recognition for that area to "Force." For example, in the event of a disaster, some of the addresses entered into the display device 2 are often limited to nearby addresses. Also, at the scene of a fire, building names and company names are often limited to the building where the fire occurred and its tenant businesses. Furthermore, in rural areas, a situation may arise where the user enters the names of fire brigades engaged in firefighting activities by area. In this case, the character strings to be handwritten are often limited to area names, fire brigade names, etc. In this way, if dictionaries are prepared according to the range of data to be handwritten, efficient handwritten input and character string display are possible.

[0217] The area does not have to be rectangular, but may be a polygon having a triangle or more sides, or may be a circle.

[0218] <Example of selectable candidates> FIG. 28 shows an example of an operation guide and selectable candidates 530 displayed by the operation guide. When the user writes a handwritten object 504 by hand (when the selectable candidate display timer times out), the operation guide 500 is displayed. The operation guide 500 has an operation header 520, operation command candidates 510, handwriting-recognized character string candidates 506, conversion character string candidates 507, character string / predictive conversion candidates 508, and a handwritten object rectangular area display 503. The selectable candidates 530 are the operation command candidates 510, handwriting-recognized character string candidates 506, conversion character string candidates 507, and character string / predictive conversion candidates 508. In this example, "No language conversion string" may be displayed. The selectable candidates 530 excluding the operation command candidates 510 are referred to as character string candidates 539.

[0219] The operation header 520 has buttons 501, 509, 502, and 505. Button 501 accepts an operation to switch between predictive conversion and "kana conversion." In the example of FIG. 28, when the user presses button 501 displaying "predictive," the handwriting input unit 21 accepts the press and notifies the handwriting input display control unit 23, and the display unit 22 changes the display to a button displaying "kana." After the handwritten object 504 is recognized, character string candidates 539 are arranged in descending order of the probability of "kana conversion."

[0220] Button 502 operates the page of the candidate display. In the example of FIG. 28, there are three candidate display pages, and the first page is currently displayed. Button 505 accepts erasure of the operation guide 500. When the user presses button 505, the handwriting input unit 21 accepts it and notifies the handwriting input display control unit 23 of that fact, and the display unit 22 erases the display except for the handwritten object. Button 509 accepts erasure of all displays. When the user presses button 509, the handwriting input unit 21 accepts it and notifies the handwriting input display control unit 23 of that fact, and the display unit 22 erases all displays shown in FIG. 28, including the handwritten object, allowing the user to start handwriting again from the beginning.

[0221] A handwritten object 504 is the character "gi" handwritten by the user. A handwritten object rectangular area display 503 is displayed surrounding the handwritten object 504. The display procedure will be explained with reference to the sequence diagrams of Figs. 45 to 53. In the example of Fig. 28, the handwritten object rectangular area display 503 is displayed in a dotted line frame.

[0222] The character string candidates are arranged in descending order of probability in the handwriting recognition character string candidates 506, conversion character string candidates 507, and character string / predictive conversion candidates 508. The character "gi" in the handwriting recognition character string candidate 506 is a candidate for the recognition result. In this example, the character "gi" is correctly recognized.

[0223] Conversion string candidate 507 is a conversion string candidate converted from a language string candidate. In this example, "Technical trial" is an abbreviation for "Technical mass production trial." Character string / predictive conversion candidate 508 is a predicted character string candidate converted from a language string candidate or a conversion string candidate. In this example, "Approve the technical trial" and "Address to send the minutes" are displayed.

[0224] The operation command candidate 510 is a candidate operation command selected based on the operation command definition data 701 to 703 and 709 to 716 in Fig. 11(a). In the example of Fig. 28, the first character of the line "》" 511 is shown to be a candidate operation command. In Fig. 28, there is no selected object for the handwritten object 504 "ぎ" to select, and the character string candidate for "ぎ" "meiji" (minutes) partially matches the operation command definition data 701 and 702 in Fig. 11(a), so it is displayed as the operation command candidate 510.

[0225] When the user selects "Load minutes template," the operation command defined in operation command definition data 701 is executed, and when the user selects "Save in minutes folder," the operation command defined in operation command definition data 702 is executed. In this way, operation command candidates are displayed only when operation command definition data containing the converted character string is found, so they may not always be displayed.

[0226] As shown in FIG. 28, character string candidates and operation command candidates are displayed simultaneously (together), so the user can freely select either the character string candidate or the operation command that he or she intended to input.

[0227] <Example of specifying a selected object> In the display device 2 of this embodiment, the user can specify a selected object by hand-selecting a fixed object. The selected object becomes a target for editing or modification, or a page name.

[0228] Fig. 29 is an example diagram for explaining an example of designating a selected object. In Fig. 29, the handwritten object 11 is displayed with a solid black line, the handwritten object rectangular area 12 is displayed with a gray shading, the confirmed object 13 is displayed with a black line, and the selected object rectangular area 14 is displayed with a dotted line. Note that lowercase alphabets are added to the symbols when distinguishing between them. In addition, the crossing line determination condition 406 or the enclosing line determination condition 407 of the predefined control data shown in Fig. 7 is used as the determination condition for determining whether a confirmed object is a selected object (whether or not there is a predetermined relationship).

[0229] 29(a) shows an example in which the user has specified two horizontally written fixed objects 13a and 13b with a crossing line (hand-drawn object 11a). In this example, the length H1 of the short side and the length W1 of the long side of the hand-drawn object rectangular area 12a satisfy the crossing line determination condition 406. In addition, in this example, the overlap rate with the fixed objects 13a and 13b satisfies the crossing line determination condition 406. Therefore, both the fixed objects 13a and 13b, "minutes" and "giji," are specified as selected objects.

[0230] 29(b) shows an example in which a horizontally written fixed object 13c is specified by an encircling line (hand-drawn object 11b). In this example, only the fixed object 13c called "minutes" is specified as the selected object because the overlap rate between the fixed object 13c and the hand-drawn object rectangular area 12c satisfies the encircling line determination condition 407.

[0231] 29(c) is an example in which multiple vertically written fixed objects 13d, 13e are specified with a crossing line (hand-drawn object 11c). In this example, similar to FIG. 29(a), the length H1 of the short side and the length W1 of the long side of the hand-drawn object rectangular area 12d satisfy the crossing line determination condition 406. Also, in this example, the overlap rate with each of the two fixed objects 13d, 13e, "minutes" and "giji", satisfies the crossing line determination condition 406. Therefore, both the fixed objects 13d, 13e, "minutes" and "giji", are specified as selected objects.

[0232] 29(d) shows an example in which a vertically written fixed object 13f is specified with an encircling line (handwritten object 11d). In this example, just like in FIG. 29(b), only the fixed object 13f called "minutes" is specified as the selected object.

[0233] <Example of displaying operation command candidates> Figure 30 shows an example of displaying operation command candidates based on the operation command definition data when there is the handwritten object shown in Figure 13. Figure 30(a) shows candidates for editing operation commands, and Figure 30(b) shows candidates for modification operation commands. Figure 30(a) also shows an example in which a selected object is specified by the handwritten object 11a in Figure 29(a).

[0234] As shown in Figures 30(a) and (b), the operation command candidates displayed following the first character "》" 511 are a main menu 550. The main menu 550 displays the name of the operation command last executed or the name of the first operation command in the operation command definition data. The first character "》" 511a on the first line is a candidate for an edit-related operation command, and the first character "》" 511b on the second line is a candidate for a modification-related operation command. The first character "》" 511c on the third line is a candidate for the operation command "Set as page name."

[0235] The ">" 512 at the end of the line indicates that there is a submenu (an example of a submenu button). The ">" 512a on the first line displays a submenu of candidate edit-related operation commands (the last selected one), and the ">" 512b on the second line displays the remaining submenus of candidate modifier-related operation commands. When the user presses the ">" 512, a submenu 560 is displayed to the right of it. All operation commands defined in the operation command definition data are displayed in the submenu 560. In the display example of Figure 30(a), the submenu 560 corresponding to the ">" 512a on the first line is also displayed from the time the main menu is displayed. It may also be displayed by pressing the ">" 512a on the first line.

[0236] The operation command candidates for page name setting on the third line are not grouped, so ">" is not displayed at the end of the line.

[0237] When the user presses one of the operation command names with the pen, the handwriting input display control unit 23 executes the command in the operation command definition data associated with the operation command name on the selected object. That is, if "Erase" 521 is selected, "Delete" is executed, if "Move" 522 is selected, "Move" is executed, if "Rotate" 523 is selected, "Rotate" is executed, and if "Select" 524 is selected, "Select" is executed.

[0238] For example, if the user presses "Delete" 521 with the pen, "Minutes" and "Giji" can be deleted. Also, if the user presses "Move" 522, "Rotate" 523, or "Select" 524, a bounding box (the circumscribed rectangle of the selected object) is displayed. When "Move" 522 or "Rotate" 523 is selected, the user can move or rotate by dragging the pen, and when "Select" 524 is selected, the user can perform other operations on the bounding box.

[0239] The character candidates "one" 541, "one," 542, "~" 543, and "→" 544, which are not candidates for operation commands, are recognition results of the strikethrough line (handwritten object 11a). When the user intends to input a character rather than an operation command, the character candidate can be selected.

[0240] In FIG. 30(b), when the "〉" 512b in the second row is pressed, the sub-menu in FIG. 30(b) is displayed. Similar to FIG. 30(a), the main menu 550 and the sub-menu 560 are displayed in the display example of FIG. 30(b). Based on the operation command definition data in FIG. 13, when "thick" 531 is selected, "Thick" is executed; when "thin" 532 is selected, "Thin" is executed; when "large" 533 is selected, "Large" is executed; when "small" 534 is selected, "Small" is executed; and when the underline 535 is selected, "Underline" is executed by the handwritten input display control unit 23 for the selected object respectively.

[0241] Note that fixed values are separately defined for how thick to make when "thick" 531 is selected, how thin to make when "thin" 532 is selected, how large to make when "large" 533 is selected, how small to make when "small" 534 is selected, and the line type when the underline 535 is selected. Alternatively, it is even better if when the sub-menu in FIG. 31(b) is selected, a separate selection menu is opened and the user can adjust it.

[0242] When the user presses "thicken" 531 with the pen, the handwriting input display control unit 23 thickens the lines that make up the final objects 13a and 13b, "minutes" and "giji." When the user presses "thinner" 532 with the pen, the handwriting input display control unit 23 can thin the lines that make up "minutes" and "giji." When the user presses "larger" 533 with the pen, the handwriting input display control unit 23 can make the character string larger. When the user presses "smaller" 534 with the pen, the handwriting input display control unit 23 can make the character string smaller. When the user presses "underline" 535 with the pen, the handwriting input display control unit 23 can add an underline to the character string.

[0243] FIG. 31 shows an example of displaying operation command candidates based on operation command definition data when there is a handwritten object as shown in FIG. 13. The difference from FIGS. 30(a) and 30(b) is that FIGS. 31(a) and 31(b) show an example in which a selection object is specified by the handwritten object 11b (encircling line) of FIG. 29(b). As can be seen by comparing FIGS. 30 and 31, there is no difference in the operation command candidates displayed depending on whether the handwritten object is a line or an encircling line. The handwritten input display control unit 23 displays operation command candidates on the display unit 22 when a selection object is specified. However, the handwritten object may be recognized and the operation command candidates displayed may be changed depending on the handwritten object. In this case, the recognized handwritten object (such as a one or a circle) is associated with operation command definition data as shown in FIG. 13.

[0244] 31, character string candidates other than operation command candidates, "○" 551, "∞" 552, "0" 553, and "00" 554, are recognition results of the encircling line (handwritten object 11b). If the user intends to input a character string instead of an operation command, the user can select a character string candidate.

[0245] <Example of angle information input> Next, a method for inputting angle information will be described with reference to Fig. 32. Fig. 32 is an example of a diagram for explaining a method for inputting angle information. Fig. 32 explains a case where a user positioned at the 3 o'clock position of the display device 2 inputs angle information. Characters, etc. handwritten from the 3 o'clock position are correctly recognized when rotated 90 degrees clockwise, so it is preferable to input angle information of 90 degrees.

[0246] 32(a) shows a state in which the angle information of the pen ID control data is 0 degrees (initial value) and the operation guide 500 is displayed because a user standing at the 3 o'clock position of the display device 2 has handwritten "gi." Because the display device 2 recognizes the character "gi" handwritten from the 3 o'clock position with the angle information remaining at 0 degrees, a selectable candidate 530 that is different from the expected one is displayed.

[0247] When inputting angle information, the user handwrites a straight line from top to bottom as seen from the user's perspective within the operation guide 500. Figure 32(b) shows an example of this straight line 521. The angle information is the angle α formed by the line 521 in the counterclockwise direction and the 6 o'clock direction, which has angle information of 0 degrees. In other words, the angle information is the angle α formed by the line 522 drawn from the starting point S in the 6 o'clock direction and the line 521 input by the user in the counterclockwise direction. Simply put, the end point direction of the line 521 is the angle information. Therefore, the angle information input by the user in Figure 32(b) is 90 degrees.

[0248] For detecting a straight line, for example, the coordinates from the start point S to the end point E are converted into a straight line by the least squares method, and the obtained correlation coefficient is compared with a threshold value to determine whether or not the line is a straight line.

[0249] Immediately after the user starts writing the straight line 521 (immediately after the pen 2500 touches the starting point S of the straight line 521), the display device 2 erases the operation guide 500. Also, immediately after finishing writing the straight line 521 (immediately after the pen 2500 leaves the end point E of the straight line 521), the display device 2 searches for the value closest to the included angle α from 45 degrees, 90 degrees, 135 degrees, 180 degrees, 215 degrees, 270 degrees, 315 degrees, 360 degrees and determines it as the angle information. The included angle α itself may be used as the angle information. The determined angle information is set in the Angle of the pen ID control data. Since the pen event transmission unit 41 of the pen 2500 transmits the pen ID to the display device 2 when the pen tip is pressed for handwriting or the like, the display device 2 can associate the angle information with the pen ID control data.

[0250] Note that the user can input angle information by handwriting a straight line only within the operation guide 500. As a result, when the user handwrites a straight line outside the operation guide 500, it is recognized as "1" or "一", and when the user handwrites a straight line within the operation guide 500, the angle information can be input. That is, the handwriting recognition control unit 26 detects a straight line from a predetermined range and converts the stroke data handwritten outside the predetermined range into text data.

[0251] FIG. 32(c) shows the operation guide 500 immediately after the operation in FIG. 32(b). Since 90 degrees is set as the angle information (Angle) in the pen ID control data, the handwritten object (stroke data) is internally rotated 90 degrees clockwise for handwriting recognition, and the operation guide 500 is rotated 90 degrees counterclockwise and displayed. Note that the angle information may be manually input by the user from the menu.

[0252] <Example of registration of handwritten signature data> Next, an example of registering handwritten signature data will be described with reference to Fig. 33. Fig. 33 is a diagram illustrating a method for registering handwritten signature data. First, Fig. 33(a) is an example of selectable candidates 530 that are displayed when the user handwrites "signature." Two operation commands 513, 514, "Register handwritten signature" and "Sign out handwritten signature," based on operation command definition data 713, 715, which partially match the character string "signature," as well as character string candidates "sign," "signing session," and "signed" are displayed. The two operation commands 513, 514 are displayed because the String in the operation command definition data 713, 715 in Fig. 11 includes "signature."

[0253] When the user presses "Register handwritten signature" with the pen 2500, the handwritten signature registration form 561 shown in FIG. 33(b) is added to the handwritten input storage unit 25 and displayed on the screen. For example, the operation guide 500 shown in FIG. 33(a) is erased, and the handwritten signature registration form 561 is displayed in the same location as the operation guide 500. The handwritten signature registration form 561 has, from the top to bottom, a name input field 561a, signature input fields 561b to 561d, and a registration confirmation field 561e. The user enters the name text in the name input field 561a, the first, second, and third handwritten signatures in the signature input fields 561b to 561d, and a check mark or cancel mark in the registration confirmation field 561e. The name text is the user's display name and is converted into text data. The handwritten signature is entered three times because the features are registered on the assumption that handwritten signatures are different each time a user writes them and will not exactly match.

[0254] Note that handwritten signatures are generally characters related to a user, such as a user name, and may include not only a user name but also numbers such as an employee number, a nickname, or a portrait. Furthermore, handwritten signatures are not limited to characters related to a user, and may be any handwritten object. For example, they may be circles, triangles, squares, symbols, or combinations thereof. Since coordinates are not the only feature data for handwritten signatures, they can be correctly authenticated even if two users with the same surname (for example, Mr. Suzuki) both register the handwritten signature "Suzuki."

[0255] When the user writes his / her handwriting in the handwritten signature registration form 561 as instructed, the handwritten signature registration form 561 shown in FIG. 33(c) appears. When the user writes a check mark in the registration confirmation field 561e, the handwritten signature data is registered in the handwritten signature data storage unit 39, and the handwritten signature registration form 561 is deleted. A Signature ID is assigned as a result of the registration, and the similarly assigned Account ID and the name text in the name input field 561a are associated with the Signature ID and registered in the user-defined data. When the user signs in by handwriting a user name or the like, the Signature ID associated with the Account ID in the user-defined data is acquired and registered in the pen ID control data in association with the pen ID of the pen 2500 used for the handwritten sign-in. Thereafter, whenever the user uses the pen 2500, the pen ID is sent to the display device 2, and the Account ID associated with the pen ID in the pen ID control data can be identified, making it possible to execute operation commands using the user-defined data without the user even being aware of it.

[0256] If the user handwrites an "X" in the registration confirmation field 561e, the handwritten signature registration is canceled and the handwritten signature registration form 561 is deleted. If any error occurs in the registration, a message to that effect is displayed in a system-reserved area on the screen.

[0257] In this way, the handwriting input display control unit 23 can accept handwriting input without distinguishing between handwriting input into a form and handwriting input outside of a form.

[0258] <Example of handwritten sign-in> Next, a method for a user to sign in after registering handwritten signature data will be described with reference to Fig. 34. Fig. 34 shows an example of operation guide 500 that is displayed when the user handwrites "Suzuki," which is registered handwritten signature data. Since "Suzuki" is registered as handwritten signature data in operation command definition unit 33, "Suzuki" matches the handwritten signature data. Therefore, operation command 512, "Sign in by handwriting," is displayed.

[0259] Furthermore, since the handwritten signature data matches, the SignatureId representing "Suzuki" is identified, and the user-defined data having the AccountId associated with this SignatureId is identified.

[0260] When the user selects the operation command 512 "Sign in by handwriting," the AccountId of "Suzuki" is associated with the pen ID of the pen 2500 being used and added to the pen ID control data. Also, the user-defined data of "Suzuki" will be used when the operation command is used.

[0261] 34 is controlled as part of the normal handwritten input of characters, etc., so the handwritten signature registration form 561 is displayed on the same operation screen as the operation screen on which characters, etc. are handwritten. There is no difference between handwriting operations inside and outside the handwritten signature registration form 561, and the user can complete input to the handwritten signature registration form 561 simply by handwriting input in the areas separated by ruled lines on the handwritten signature registration form 561.

[0262] <Example of changing user-defined data> Next, a method for changing user-defined data will be described with reference to Figure 35. Figure 35 is a diagram for explaining a method for changing user-defined data. Figure 35(a) is an example of operation guide 500 that is displayed when the user handwrites "se." In the operation command definition data 716 shown in Figure 11, "settings" is defined in the String, and "settings" was included in the predicted character string for "se," so the operation command "change settings" was displayed.

[0263] When a user selects "Change settings" using the pen 2500 with which they have signed in by handwriting, the Account ID of the user associated with the pen ID of this pen 2500 is identified in the pen ID control data. This identifies the user-defined data of the signed-in user, and a user-defined data change form 562 shown in FIG. 35(b) is added to the handwriting input storage unit 25 and displayed on the screen. In the example of FIG. 35, the user-defined data change form 562 is created in accordance with the user-defined data 718 shown in FIG. 14. The user-defined data change form 562 has a name field 562a, a password field 562b, a folder user name field 562c, a folder password field 562d, a folder file name field 562e, and a register or cancel field 562f.

[0264] If the user has not signed in by handwriting in advance, the display device 2 will not be able to identify the user's Account ID, resulting in an error, and an error message will be displayed in a system reserved area on the screen.

[0265] In the user-defined data change form 562 of Figure 35 (b), the user handwrites a password in the password field 562b. The user handwrites a folder user name in the folder user name field 562c. The user handwrites a folder password in the folder password field 562d. The user handwrites a folder file name in the folder file name field 562e. The user handwrites a "check mark" or "x" in the register or cancel field 562f. This executes the change to the user-defined data, and the user-defined data change form 562 is deleted.

[0266] In this way, the user can display user-defined data change form 562 and change the user-defined data as desired by handwriting stroke data that calls user-defined data change form 562. Handwriting input display control unit 23 accepts handwriting input without distinguishing between handwriting input into a form and handwriting input outside of a form.

[0267] The AccountUsername of the user-defined data is automatically displayed in the name field 562a. The user-defined data change form 562 can be used not only for changes but also for registration.

[0268] 35 is controlled as part of the normal handwritten input of characters, etc., so user-defined data change form 562 is displayed on the same operation screen as the operation screen on which characters, etc. are handwritten. There is no difference between handwriting operations inside and outside user-defined data change form 562, and the user can complete input to user-defined data change form 562 simply by handwriting in the areas separated by ruled lines on user-defined data change form 562.

[0269] <Example of operation menu display> Next, an operation menu having information regarding data processing according to the display position will be explained using Figure 36. Figure 36 is a top view of the display device 2 placed flat. In Figure 36, users 250 and 251 are present at the top and bottom of the drawing, respectively, and two users can simultaneously operate operation menus 301 and 302 from the top and bottom. However, there is no problem if there is only one user. The display device 2 displays the operation menu according to the operation button definition data. The number of operation menus will be the same as the number of expected users.

[0270] Each operation menu 301, 302 has information about data processing corresponding to the display position of the operation menu. In this embodiment, the operation menu has a rotation angle corresponding to the display position of the operation menu. The operation menus 301, 302 have the same design (icon appearance), but each has information about data processing corresponding to the display position of the operation menu.

[0271] Each operation menu 301, 302 has a pen operation panel 2001, 2008 that allows the user to select a pen type and perform pen operations, a page operation panel 2002, 2009 that allows the user to perform page operations, a page navigation operation panel 2003, 2010 that allows the user to operate a list display of file names or pages, and a page navigation 2004, 2011 that allows the user to switch pages from a list display of file names or page names. For ease of use by the user, the operation menu 301 for the lower user is positioned near the bottom of the screen (along the bottom edge of the display 220), and the operation menu for the upper user is positioned near the top of the screen (along the top edge of the display 220).

[0272] The user operation menu 301 on the lower side has a pen operation panel 2001, a page operation panel 2002, a page navigation operation panel 2003, and a page navigation 2004. The page navigation 2004 extends from top to bottom in a page navigation window area 2005. The page navigation window area 2005 may be simply called a page navigation window.

[0273] Similarly, the operation menu 302 for the upper user has a pen operation panel 2008, a page operation panel 2009, a page navigation operation panel 2010, and a page navigation 2011. The page navigation 2011 extends downward when the page navigation window area 2012 is viewed from the upper user.

[0274] Handwritten objects and character string objects are displayed near the center of the display 220. Handwritten object 2006 was written by the user on the bottom with a black pen (solid black). Character string object 2007 is a character string that was written by the user on the bottom with a black pen and recognized. Handwritten object 2013 is handwritten data written by the user on the top with a blue pen (solid white with a black border).

[0275] The pen operation panels 2001 and 2008, page operation panels 2002 and 2009, page navigation operation panels 2003 and 2010, and page navigation panels 2004 and 2011 are divided into two groups: one for the lower user and one for the upper user. Of these, the icons (buttons), file names, and page names included in the operation menu 302 for the upper user are displayed rotated 180 degrees. This angle of rotation is set in advance in the operation button definition data (described later).

[0276] When the user presses either the button area of ​​the operation menus 301 and 302, or the file name or page name of the page navigation 2004 and 2011 with the pen 2500, the display device 2 stores the rotation angle set in the operation button definition data in the pen ID control data of the pen 2500. This rotation angle becomes information related to data processing.

[0277] <<Pen operation panel>> 37 is an enlarged view of the pen operation panels 2001 and 2008. The pen operation panels 2001 and 2008 have buttons for the user to select colors, etc. Buttons 2101 to 2108 represent thin or thick pens in opaque black, red, blue, and green, while buttons 2109 and 2110 represent thick pens in transparent magenta and cyan.

[0278] Buttons 2111 and 2112 represent undo (return to original state) / redo (redo). Button 2113 represents page deletion, and button 2114 represents page sweep (a button for clearing and redisplaying a degraded screen in high-speed drawing mode). Button 2115 accepts the on / off setting for displaying character string candidates 539 (a button for accepting the setting of the handwriting recognition candidate off state (RecommendMenuOff) in the pen ID control data).

[0279] As shown in the operation button definition data in Figures 40 to 43, each button is defined with a button identifier (ButtonId), a button display position (Position), an icon image (Icon) of the button, a rotation angle (Angle) of the button, and a command (Command) that the display device 2 executes when the button is pressed. When the user presses a button 2101 to 2115 with the pen 2500, the display device 2 saves the rotation angle defined for that button in the pen ID control data of that pen 2500, and executes the command defined for that button. It should be noted that buttons 2101 to 2115 correspond to ButtonId=1 to 15 in the operation button definition data in order from left to right.

[0280] The commands of operation button definition data 2401-2410 corresponding to buttons 2101-2110 specify the ColorId of the color definition data in the ChangePen command, which is the same as the operation command definition data 709-711 and 720-726 in Fig. 11. The display unit 22 displays a frame 260 on the button of the pen that was last selected to indicate that it was selected. In Fig. 37, the thin black pen of button 2101 is selected.

[0281] Operation button definition data 2411, 2412 corresponding to buttons 2111, 2112 define the Undo / Redo commands. Buttons 2111, 2112 (Undo / Redo) each accept the execution of the Undo / Redo commands. Undo is a command to return to the previous operation state, and Redo is a command to advance the undone operation state one step forward.

[0282] The operation button definition data 2413 corresponding to the button 2113 defines the command of EragePage. The button 2113 (erase page) accepts the execution of the command of EragePage. EragePage erases all data entered on the current page and redisplays it.

[0283] The operation button definition data 2414 corresponding to the button 2114 defines a SweepPage command. The button 2114 (sweep button) accepts the execution of the SweepPage command. On display devices such as electronic paper, some of the image remains in the high-speed drawing mode. This is cleared by redisplaying it.

[0284] The operation button definition data 2415 corresponding to the button 2115 defines the ToggleRecommendMenu command. The button 2115 (handwriting recognition candidate on / off) accepts execution of the ToggleRecommendMenu command. ToggleRecommendMenu switches the display of the character string candidates 539 obtained by handwriting recognition on and off. As a result, if the display is turned off, RecommendMenuOff="True" is added to the pen ID control data of the pressed pen 2500. When the display of the character string candidates 539 is on, all of the character string candidates 539 are displayed, but when it is off, only the operation commands are displayed. If there are no operation commands, the entire selectable candidate set is not displayed, allowing the user to concentrate on handwriting. When the display device 2 executes ToggleRecommendMenu, the current on / off state is returned, and the icon is switched according to that state.

[0285] It should be noted that buttons 2101 to 2110 are defined as separate buttons from the pen color selection buttons 81 to 86 in FIG. 21, and since the ColorId may be set in duplicate by the pen color selection buttons 81 to 86, in such cases the result of pressing buttons 2101 to 2110 takes priority and is set in the pen ID control data.

[0286] <<Page Operation Panel>> Figure 38 shows an example of the page operation panels 2002 and 2009. The page operation panels 2002 and 2009 are operation menus related to page operations. Button 2201 is a button that allows the user to rotate the currently displayed page by 90 degrees. Buttons 2202 and 2204 are buttons that allow the user to switch to the previous page or next page, respectively. Current page 2203 indicates the current page number (numerator) and the total number of pages (denominator). Button 2205 is a button that allows the user to open or close the page navigation window. File name 2206 indicates the name of the file currently being displayed. Buttons 2201, 2202, 2203, and 2205 are defined in operation button definition data 2416 to 2419 and 2440 to 2443.

[0287] The button 2201 (Rotate Page Display) accepts execution of the RotatePage command. RotatePage rotates only the lower user's handwritten object 2006 and character string object 2007 and the upper user's handwritten object 2013 shown in FIG. 36 by 90 degrees counterclockwise and displays them. The operation menu, including the page navigation, is not rotated. In other words, in the example of FIG. 36, the upper user's handwritten object 2013 is upside down and difficult to read from the lower user's perspective, so by executing the RotatePage command, the display device 2 temporarily rotates the page to make it easier to read.

[0288] Buttons 2202 and 2204 (page switching) accept the execution of the command PageBack or PageNext, which switches the current page to the previous page or the next page.

[0289] A button 2205 (opens or closes the page navigation window) accepts execution of the TogglePageNavi command, which opens or closes the page navigation window areas 2005 and 2012 in FIG.

[0290] Current page 2203 and file name 2206 are for display only and do not accept any operations. Current page 2203 displays the current page number (numerator) and the total number of pages (denominator). In this example, the second page of a four-page file is displayed. File name 2206 displays the name of the file currently being displayed. In this example, "Technical Review Meeting.pdf" is displayed.

[0291] <<Page navigation operation panel>> 39 shows an example of the page navigation operation panels 2003 and 2010. The page navigation operation panels 2003 and 2010 display the page names in page order for each file. In other words, when a page name is selected from a character string object, the user can easily identify what is written on which page later.

[0292] Buttons 2301 and 2305 are buttons that allow the user to switch to the previous page navigation window or the next page navigation window, respectively. Buttons 2302 and 2304 are buttons that allow the user to switch to the previous file or the next file, respectively (the first page of the file after switching is displayed on the display 220). Current page navigation 2303 displays the current number (numerator) and total number (denominator) of the page navigation window. Buttons 2301, 2302, 2304, and 2305 are defined in operation button definition data 2420 to 2423 and 2444 to 2447.

[0293] The page navigation 2306 displays character strings in file order and page order. It displays the file names of multiple loaded files and the names of multiple pages within the files in a tree format, and when a file name or page name is pressed, the display device 2 displays that page on the display 220. For example, if the page navigation has 200 lines and the page navigation window has 50 lines, the current number of the current page 2203 can take on values ​​from 1 to 4, with the total number being 4. When the user presses button 2301, the display device 2 decrements the current number, and when the user presses button 2305, the display device 2 increments the current number and redisplays the page navigation window. For ease of use by the user, the page navigation 2306 is arranged so that the lower user's version is aligned to the bottom of the display 220 and the upper user's version is aligned to the top of the display 220.

[0294] In the page navigation 2306 in Figure 39, two file names 2310 and 2308 are displayed, and in each file, page names 2307 and 2309 of each page are displayed in a tree structure. The page names are character string objects specified by the user or automatically extracted character strings. The page names are displayed indented by the user's operation.

[0295] Buttons 2301 and 2305 (page navigation window switching) accept the execution of the PageNaviWinBack / PageNaviWinNext commands. PageNaviWinBack / PageNaviWinNext switches the current page navigation window to the previous or next page navigation window. If the number of lines that can be displayed in a page navigation window is 50, pressing buttons 2301 and 2305 will switch between page navigation windows of 50 lines each.

[0296] Buttons 2302 and 2304 (file switching) accept the execution of the commands PageNaviFileBack and PageNaviFileNext, which switch the currently displayed page to the first page of the previous or next file.

[0297] The page navigation 2306 is defined by the page navigation data in Fig. 44. The tree open icon 2311 is a downward triangle and means that the tree is being displayed. The tree closed icon 2312 is a sideways triangle and means that the tree is being closed. When the tree is closed, the page names of the rows below the row where the tree is located will no longer be displayed.

[0298] When the Tree in the page navigation data is "Opened," the display device 2 displays a tree open icon 2311, which, when pressed, changes the Tree to "Closed" and closes the tree. Conversely, when Tree="Closed," the display device 2 displays a tree closed icon 2312, which, when pressed by the user, changes the Tree to "Opened" and opens the tree.

[0299] The page name 2313 indicates the indented page. The indent level is controlled by the Level of the page navigation data. The file name has an initial value of Level="0", and the page name has an initial value of Level="1". When the user presses the file name or page name to indent to the right, the indent level Level value increases. The page name 2313 indicates an indent of Level="2". To change the Level, the user presses and holds the page name and moves it left or right. For example, when the user moves it to the right, the display device 2 searches for a line above the current line at the same Level (the Level before the move) and adds Tree="Opened" to the line above. It is possible to hide indented lines, and the display or non-display of indented lines is controlled by the Tree of the line at the same Level as the line before it was indented. The display device 2 also increments the Level of the indented line and displays it again.

[0300] Page name 2314 indicates a blank page, and indicates the blank page following the last page of the file (page name 2313). A blank page means that nothing has been handwritten on it yet. Blank pages continue up to the maximum page number, but only the first blank page is displayed as a "blank page" in the page navigation. For example, if the maximum page number is 100 and there are four pages in the file, pages 5 to 100 are blank pages, but only the first page, page 5, is displayed as a "blank page" in the page navigation 2306.

[0301] Page name 2315 is the page currently being displayed on display 220. Here, page name 2315 is highlighted by being surrounded by a frame 262.

[0302] Page name 2316 is a default page name. When there is only handwriting and no handwriting-recognized character string, display device 2 displays the page name with "nnnn page" (nnnn is an integer) added, as shown here. This indicates that page names other than the default page name of page name 2316, the blank page of page name 2314, and file names 2310 and 2308 contain handwriting-recognized character strings.

[0303] In the page navigation data described below, if AutoName="True" is set, the display device 2 will automatically set the page name if there is a character string that has been recognized by handwriting. In the example screen layout of Figure 36, the character string "Hardware Configuration" is automatically set as the page name. There are various possible logics for determining the page name from a character string that has been recognized by handwriting. For example, there is a method of joining the character strings in the order in which they were input and extracting a fixed number of characters from the beginning, or a method of determining the priority of the character string positions and joining the character strings in order of highest priority and extracting a fixed number of characters from the beginning.

[0304] As shown in the display examples of operation commands in Figures 30 and 31, when a confirmed object that has already been recognized by handwriting is selected with an enclosing line or a crossing line, "Set as page name" is displayed in the operation commands. When this is selected, the SetPageName command is executed. SetPageName changes the current page name in the page navigation 2306 to the selected character string, and changes AutoName in the page navigation data to "False."

[0305] When a file name or page name on the page navigation operation panel 2003, 2010 is selected with the pen 2500, "Angle" is also set in the pen ID control data. This Angle is the angle by which the page navigation window area is rotated. The display device 2 saves the angle (Angle) by which the page navigation window area is rotated, which is obtained from the operation button definition data for display purposes. When a file name or page name is selected with the pen 2500, the display device 2 sets the stored rotation angle to "Angle" in the pen ID control data. Thereafter, data is processed using the Angle set in the pen ID control data.

[0306] <Operation button definition data> 40 to 43 show examples of operation button definition data 2401 to 2448. The operation button definition data 2401 to 2448 define the buttons on the pen operation panels 2001 and 2008, the page operation panels 2002 and 2009, and the page navigation operation panels 2003 and 2010. One line of the operation button definition data defines one button. Also, Figs. 40 and 41 define the operation menu 301 for the lower user, and Figs. 42 and 43 define the operation menu 302 for the upper user.

[0307] Each operation button definition data 2401-2448 defines a button identifier (ButtonId), a button display position (Position), a button icon image (Icon), an angle (Angle) that is information related to data processing, and a command (Command) that the display device 2 executes when the button is pressed. The angle (Angle) also represents the user's operation position and the rotation angle of the operation menu 302. The user presses buttons 2101-2115, 2201, 2202, 2203, 2205, 2301, 2302, 2304, and 2305 with the pen 2500. As a result, the display device 2 saves the angle (Angle) defined for that button in the pen ID control data of that pen, and executes the command defined for that button.

[0308] In this way, the operation menu is associated with data processing according to the display position of the operation menu. In Figures 40 to 43, the character string object is rotated according to the display position, but if you change the Command with the same button on the operation menu for the upper user 301 and the operation menu for the lower user 302, various data processing becomes possible.

[0309] Operation button definition data 2401-2415 define the pen operation panel 2001 for the lower user, and operation button definition data 2416-2419 define the page operation panel 2002 for the lower user. Operation button definition data 2420-2424 define the page navigation operation panel 2003 for the lower user, and operation button definition data 2425-2439 define the pen operation panel 2008 for the upper user. Operation button definition data 2440-2443 define the page operation panel 2009 for the upper user, and operation button definition data 2444-2448 define the page navigation operation panel 2010 for the upper user.

[0310] The operation button definition data 2415, 2439 must indicate the current on / off state of the handwriting recognition candidate, so an icon image for on (IconOn) and an icon image for off (IconOff) are defined. The operation button definition data 2419, 2443 must represent a button to open or close the page navigation window, so an icon image for on (IconOn) and an icon image for off (IconOff) are defined. The display device 2 displays the icon image for on / off based on the return value of the execution result of Command.

[0311] The operation button definition data 2424, 2448 is definition data for the page navigation window area, and defines only the position and angle of the rectangular area. When the user presses these rectangular areas with the pen 2500, the display device 2 saves the defined angle in the pen ID control data of the pen 2500 used for the press.

[0312] <Page navigation data> Figure 44 shows an example of page navigation data. The page navigation data defines the display content of the page navigation window areas 2005 and 2012. Each line of the page navigation data defines one line of the page navigation. The page navigation data defines the file name (FileName), page number (Page), indent level (Level), page name (PageName), auto name (AutoName), tree state (Tree), and current page (CurrentPage). Page="0" and Level="0" are the file name. As the Level increases, the page name on the page navigation is indented to the right.

[0313] The default values are PageName = "Page nnnn" (nnnn is the page number) or "Blank Page", AutoName = "True", and Tree = "Opened". When AutoName = "True", the display device 2 automatically sets the handwritten recognized string to PageName. When the user specifies a confirmed object recognized by handwriting and selects "Set as Page Name" from the operation commands, the page name is changed to that confirmed object and AutoName = "False". That is, the page name set by the user takes precedence. "Tree" is a setting where when Tree = "Opened", all page names of the lines below this line and with a level one less than this line are displayed in a tree view, and when Tree = "Closed", they are not displayed in a tree view (only the page name of this line is displayed). When a page name is displayed, the same process is repeated for the displayed page name. Therefore, when the display device 2 has a level greater than this line in the next line, it sets "Opened" for the Tree of this line. CurrentPage indicates the page currently being displayed on the display 220.

[0314] <Operation Procedure> Using the above configuration and FIGS. 45 to 53, the operation of the display device 2 will be described. FIGS. 45 to 53 are sequence diagrams illustrating an example of the process in which the display device 2 displays candidates for character strings and candidates for operation commands. The process in FIG. 45 starts when the display device 2 is activated (when the application is activated). In FIGS. 45 to 53, for the sake of space, the functions in FIG. 6 are indicated by reference numerals.

[0315] Also, as described in regions A to C of FIG. 24, for each pen coordinate on the screen, the validity / invalidity of each of handwriting recognition, display of selectable candidates, display of strokes, dialog control, or UI operation is defined. The control sequence is executed according to this definition. In this embodiment, attention should be paid to the processing related to the mask display timer control unit 40.

[0316] Before starting to input handwritten data, the user has already selected the buttons 2101 to 2110 on the pen operation panels 2001 and 2008 (the PenId has been specified). a. The pen button ID, Color ID and Angle are specified in the operation button definition data, b. Pen ID, Color ID, and Angle are registered in the pen ID control data.

[0317] When UI operation is disabled (areas A and C), step S1 is executed.

[0318] S1: First, the handwriting input display control unit 23 sends a handwriting object start signal to the handwriting input storage unit 25. The handwriting input storage unit 25 allocates a handwriting object area (a memory area for storing handwritten objects). The handwriting object area may be allocated after the user brings the pen into contact with the handwriting input unit 21.

[0319] S2: Next, the user touches the pen to the handwriting input unit 21. The handwriting input unit 21 detects the pen down and transmits it to the handwriting input display control unit 23.

[0320] When UI operation is disabled (areas A and C), steps S3 to S50 are executed.

[0321] S3: The handwritten input display control unit 23 transmits a stroke start signal to the handwritten input storage unit 25, and the handwritten input storage unit 25 reserves a stroke area.

[0322] S4: When the user moves the pen while keeping it in contact with the handwriting input unit 21, the handwriting input unit 21 transmits the pen coordinates to the handwriting input display control unit 23.

[0323] S5: The handwriting input display control unit 23 specifies the pen ID received from the pen 2500 at the same time as the coordinates are input, and acquires the current pen ID control data stored in the pen ID control data storage unit 36. Since the pen ID is sent when the coordinates are input, the strokes and the pen ID are associated with each other. The pen ID control data storage unit 36 ​​sends the pen ID control data to the handwriting input display control unit 23. Note that there is no AccountId because the user has not signed in.

[0324] When stroke display is enabled (ForceForDialog in area A or area C), step S6 is executed.

[0325] S6: The handwriting input display control unit 23 transmits pen coordinate complement display data (data for interpolating discrete pen coordinates) to the display unit 22. The display unit 22 interpolates the pen coordinates using the pen coordinate complement display data, specifies the line type and thickness from the color definition data based on ColorId, and displays the stroke.

[0326] S7: The handwriting input display control unit 23 transmits the pen coordinates, the reception time thereof, ColorId, and angle information to the handwriting input saving unit 25. The handwriting input saving unit 25 adds the pen coordinates to the stroke. While the user is moving the pen, the handwriting input unit 21 periodically repeats transmitting the pen coordinates to the handwriting input display control unit 23, and therefore, the processes of steps S4 to S7 are repeated until the pen is lifted.

[0327] S8: When the user removes the pen from the handwriting input unit 21, the handwriting input unit 21 transmits a pen-up signal to the handwriting input display control unit 23.

[0328] S9: The handwriting input display control unit 23 sends a stroke end signal to the handwriting input saving unit 25, and the handwriting input saving unit 25 confirms the pen coordinates of the stroke. Once the pen coordinates of the stroke have been confirmed, it becomes impossible to add pen coordinates to the stroke.

[0329] S10: Next, the handwriting input display control unit 23 transmits an overlap status acquisition of the rectangular area near the handwritten object and the stroke rectangular area based on the rectangular area near the handwritten object 403 to the handwriting input saving unit 25. The handwriting input saving unit 25 calculates the overlap status and transmits the overlap status to the handwriting input display control unit 23.

[0330] The subsequent steps S11 to S17 are executed if the handwritten object vicinity rectangular area and the stroke rectangular area do not overlap.

[0331] S11: If the rectangular area near the handwritten object and the rectangular area of ​​the stroke do not overlap, one handwritten object is determined, and the handwritten input / display control unit 23 transmits a command to clear the stored data to the handwriting recognition control unit 26.

[0332] S12 to S14: The handwriting recognition control unit 26 sends a retained data clear command to the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32. These clear the data relating to character string candidates and operation command candidates that have been retained by the handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32. At the time of clearing, the last handwritten stroke has not been added to the handwritten object.

[0333] S15: The handwriting input display control unit 23 transmits a handwriting object end notification to the handwriting input saving unit 25. The handwriting input saving unit 25 finalizes the handwritten object. Finalizing a handwritten object means that one handwritten object has been completed (no more strokes can be added).

[0334] S16: The handwriting input display control unit 23 transmits a handwriting object start signal to the handwriting input saving unit 25. In preparation for the start of handwriting of the next handwritten object (pen down), the handwriting input saving unit 25 reserves a new handwritten object area.

[0335] S17: Next, the handwriting input display control unit 23 transmits a stroke addition for the stroke completed in step S9 to the handwriting input saving unit 25. If steps S11 to S17 have been executed, the added stroke is the first stroke of the handwritten object, and the handwriting input saving unit 25 adds the stroke data to the currently started handwritten object. If steps S11 to S17 have not been executed, the added stroke is added to the currently handwritten object.

[0336] S18: Next, the handwriting input display control unit 23 transmits a stroke addition to the handwriting recognition control unit 26. The handwriting recognition control unit 26 adds the stroke data to a stroke data storage area (area where stroke data is temporarily stored) where character string candidates are stored.

[0337] S19: The handwriting recognition control unit 26 performs gesture handwriting recognition on the stroke data storage area. Gesture handwriting recognition refers to recognizing angle information from a straight line. Since gesture handwriting recognition is performed inside the operation guide 500, the handwriting recognition control unit 26 detects straight lines inside the operation guide 500. The position information of the operation guide 500 is sent to the handwriting recognition control unit 26 in step S67, which will be described later.

[0338] S20: If a straight line is detected in the operation guide 500, the angle α formed by the counterclockwise direction of the line 522 drawn from the starting point of the line in the 6 o'clock direction and the line 521 input by the user is determined in 45-degree increments. Then, the handwriting recognition control unit 26 associates the determined angle information with the pen ID contained in the stroke data of the line 521 and stores it in the pen ID control data storage unit 36. Note that step S20 is executed when a straight line is detected in the operation guide 500. Therefore, it is possible to set a different angle in preference to the angle determined by pressing the buttons 2101 to 2115 on the pen operation panels 2001 and 2008.

[0339] S20-2: The handwriting recognition control unit 26 clears the selectable candidate display rectangle.

[0340] S21: Next, the handwriting recognition control unit 26 acquires angle information of the current pen ID control data from the pen ID control data storage unit 36, specifying the pen ID received from the handwriting input unit 21.

[0341] S22: The handwriting recognition control unit 26 rotates the stroke data in the stroke data storage area clockwise using the acquired angle information. In this way, the stroke data can be rotated in accordance with the information on data processing according to the display position, thereby enabling character recognition.

[0342] S23: The handwriting recognition control unit 26 transmits the rotated stroke data to the handwritten signature authentication control unit 38. In this way, the stroke data is always transmitted to the handwritten signature authentication control unit 38 in a state where it is unclear whether it is a handwritten signature or not.

[0343] S24: The handwritten signature authentication control unit 38 receives the stroke data and receives the registered handwritten signature data from the handwritten signature data storage unit 39. The stroke data and the handwritten signature data are then compared (matched), and the handwritten signature authentication result is stored so that it can be obtained in subsequent step S61. If the authentication is successful, the AccountId is registered in the pen ID control data.

[0344] S25: Next, the handwriting recognition control unit 26 performs handwriting recognition on the stroke data, and if there is a "check mark" or "x" in the registration or cancellation column of the form, processes the form; otherwise, it performs normal handwriting recognition processing.

[0345] S26: A case where a "check mark" is entered in the registration or cancellation field of the handwritten signature data registration form will be described below. In this case, the handwritten input display control unit 23 generates a handwritten signature registration form for the handwritten input storage unit 25 in step S100 described later, and the handwritten signature data (stroke data) entered by the user is sent by the handwritten input recognition control unit 26 to the handwritten signature authentication control unit 38.

[0346] S27: The handwritten signature authentication control unit 38 registers the received handwritten signature data (stroke data) in the handwritten signature data storage unit 39. As a result, a SignatureId is assigned. The SignatureId is returned to the handwriting recognition control unit 26. If the SignatureId and the name entered in the name input field 561a of the handwritten signature registration form 561 are not present in the user-defined data, the handwriting recognition control unit 26 adds new user-defined data. The handwriting recognition control unit 26 also assigns a number to an AccountId and saves the SignatureId in the user-defined data. If the name entered in the name input field 561a is present in the user-defined data, the SignatureId is saved in the user-defined data. This process links the AccountId and SignatureId. Note that when new user-defined data is added, no other values ​​are set, but the user can register and change them using the user-defined data change form 562.

[0347] S28: By registering the handwritten signature data, the handwriting recognition control unit 26 deletes the handwritten signature registration form 561 from the handwritten input storage unit 25.

[0348] S29: A case where a "check mark" is entered in the registration or cancellation field of the user-defined data change form will be described. In this case, the handwriting recognition control unit 26 transmits the change value entered in the user-defined data change form 562 generated in the handwriting input storage unit 25 by the handwriting input display control unit 23 in step S100 described later to the operation command definition unit 33.

[0349] S30: When the user-defined data change is executed, the handwriting recognition control unit 26 deletes the user-defined data change form 562 from the handwriting input storage unit 25.

[0350] S31: If the registration or cancellation column of the form added in step S100 (to be described later) has an "X" mark, the handwriting recognition control unit 26 deletes the form added in step S100 from the handwriting input storage unit 25.

[0351] S33: If the processing is not form processing, the handwriting recognition control unit 26 transmits handwriting recognition character string candidates handwritten by the user to the handwriting recognition dictionary unit 27. The handwriting recognition dictionary unit 27 transmits language character string candidates that are linguistically likely to be correct to the handwriting recognition control unit 26.

[0352] If "RecommendMenuOff="True"" is currently set in the pen ID control data and the operation commands are also set not to be displayed, the handwriting recognition control unit 26 does not perform the control related to the recognition in steps S33 to S47. This reduces the processing load.

[0353] S34: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidate and the received language character string candidate to the character string conversion control unit 28.

[0354] S35: The character string conversion control unit 28 transmits the handwriting recognition character string candidates and the language character string candidates to the character string conversion dictionary unit 29. The character string conversion dictionary unit 29 transmits the converted character string candidates to the character string conversion control unit 28.

[0355] S36: The character string conversion control unit 28 transmits the received converted character string candidate to the predictive conversion control unit 30.

[0356] S37: The predictive conversion control unit 30 transmits the received converted character string candidates to the predictive conversion dictionary unit 31. The predictive conversion dictionary unit 31 transmits the predicted character string candidates to the predictive conversion control unit 30.

[0357] S38: The predictive conversion control unit 30 transmits the received predicted character string candidate to the operation command recognition control unit 32.

[0358] S39: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) matching the predicted character string candidate.

[0359] Thereafter, the same processing is carried out up to the transmission of operation command candidates in steps S40 to S47. S40: The character string conversion control unit 28 transmits the received converted character string candidate to the operation command recognition control unit 32.

[0360] S41: The operation command recognition control unit 32 transmits the received conversion character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) that matches the conversion character string candidate.

[0361] S42: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidates and the language character string candidates to the predictive conversion control unit 30.

[0362] S43: The predictive conversion control unit 30 transmits the handwritten recognition character string candidate and the received language character string candidate to the predictive conversion dictionary unit 31. The predictive conversion dictionary unit 31 transmits the predicted character string candidate to the predictive conversion control unit 30.

[0363] S44: The predictive conversion control unit 30 transmits the received predicted character string candidates to the operation command recognition control unit 32.

[0364] S45: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) matching the predicted character string candidate.

[0365] S46: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidate and the received language character string candidate to the operation command recognition control unit 32.

[0366] S47: The operation command recognition control unit 32 transmits the handwriting recognition character string candidate and the received language character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) that matches the language character string candidate.

[0367] S48: Next, the handwriting recognition control unit 26 transmits the stroke addition to the operation command recognition control unit 32.

[0368] S49: The operation command recognition control unit 32 transmits the position information of the confirmed object to the handwriting input storage unit 25. The handwriting input storage unit 25 transmits the position information of the confirmed object to the operation command recognition control unit 32.

[0369] S50: The operation command recognition control unit 32 determines the selected object. That is, it determines whether or not the position information of the stroke received from the handwriting recognition control unit 26 when the stroke was added in step S48 and the position information of the confirmed object received from the handwriting input storage unit 25 have a predetermined relationship based on the straddling line determination condition 406 and the encircling line determination condition 407. The operation command recognition control unit 32 stores the confirmed object that can be determined to be selected as the selected object. In this case, since the selected object is identified, candidates for the operation command when the selected object exists are obtained from the operation command definition unit 33.

[0370] In addition, the handwriting recognition control unit 26, the string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 each store handwriting recognition string candidates, language string candidates, conversion string candidates, predicted string candidates, operation command candidates, and data related to the selected object so that they can be acquired in subsequent steps S55 to S58.

[0371] S18-2: Immediately after transmitting the stroke addition command to the handwriting recognition control unit 26 in step S18, the handwriting input display control unit 23 transmits a selectable candidate display timer start command to the candidate display timer control unit 24. The candidate display timer control unit 24 starts this timer.

[0372] The following steps S51 to S53 are executed if a pen-down occurs before a certain time has elapsed (before the timer times out).

[0373] When UI operation is disabled (areas A and C), steps S51 and below are executed.

[0374] S51: If the user touches the pen to the handwriting input unit 21 before the timer times out, the handwriting input unit 21 transmits a pen-down signal (the same event as in step S2) to the handwriting input display control unit 23.

[0375] S52: The handwriting input display control unit 23 transmits a stroke start (same as step S3) to the handwriting input saving unit 25. The subsequent sequence is the same as step S3 and after.

[0376] S53: Furthermore, the handwriting input display control unit 23 transmits a selectable candidate display timer stop command to the candidate display timer control unit 24. The candidate display timer control unit 24 stops the timer because the timer is no longer necessary since pen-down has been detected.

[0377] Steps S54 to S117 are executed if the pen does not go down before a certain time has elapsed (before the timer times out), and therefore the operation guide 500 shown in FIG.

[0378] S54: If the user does not bring the pen into contact with the handwriting input unit 21 while the selectable candidate display timer is in progress, the candidate display timer control unit 24 transmits a timeout to the handwriting input display control unit 23.

[0379] S55: The handwriting input display control unit 23 transmits a request to acquire handwriting recognition character string / language character string candidates to the handwriting recognition control unit 26. The handwriting recognition control unit 26 transmits the handwriting recognition character string / language character string candidates currently held to the handwriting input display control unit 23.

[0380] S56: The handwriting input display control unit 23 transmits a request to acquire a conversion character string candidate to the character string conversion control unit 28. The character string conversion control unit 28 transmits the conversion character string candidate currently held to the handwriting input display control unit 23.

[0381] S57: The handwriting input display control unit 23 transmits a predicted character string candidate acquisition request to the predictive conversion control unit 30. The predictive conversion control unit 30 transmits the predicted character string candidates currently held to the handwriting input display control unit 23.

[0382] S58: The handwriting input display control unit 23 transmits the acquired operation command candidates to the operation command recognition control unit 32. The operation command recognition control unit 32 transmits the operation command candidates and the selected object currently held to the handwriting input display control unit 23.

[0383] S59: Furthermore, the handwriting input display control unit 23 transmits the estimated writing direction acquisition to the handwriting input saving unit 25. The handwriting input saving unit 25 determines the estimated writing direction from the stroke addition time, horizontal distance, and vertical distance of the handwritten object rectangular area, and transmits the estimated writing direction to the handwriting input display control unit 23.

[0384] S60: Next, the handwriting input display control unit 23 specifies the pen ID received from the handwriting input unit 21 and acquires the ColorId, AccountId (if any) and Angle of the current pen ID control data from the pen ID control data storage unit 36.

[0385] S61: The handwriting input display control unit 23 acquires the authentication result of the handwritten signature from the handwritten signature authentication control unit 38. As a result, the user's SignatureId can be obtained. Therefore, when executing the operation commands described later, the AccountId is registered in association with the pen ID control data. When the user signs in, the color definition data associated with the user-defined data specified by the AccountId is specified. The handwriting input display control unit 23 can display the handwritten data in which the color information of the user-defined data is emphasized in black and white, or the text data converted from the handwritten data.

[0386] · When the selectable candidate display is valid (area A), steps S62, S63, and S64 are executed.

[0387] S62: The handwriting input display control unit 23 determines whether "RecommendMenuOff = "True"" is set in the pen ID control data of the pen 2500 used by the user. If "RecommendMenuOff = "True"", the string candidate 539 is not displayed. In this case, only the operation commands can be displayed, or the operation commands may not be displayed either. If "RecommendMenuOff = "False"", the handwriting input display control unit 23 creates selectable candidate display data having these handwriting recognition string candidates (e.g., "gi" in FIG. 28), language string candidates (not displayed in FIG. 28 but e.g., "議"), conversion string candidates (e.g., "議事録", "技量試" in FIG. 28), prediction string candidates (e.g., "技量試を決済", "議事録の送付先" in FIG. 28), and operation command candidates (e.g., "議事録テンプレートを読み込む", "議事録フォルダーに保存する" in FIG. 28). At this time, each selection probability and the estimated writing direction are referred to.

[0388] Also, the handwriting input display control unit 23 rotates the selectable candidate display data (operation guide 500) acquired in step S60 counterclockwise, and displays it by transmitting the rotated selectable candidate display data (operation guide 500) to the display unit 22.

[0389] S63: The handwriting input display control unit 23 also rotates the rectangular area display data (rectangular frame) of the handwritten object and the selected object (handwritten object rectangular area display 503 in FIG. 28) counterclockwise using the angle information acquired in step S60, and transmits it to the display unit 22 for display.

[0390] S64: The handwriting input display control unit 23 transmits a selectable candidate display deletion timer start signal to the candidate display timer control unit 24 to delete the selectable candidate display data after a certain time has passed since it was displayed. The candidate display timer control unit 24 starts this timer. If the operation guide 500, including the operation commands, is not displayed because "RecommendMenuOff="True", the selectable candidate display deletion timer is not started.

[0391] When the selectable candidate display is enabled (area A), it determines whether a candidate has been selected within a certain period of time.

[0392] Steps S65 to S70 are executed when the user erases the selectable candidate display on the display unit 22 while the selectable candidate erasure timer is running, or when a change occurs in the handwritten object (i.e., when a stroke of the handwritten object is added, deleted, moved, transformed, or divided), or when no candidate is selected before the timer times out.

[0393] Furthermore, steps S65 to S67 are executed when the candidate display is erased or a change occurs in the handwritten object.

[0394] S65: The handwriting input unit 21 transmits to the handwriting input display control unit 23 a notice that the selectable candidate display has been deleted or that the handwritten object has been changed.

[0395] S66: The handwriting input display control unit 23 sends a command to stop the selectable candidate deletion timer. The candidate display timer control unit 24 stops the timer. This is because an operation has been performed on the handwritten object within a certain period of time, making the timer unnecessary.

[0396] S67: The handwriting input display control unit 23 stores the position information of the operation guide 500 in the handwriting recognition control unit 26 so that it can be used in the gesture determination for the gesture handwriting recognition in step S19. The position information is, for example, the coordinates of the upper left corner and the lower right corner or coordinates equivalent thereto. This allows the handwriting recognition control unit 26 to determine whether the straight line used to input the angle information is within the operation guide 500.

[0397] S69: The handwriting input display control unit 23 erases the display by sending a command to erase the selectable candidate display data to the display unit 22. If "RecommendMenuOff="True", only the operation command needs to be erased, or nothing needs to be erased.

[0398] S70: The handwriting input display control unit 23 erases the display by transmitting a command to erase the rectangular area display data of the handwritten object and the selected object to the display unit 22. Therefore, when the display of the operation command candidate is erased under conditions other than the selection of the operation command candidate, the handwritten object remains displayed as is.

[0399] S68: On the other hand, if the selectable candidate display is not erased or the handwritten object is not changed while the selectable candidate deletion timer is started (if the user does not perform a pen operation), the candidate display timer control unit 24 sends a timeout to the handwritten input display control unit 23.

[0400] Similarly, after the selectable candidate display deletion timer times out, the handwriting input display control unit 23 executes steps S69 and S70. This is because the selectable candidate display data and the rectangular area display data of the handwritten object and the selected object may be deleted after a certain period of time has elapsed.

[0401] When the display of selectable candidates is disabled (areas B and C) and the selectable candidate deletion timer is running, if the user selects a selectable candidate, steps S71 to S117 are executed. When the display of selectable candidates is disabled, the handwriting recognition control unit 26 displays the candidate with the highest probability.

[0402] S71: When the user selects a selectable candidate while the selectable candidate deletion timer is starting, the handwriting input unit 21 transmits the character string candidate or operation command candidate selection to the handwriting input display control unit 23.

[0403] S71-2: The handwriting input display control unit 23 transmits a selectable candidate display deletion timer stop command to the candidate display timer control unit 24. The candidate display timer control unit 24 stops this timer.

[0404] S72: Next, the handwriting input / display control unit 23 transmits a command to clear the stored data to the handwriting recognition control unit 26.

[0405] S73: The handwriting recognition control unit 26 sends a command to clear the stored data to the character string conversion control unit 28.

[0406] S74: The handwriting recognition control unit 26 transmits a command to clear the stored data to the predictive conversion control unit 30.

[0407] S75: The handwriting recognition control unit 26 sends a retained data clear command to the operation command recognition control unit 32. The handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 clear the data relating to character string candidates and operation command candidates that have been retained up to this point.

[0408] When the selectable candidate display is enabled (area A), steps S76 and S77 are executed.

[0409] S76: Next, the handwriting input display control unit 23 transmits a command to erase the selectable candidate display data to the display unit 22, thereby erasing the display.

[0410] S77: The handwriting input display control unit 23 transmits a command to erase the rectangular area display data of the handwritten object and the selected object to the display unit 22, thereby erasing the display.

[0411] If stroke display is enabled (ForceForDialog in area A or area C), step S78 is executed. S78: The handwriting input display control unit 23 erases the display by transmitting to the display unit 22 a command to erase the handwritten object display data and the pen coordinate complement display data transmitted in step S6. This is because the handwritten object, etc., is no longer necessary since a character string candidate or an operation command candidate has been selected.

[0412] S79: The handwriting input display control unit 23 transmits a request to delete the handwritten object to the handwriting input storage unit 25.

[0413] If a character string candidate is selected, steps S80 to S82 are executed.

[0414] When the display of selectable candidates is disabled (areas B and C), or when a character string candidate is selected, steps S80 to S82 are executed.

[0415] S80: When the dialog control is disabled (areas A and B), step S80 is executed. The handwriting input display control unit 23 transmits a request to add a character string object to the handwriting input saving unit 25. That is, even if a selectable candidate is not displayed (even if the user does not select one), the character string object is forcibly added.

[0416] S81: Furthermore, the handwriting input display control unit 23 transmits a character string object font acquisition to the handwriting input saving unit 25. The handwriting input saving unit 25 identifies the font name of the character string object associated with the PenId of the pen ID control data from the pen color selection button definition data, and also transmits an estimated character size of the handwritten object to the handwriting input display control unit 23.

[0417] S82: Next, the handwriting input display control unit 23 uses the defined font received from the handwriting input storage unit 25 to transmit character string object display data to the display unit 22 to be displayed in the same position as the handwritten object, thereby displaying the character string object. The handwriting input display control unit 23 identifies the line type and thickness from the color definition data based on the ColorId of the pen ID control data, and displays the text (character string object) rotated using the angle information. Therefore, it is possible to display a character string object with the top and bottom facing up and down as viewed from the user's operating position. Also, if the handwriting input area is area C in Figure 24, the character string object is displayed at a position and with a font (predetermined position and font) specified by the dialog control. In Figure 22, this is a mask display 2704 or a handwritten data recognition result 2709.

[0418] When a character string object is displayed in the mask display area, steps S83 to S94 are executed. Steps S83 to S94 are looped for each mask display area.

[0419] S83: Since the character string object is displayed in the mask display area, the handwriting input display control unit 23 requests the mask display timer control unit 40 to start the mask display timer. Immediately after the character string object is displayed, the mask display timer (1) 408 is used.

[0420] S84: When the user puts the pen down on the mask display area, the handwriting input unit 21 detects a pen down event and transmits to the handwriting input display control unit 23 a message that the pen has been put down on the mask display area.

[0421] S85: The handwriting input display control unit 23 requests the mask display timer control unit 40 to start (restart) the mask display timer. As a result, the mask display timer (1) 408 is reset.

[0422] S86: When the mask display timer (1) 408 times out, the mask display timer control unit 40 transmits a timeout to the handwriting input display control unit 23.

[0423] S87: To perform masked display, the handwriting input display control unit 23 requests the display unit 22 to erase the character string object display data.

[0424] S88: Next, the handwriting input display control unit 23 transmits mask display data of the character string object (for example, ● for the number of characters) to the display unit 22.

[0425] S89: If the user wants to display the character string object again in the mask display area, the user inputs a gesture into the mask display area. The handwriting input unit 21 detects a pen-down event. The handwriting input unit 21 transmits to the handwriting input display control unit 23 a message that the pen has been brought down on the mask display area.

[0426] S90: The handwriting input / display control unit 23 transmits a gesture recognition request to the handwriting recognition control unit 26.

[0427] S91: The handwriting recognition control unit 26 transmits the gesture recognition result indicating whether the gesture is determined to be a tap or the like to the handwriting input / display control unit 23.

[0428] If the gesture recognition result is a gesture to release the mask display of a character string object, steps S92 to S94 are executed.

[0429] S92: The handwriting input display control unit 23 transmits a command to the display unit 22 to erase the mask display data of the character string object.

[0430] S93: Next, the handwriting input display control unit 23 transmits character string object display data to the display unit 22. The display unit 22 displays the character string object in the mask display area.

[0431] S94: Since the character string object is displayed again in the mask display area, the handwriting input display control unit 23 requests the mask display timer control unit 40 to start the mask display timer. Immediately after the character string object is displayed again, the mask display timer (2) 409 is used.

[0432] Note that the removal of the mask display may be limited to the user who entered the character string. As shown in FIG. 16, the handwritten data includes a Pen ID. The handwriting input display control unit 23 permits the removal of the mask display only if the Pen ID of the pen 2500 used for tapping matches the Pen ID of the handwritten data that is masked. This prevents any user from removing the mask display. Alternatively, since the Pen ID and the Account ID are associated with each other in the pen ID control data in FIG. 17, the removal of the mask display may be permitted if the user-defined data in FIG. 14 identified by the Account ID has the authority to remove the mask display.

[0433] When a candidate for an operation command is selected, steps S95 to S115 are executed. An operation command for the selected object for editing or modification is executed in steps S95 to S97.

[0434] S95: When a candidate for an operation command for a selected object is selected (when a selected object exists), the handwriting input display control unit 23 erases the display by sending a command to erase the display data of the selected object to the display unit 22. This is to erase the original selected object for now.

[0435] S96: Next, the handwriting input display control unit 23 transmits an operation command execution for the selected object to the handwriting input saving unit 25. The handwriting input saving unit 25 transmits display data of the newly selected object (display data after editing or modification) to the handwriting input display control unit 23.

[0436] S97: Next, the handwriting input display control unit 23 transmits the selected object display data to the display unit 22, thereby re-displaying the selected object after the operation command is executed.

[0437] Although there is a selected object, the operation command "add page name" is executed in steps S98 to S99.

[0438] S98: When a candidate for the operation command "Add page name" is selected (when a selected object exists), the handwriting input display control unit 23 requests the handwriting input saving unit 25 to acquire the selected object. The handwriting input saving unit 25 transmits the selected object to the handwriting input display control unit 23.

[0439] S99: The handwritten input display control unit 23 checks the page number currently displayed on the display 220 using the page navigation data, and adds the selected object to the handwritten input storage unit 25 as a page name.

[0440] S100: A case will be described where “register handwritten signature” in the operation command definition data 713 or “change settings” in the operation command definition data 716 is specified. The handwritten input display control unit 23 adds the handwritten signature registration form 561 or the user-defined data change form to the handwritten input storage unit 25.

[0441] S101: When the operation command candidate of “save file” or “print” is selected, the handwriting input display control unit 23 transmits a file transmission request to the file transmission / reception control unit 37.

[0442] S102: The file transmission / reception control unit 37 transmits to the handwritten input storage unit 25 a request to acquire the handwritten input storage data to be transmitted.

[0443] S103: The handwritten input storage unit 25 determines whether the destination is a color-compatible device using the MIB or the like. In the case of a color-compatible device, the handwriting input saving unit 25 converts the handwriting input saved data into color and transmits it to the file transmission / reception control unit 37. That is, the handwriting input saving unit 25 determines whether the data originates from handwriting based on the ColorId of the handwriting input saved data. In the case of data originating from handwriting, the handwriting input saving unit 25 obtains color information associated with the ColorId from the color definition data and converts it to this color information. Data not originating from handwriting originally contains color information, so the handwriting input saving unit 25 transmits this color information. It is sufficient to be able to determine whether the data originates from handwriting (handwritten data drawn on the display device 2) without using the ColorId. Alternatively, a value indicating a device indicating whether the data was drawn on the display device 2 may be used. A case where the destination is a black-and-white compatible device will be described. In the case of data originating from handwriting, the handwriting input saving unit 25 may ignore the color information associated with the ColorId of the handwriting input saving data and simply send a sequence of coordinate points or text. The handwriting input saving unit 25 may obtain the color information associated with the ColorId from the color definition data and convert it into this color information. However, the handwriting input saving unit 25 may also obtain the color information associated with the ColorId from the color definition data and convert it into black-and-white highlighting (line type and thickness) before sending it. Since data not originating from handwriting originally contains color information, the handwriting input saving unit 25 sends this color information. When the destination is a black-and-white emphasis compatible device, the handwriting input saving unit 25 sends only the handwriting input saved data, or the handwriting input saved data and color definition data to the file transmission / reception control unit 37. When only the handwriting input saved data is sent, the line type and thickness are determined according to the color definition data held by the destination black-and-white emphasis compatible device. When the handwriting input saving unit 25 sends both the handwriting input saved data and color definition data, the line type and thickness are determined according to the color definition data held by the source black-and-white emphasis compatible device. Note that the source black-and-white emphasis compatible device may convert the data to black-and-white emphasis display before sending. Furthermore, when the file transmission / reception control unit 37 writes out data derived from handwriting to, for example, a PDF file, it obtains color information associated with ColorId from the color definition data and sets it in the PDF file according to the format of the PDF file. The same applies to fonts, character sizes, etc. Furthermore, all or part of the saved handwritten input data (e.g., ColorId, FontName, etc.) is saved in the metadata of the PDF file. Because PDF file readers ignore metadata, there is no effect on the display, and when the display device 2 reads this PDF file, it can reproduce the saved handwritten input data from the metadata. In the case of data not derived from handwriting, the color information, font, character size, etc. contained in the saved handwritten input data are set in the PDF file according to the format of the PDF file.

[0444] S104: The file transmission / reception control unit 37 transmits the handwritten input data received from the handwritten input storage unit to the destination or writes it to a file.

[0445] S105: When the operation command candidate “read file” is selected, the handwriting input display control unit 23 transmits a file list information acquisition request to the file transmission / reception control unit 37.

[0446] S106: The file transmission / reception control unit 37 receives file list information from a storage medium such as a USB memory, a network storage, or a Web server, or from an external device.

[0447] S107: The file transmission / reception control unit 37 transmits the file list information to the handwriting input display control unit 23.

[0448] S108: The handwriting input display control unit 23 transmits file list display data to the display unit 22. As a result, the display unit 22 displays a file list on the display as shown in FIG.

[0449] S109: When the user selects a file and the handwriting input unit 21 accepts it, the handwriting input unit 21 transmits the file selection to the handwriting input display control unit 23.

[0450] S110: The handwriting input display control unit 23 transmits a file reception request for the selected file to the file transmission / reception control unit 37.

[0451] S111: The file transmission / reception control unit 37 acquires a file from an external device.

[0452] S112: The file transmission / reception control unit 37 stores the file in the handwritten input storage unit 25.

[0453] S113: The handwriting input saving unit 25 analyzes the file received from the file transmission / reception control unit 37 and converts the handwritten data into handwritten input saved data (black and white emphasis / color convertible data). That is, it determines whether or not there is metadata, and if there is metadata, it determines whether conversion to handwritten input saved data is possible (whether or not there is a Color ID, etc.), and saves the data as handwritten input saved data. Then, it reads the Color ID of the handwritten data, refers to the color definition data, and converts it to black and white emphasis display associated with the Color ID. The handwriting input saving unit 25 transmits display data of the handwritten object to the handwriting input display control unit 23. In the case of data not originating from handwriting, the handwriting input saving unit 25 reads color information, font, character size, etc. according to the file format and saves it as handwritten input saved data.

[0454] S114: The handwriting input display control unit 23 displays the display data of the handwriting input data on the display unit 22. As a result, the display unit 22 displays the data originating from handwriting as handwritten data with enhanced black and white, and displays the data that is not originating from handwriting in black and white using conventional brightness conversion.

[0455] When the operation command 512 for signing in is executed, the handwriting input display control unit 23 acquires the pen ID received by the display device 2 when the operation command 512 is executed. The handwriting input display control unit 23 identifies the user-defined data having the Signature ID acquired in step S61 and acquires the Account ID from the user-defined data. Then, the Account ID is registered in the pen ID control data in association with the pen ID. This associates the pen 2500 with the user, and the display device 2 becomes able to perform processing using the user-defined data.

[0456] When the user writes by hand or loads a file after signing in, the handwriting input display control unit 23 obtains from the pen ID control data the Account Id associated with the pen ID received by the display device 2 when the operation command is executed. The handwriting input display control unit 23 identifies the user-defined data by this Account Id, and sets color definition data, etc. in % to % of the operation command before executing it.

[0457] S115: If an input / output operation command is selected, the handwriting input display control unit 23 executes the operation command character string (Command) of the operation command definition data corresponding to the operation command selected by the user.

[0458] S116: If the user manually inputs angle information, the handwriting input display control unit 23 stores the received angle information in the pen ID control data storage unit 36 ​​in association with the pen ID received from the pen 2500 when the rotation operation button 511 is pressed.

[0459] S117: For the next handwritten object, the handwritten input display control unit 23 transmits a handwritten object start signal to the handwritten input saving unit 25. The handwritten input saving unit 25 reserves a handwritten object area. After this, the processes of steps S2 to S117 are repeated.

[0460] <Display control based on definition data of handwriting input area> With reference to Fig. 54, the display control based on the definition data of the handwriting input area will be explained in more detail. Fig. 54 is a diagram for explaining a method in which the display device controls the display of data depending on the area including the pen coordinates. When the user writes by hand with an input means (pen or fingertip), the handwriting input display control unit 23 determines which of areas A to C the coordinates at which the input means contacts the display 220 are (S201). In detail, the handwriting input display control unit 23 determines which handwriting input area definition data to apply based on the Position and Depth of the definition data of the handwriting input area in Fig. 23.

[0461] In the case of area A (Yes in S201), Recognition="Enable" is satisfied, so the display unit 22 displays the handwritten data, the handwriting recognition control unit 26 recognizes the characters, and the display unit 22 displays the operation guide 500. When the user selects a character string candidate, the display unit 22 displays the selected character string candidate (S202).

[0462] In the case of area B (Yes in S203), Recognition="Disable" is set, so the handwriting recognition control unit 26 accepts the stroke as a button operation (S204), and therefore the handwritten data is not displayed or otherwise processed.

[0463] If the area is C and Recognition="ForceForDialog", the display unit 22 displays the handwritten data, the handwriting recognition control unit 26 performs character recognition, and the character string with the highest probability is displayed in a predetermined position on the display unit 22 (S206). In other words, the operation guide 500 is not displayed, and the character string can be displayed without the user having to select it.

[0464] If the area is C and Recognition="ForceForDialogWithoutStroke", the display unit 22 does not display the handwritten data, and the handwriting recognition control unit 26 performs character recognition, and the character string with the highest probability is displayed in a masked position on the display unit 22 (S207). In other words, the strokes are not displayed so that it is not possible to tell what the user has handwritten, and the recognition result can also be displayed in a masked position.

[0465] <Display control based on mask display area definition data> Display control based on definition data of the mask display area will be explained further with reference to Fig. 55. Fig. 55 is a diagram for explaining a method in which the display device controls the display of data according to the mask display area including the pen coordinates. When the user writes by hand with an input means (pen or fingertip), the handwriting input display control unit 23 determines whether the coordinates where the input means contacts the display 220 are in the name area, the phonetic reading area, the gender area, or the age area (S301).

[0466] In the case of a name area (Yes in S301), Recognition="Enable" so that the display unit 22 displays the handwritten data, the handwriting recognition control unit 26 recognizes the characters, and the display unit 22 displays the operation guide 500. When the user selects a character string candidate, the display unit 22 displays the selected character string candidate (S302). Also, because MaskedDisplay="True", the character string is displayed in a masked state when the mask display timer (1) 408 times out.

[0467] If it is a furigana area (Yes in S303), Recognition="Force" so the display unit 22 displays the handwritten data, and the handwriting recognition control unit 26 forcibly recognizes the characters while handwriting is being displayed. The display unit 22 displays hiragana in the furigana area (S304). Also, MaskedDisplay="True" so the character string is displayed in a masked state when the mask display timer (1) 408 times out.

[0468] In the case of the gender area (Yes in S305), since Recognition="Force", the display unit 22 displays the handwritten data, and the handwriting recognition control unit 26 forcibly recognizes the characters as either male or female. The display unit 22 displays male or female in the gender area (S306). Also, since there is no definition of MaskedDisplay in the gender area, no masked display is performed.

[0469] In the case of the age field (Yes in S307), Recognition="Force" is set, so the display unit 22 displays the handwritten data, and the handwriting recognition control unit 26 forcibly recognizes the characters as numbers. The display unit 22 displays the numbers in the age field (S305). Since there is no definition of MaskedDisplay in the gender field, no masked display is performed.

[0470] <Mask display and unmask of mask display area> Fig. 56 is an example of a flowchart illustrating the flow of mask display and mask cancellation in the mask display area based on the mask display timer (1) and the mask display timer (2). The process in Fig. 56 starts with a handwritten object displayed in the mask display area.

[0471] The handwriting input display control unit 23 determines whether or not a character string is displayed in the mask display area (Recognition=Force) (S401).

[0472] When a character string is displayed in the mask display area (Yes in S401), the mask display timer control unit 40 starts the mask display timer (1) to measure the time until the mask is displayed (S402).

[0473] The mask display timer control unit 40 monitors the mask display timer (1) and determines whether or not it has timed out (S403).

[0474] Until the timeout occurs (No in S403), the handwriting input display control unit 23 monitors whether or not the user has touched the mask display area (S404).

[0475] When the user touches the mask display area (Yes in S404), the handwriting input display control unit 23 requests the mask display timer control unit 40 to restart the mask display timer (1).

[0476] If the mask display timer (1) times out (Yes in S403), the handwriting input display control unit 23 displays the character string in the mask display area as a mask (S405).

[0477] After the mask display, the handwriting input display control unit 23 determines whether or not a gesture (tap) operation is performed on the mask display area (S406).

[0478] If a gesture operation has been performed (Yes in S406), the handwriting input display control unit 23 removes the mask and displays the character string (S407).

[0479] In order to measure the time until the mask display is performed again, the mask display timer control unit 40 starts the mask display timer (2) (S408).

[0480] The mask display timer control unit 40 monitors the mask display timer (2) and determines whether or not it has timed out (S409).

[0481] Until the timeout occurs (No in S409), the handwriting input display control unit 23 monitors whether or not the user has touched the mask display area (S410).

[0482] When the user touches the mask display area (Yes in S410), the handwriting input display control unit 23 requests the mask display timer control unit 40 to restart the mask display timer (2).

[0483] If the mask display timer (2) times out (Yes in S409), the handwriting input display control unit 23 displays the character string in the mask display area again in a masked state (S411).

[0484] In this way, the display device 2 can mask the character string in the mask display area under timer control and then display it again with a gesture from the user.

[0485] <Major Effects> As described above, the display device 2 of this embodiment displays the entire confidential information for a short period of time immediately after the confidential information is displayed, making it useful for business purposes. Furthermore, the confidential information is masked after a certain period of time has passed, so that it is protected from third parties.

[0486] In addition, users can remove the mask display with a gesture, but if the gesture is complicated, a third party cannot easily remove the mask. Therefore, it is easy for users to use and confidentiality can be improved. If the gesture is simple, even inexperienced users can easily remove the mask of confidential information.

[0487] The user can set the timer from when confidential information is displayed until it is masked, or the timer from when confidential information is displayed again by gesture until it is masked, according to the manner in which the display device 2 is used. For example, the timer can be set to a shorter time in places where there is a lot of third party traffic, and a longer time in places where there is not. [Example]

[0488] In the first embodiment, the display device 2 is described as having a large touch panel, but the display device is not limited to having a touch panel. In this embodiment, a projector-type display device will be described.

[0489] <<Another example 1 of the display device configuration>> Fig. 57 is a diagram showing another example of the configuration of a display device. In Fig. 57, a projector 411 is installed on the upper side of a normal whiteboard 413. This projector 411 corresponds to the display device. The normal whiteboard 413 is not a flat panel display integrated with a touch panel, but a whiteboard on which the user directly writes by hand with a marker. The whiteboard may also be a blackboard, as long as it is flat and large enough to project an image.

[0490] The projector 411 has an ultra-short focus optical system and can project an image with little distortion from a distance of about 10 cm onto the whiteboard 413. This image may be transmitted from the PC 400-1 connected wirelessly or by wire, or may be stored in the projector 411.

[0491] The user writes by hand on the whiteboard 413 using a dedicated electronic pen 2700. The electronic pen 2700 has a light-emitting part, for example at its tip, that is switched on and emits light when the user presses it against the whiteboard 413 to write. The light has a wavelength of near-infrared or infrared, so it is invisible to the user's eyes. The projector 411 has a camera that captures an image of the light-emitting part and analyzes the image to identify the direction of the electronic pen 2700. The electronic pen 2700 also emits sound waves along with the light emission, and the projector 411 calculates the distance based on the time it takes for the sound waves to reach the screen. The position of the electronic pen 2700 can be identified based on the direction and distance. A stroke is drawn (projected) at the position of the electronic pen 2700.

[0492] The projector 411 projects the menu 430, so when the user presses a button with the electronic pen 2700, the projector 411 identifies the pressed button based on the position of the electronic pen 2700 and the ON signal of the switch. For example, when the save button 431 is pressed, the projector 411 saves the strokes (a set of coordinates) handwritten by the user. The projector 411 saves the handwritten information in a predetermined server 412, USB memory 2600, or the like. The handwritten information is saved for each page. Since the coordinates are saved as they are, rather than as image data, the user can re-edit the information. However, in this embodiment, the menu 430 does not need to be displayed because operation commands can be called up by handwriting.

[0493] <<Another example 2 of the display device configuration>> 58 is a diagram showing another example of the configuration of the display device 2. In the example of Fig. 58, the display device 2 includes a terminal device 600, an image projection device 700A, and a pen motion detection device 810.

[0494] The terminal device 600 is connected by wire to the image projection device 700A and the pen motion detection device 810. The image projection device 700A projects image data input by the terminal device 600 onto the screen 800.

[0495] The pen operation detection device 810 communicates with the electronic pen 820 and detects the operation of the electronic pen 820 in the vicinity of the screen 800. Specifically, the electronic pen 820 detects coordinate information indicating a point indicated by the electronic pen 820 on the screen 800 and transmits the coordinate information to the terminal device 600.

[0496] The terminal device 600 generates image data of a stroke image input by the electronic pen 820 based on the coordinate information received from the pen operation detection device 810, and causes the image projection device 700A to draw the stroke image on the screen 800.

[0497] Furthermore, the terminal device 600 generates superimposed image data that indicates a superimposed image obtained by combining the background image projected by the image projection device 700A and the stroke image input by the electronic pen 820.

[0498] <<Another example 3 of the display device configuration>> 59 is a diagram showing an example of the configuration of a display device. In the example of Fig. 59, the display device includes a terminal device 600, a display 800A, and a pen motion detection device 810.

[0499] The pen operation detection device 810 is placed near the display 800A, detects coordinate information indicating a point indicated by the electronic pen 820A on the display 800A, and transmits the coordinate information to the terminal device 600. In the example of Fig. 59, the electronic pen 820A may be charged by the terminal device 600 via a USB connector.

[0500] Based on the coordinate information received from the pen operation detection device 810, the terminal device 600 generates image data of the stroke image input by the electronic pen 820A, and displays the image data on the display 800A.

[0501] <<Another example 4 of the display device configuration>> Fig. 60 is a diagram showing a configuration example of a display device. In the example of Fig. 60, the display device includes a terminal device 600 and an image projection device 700A.

[0502] The terminal device 600 performs wireless communication (Bluetooth (registered trademark) or the like) with the electronic pen 820B and receives coordinate information of a point indicated by the electronic pen 820B on the screen 800. Then, based on the received coordinate information, the terminal device 600 generates image data of a stroke image input by the electronic pen 820B and causes the image projection device 700A to project the stroke image.

[0503] Furthermore, the terminal device 600 generates superimposed image data that indicates a superimposed image obtained by combining the background image projected by the image projection device 700A and the stroke image input by the electronic pen 820.

[0504] As described above, the above-described embodiments can be applied to various system configurations.

[0505] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0506] For example, the display method of the present embodiment can be suitably applied to any information processing device having a touch panel. Furthermore, devices having the same functions as a display device are also referred to as electronic blackboards, electronic whiteboards, electronic information boards, interactive boards, etc. Examples of information processing devices equipped with a touch panel include output devices such as PJs (Projectors), digital signage, HUDs (Head Up Display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, network appliances, notebook PCs (Personal Computers), mobile phones, smartphones, tablet devices, game consoles, PDAs (Personal Digital Assistants), digital cameras, wearable PCs, and desktop PCs.

[0507] In this embodiment, the server may perform part of the processing performed by the display device 2. For example, the display device transmits stroke information to the server, and acquires and displays information to be displayed on the operation guide 500 from the server.

[0508] In this embodiment, the coordinates of the pen tip are detected by detecting them on a touch panel, but the coordinates of the pen tip may also be detected using ultrasonic waves. The pen emits light and also emits ultrasonic waves, and the display device 2 calculates the distance based on the time it takes for the ultrasonic waves to reach the screen. The position of the pen can be identified based on the direction and distance. The projector draws (projects) the trajectory of the pen as a stroke.

[0509] In this embodiment, when there is a selected object, candidate edit, modify and "add to page name" operation commands are displayed, and when there is no selected object, candidate input / output operation commands are displayed. However, candidate edit, modify and "add to page name" operation commands and candidate input / output operation commands may be displayed simultaneously.

[0510] Furthermore, the user's handwritten signature data does not have to be stored in the display device 2. It may be stored on the cloud or in an in-house information processing device.

[0511] In addition, the configuration examples in Figure 6 and the like are divided according to main functions to make it easier to understand the processing by the display device 2. The method of dividing the processing units and their names do not limit the present invention. The processing by the display device 2 can be divided into even more processing units depending on the processing content. Also, it can be divided so that one processing unit includes even more processes.

[0512] Furthermore, in this embodiment, even if a threshold value is exemplified as a comparison target, the threshold value is not limited to the exemplified value. Therefore, in this embodiment, with respect to all threshold values, the terms "less than threshold" and "below threshold" have the same meaning, and the terms "exceeding threshold" and "above threshold" have the same meaning. For example, when the threshold value is 11, "below threshold" has the same meaning as "below threshold" when the threshold value is 10. Furthermore, when the threshold value is 10, "exceeding threshold" has the same meaning as "above threshold" when the threshold value is 11.

[0513] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0514] The handwriting input display control unit 23 is an example of a display control means, and the handwriting recognition control unit 26 is an example of a conversion means. [Explanation of symbols]

[0515] Display device 2 [Prior art documents] [Patent documents]

[0516] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-002522

Claims

1. A display device that displays handwritten data input by an input means, a conversion means for converting the handwritten data into a character string; a display control means for concealing the character string converted by the conversion means after a first time period has elapsed since the character string was displayed; A display device comprising:

2. The display device according to claim 1, characterized in that, when the coordinates at which handwritten data is input by the input means are included in a predetermined area, the display control means conceals the character string converted by the conversion means after the first time has elapsed since the character string was displayed.

3. The display device according to claim 2, characterized in that, after the display control means displays the character string converted by the conversion means, if the handwritten data is input into the area by the input means before the first time period has elapsed, the first time period is restarted.

4. whether or not the conversion means converts the handwritten data into the character string is set in association with the area; A display device according to any one of claims 1 to 3, characterized in that when the coordinates at which handwritten data is input by the input means are included in an area where the conversion means converts the handwritten data into a character string, the conversion means converts the handwritten data into the character string.

5. whether to display selectable candidates having one or more character strings converted from the handwritten data by the conversion means and to accept selection of the candidates; A display device according to any one of claims 1 to 4, characterized in that when the coordinates at which handwritten data is input by the input means are included in an area where the selectable candidates are displayed and selection is accepted, the selectable candidates are displayed and one of the character strings is selected from the selectable candidates.

6. The display device according to claim 4 or 5, characterized in that for each area, it is set whether the conversion means converts the handwritten data into the character string, whether the conversion means displays and accepts selection of selectable candidates having one or more character strings converted from the handwritten data, and whether the handwritten data input by the input means is displayed.

7. a type of dictionary for converting the handwritten data into the character string by the conversion means is set in association with the area; The display device according to any one of claims 1 to 6, characterized in that the conversion means converts the handwritten data into the character string using the dictionary associated with an area including coordinates into which the handwritten data is input by the input means.

8. If a dictionary for converting numbers or hiragana is set in the area, 8. The display device according to claim 7, wherein the area is associated with display of selectable candidates having one or more character strings converted from the handwritten data by the conversion means and an indication that selection will not be accepted.

9. When predetermined handwritten data is input by the input means into the area where the display control means has concealed the character string, 9. The display device according to claim 2, wherein the display control means displays the concealed character string again.

10. 10. The display device according to claim 9, wherein the display control means conceals the character string that has been displayed again after a second time has elapsed since the character string was displayed.

11. 11. The display device according to claim 10, wherein the first time and the second time are different from each other.

12. 10. The display device according to claim 9, wherein the predetermined handwritten data is a dot.

13. 10. The display device according to claim 9, wherein the predetermined handwritten data can be changed by a user.

14. A display method performed by a display device that displays handwritten data input by an input means, comprising: A conversion means converts the handwritten data into a character string; a step of display control means concealing the character string converted by the conversion means after a first time period has elapsed since the character string was displayed; A display method comprising:

15. a display device for displaying handwritten data input by the input means, a conversion means for converting the handwritten data into a character string; a display control means for concealing the character string converted by the conversion means after a first time period has elapsed since the character string was displayed; A program to function as a

Images