Display device, color-compatible device, display method, and program

Abstract

To provide a display device capable of identifying handwritten data drawn on a specific display device.SOLUTION: A display device for displaying data is provided, the display device being configured to record information showing that handwritten data was drawn on the display device in association with the handwritten data.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a display device, a color-compatible device, a display method, and a program. [Background technology]

[0002] Display devices that display data handwritten on a touch panel with a pen or finger are known. Display devices equipped with relatively large touch panels are placed in places such as conference rooms and used by multiple users as electronic whiteboards.

[0003] On a typical display device, the user can switch the color and thickness of the characters using a predetermined button, and the display device can display handwritten data in the color and thickness selected by the user.

[0004] Meanwhile, there have been attempts to use electronic paper and other devices as display devices, but general electronic paper cannot display color. Therefore, when a monochrome device such as electronic paper reads a color file from an external device, the monochrome device converts the color information into monochrome density and displays it (see, for example, Patent Document 1). Patent Document 1 discloses a conversion method for converting image data by making appropriate corrections according to the characteristics of the color image. Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional technology has a problem in that it is not possible to specify handwritten data drawn on a specific display device.

[0006] In view of the above-mentioned problems, an object of the present invention is to provide a display device that can identify handwritten data drawn on a specific display device. [Means for solving the problem]

[0007] In view of the above-mentioned problems, the present invention is a display device for displaying data, characterized in that it records, in association with the handwritten data, information that the data is handwritten data drawn on the display device. [Effects of the Invention]

[0008] It is possible to provide a display device that can identify handwritten data drawn on a specific display device. [Brief explanation of the drawings]

[0009] [Figure 1] 1A and 1B are diagrams illustrating an example of conversion from color to black and white displayed by a black and white device. [Figure 2] 1A and 1B are diagrams for explaining an outline of a method for displaying handwritten input-derived data by the display device of the present embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a perspective view of a pen. [Figure 4] FIG. 1 is a diagram showing an example of an overall configuration diagram of a display device. [Figure 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device. [Figure 6] FIG. 2 is a diagram illustrating functions of a display device and a pen. [Figure 7] FIG. 10 is a diagram illustrating an example of predefined control data. [Figure 8] FIG. 4 is a diagram showing an example of dictionary data in a handwriting recognition dictionary unit. [Figure 9] FIG. 10 is a diagram illustrating an example of dictionary data in a character string conversion dictionary unit. [Figure 10] FIG. 10 is a diagram illustrating an example of dictionary data in a predictive conversion dictionary unit. [Figure 11] 10A and 10B are diagrams illustrating an example of operation command definition data and system definition data held by an operation command definition unit. [Figure 12] FIG. 10 is a diagram showing an example of file list display data displayed on a display unit. [Figure 13] 10A and 10B are diagrams showing an example of operation command definition data in the case where a selected object is selected by a handwritten object; [Figure 14] 10 is a diagram showing an example of user-defined data held in an operation command definition unit; FIG. [Figure 15] 10 is a diagram showing an example of handwritten signature data stored in a handwritten signature data storage unit. FIG. [Figure 16] 10 is a diagram showing an example of handwritten input saved data saved in a handwritten input saving unit; FIG. [Figure 17] 10A and 10B are diagrams illustrating pen ID control data stored in a pen ID control data storage unit. [Figure 18] FIG. 10 is a diagram illustrating an example of color definition data. [Figure 19] 19A and 19B are diagrams showing an example of data displayed in black and white highlighting and data displayed in a color display method using the color definition data of FIG. 18. [Figure 20] FIG. 10 is a diagram illustrating an example of pen color selection button definition data. [Figure 21] FIG. 4 is a diagram showing an example of pen buttons displayed on a display. [Figure 22] 10A and 10B are diagrams illustrating an example of an operation guide and selectable candidates displayed by the operation guide. [Figure 23] 10A and 10B are diagrams illustrating an example of designating a selected object; [Figure 24] 10A and 10B are diagrams showing an example of display of operation command candidates based on operation command definition data when a handwritten object is present; [Figure 25] 10A and 10B are diagrams showing an example of display of operation command candidates based on operation command definition data when a handwritten object is present; [Figure 26] FIG. 10 is a diagram illustrating a method for inputting 90-degree angle information. [Figure 27] FIG. 10 is a diagram illustrating a method for registering handwritten signature data. [Figure 28] FIG. 10 is a diagram showing an example of an operation guide that is displayed when a user handwrites "Suzuki," which is handwritten signature data that has been registered. [Figure 29] FIG. 10 is a diagram illustrating a method for changing user-defined data. [Figure 30]FIG. 10 is a sequence diagram (part 1) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 31] FIG. 10 is a sequence diagram (part 2) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 32] FIG. 10 is a sequence diagram (part 3) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 33] FIG. 10 is a sequence diagram (part 4) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 34] FIG. 5 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 5). [Figure 35] FIG. 10 is a sequence diagram (part 6) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 36] FIG. 10 is a sequence diagram (part 7) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 37] FIG. 8 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 8). [Figure 38] 10A and 10B are diagrams illustrating an outline of a method for displaying handwritten input-derived data on a display device in the case of English conversion. [Figure 39] 10 is a diagram for explaining an outline of a method for displaying handwriting input-derived data by the display device 2 of this embodiment in the case of Chinese conversion. FIG. [Figure 40] FIG. 10 is a diagram showing an example of operation command definition data when there is no selected object in English conversion. [Figure 41] FIG. 10 is a diagram showing an example of operation command definition data when there is a selected object in English conversion. [Figure 42] FIG. 10 is a diagram showing an example of user-defined data in English conversion. [Figure 43] FIG. 10 is a diagram showing an example of dictionary data in a handwriting recognition dictionary unit in English conversion. [Figure 44] FIG. 10 is a diagram showing an example of dictionary data in a character string conversion dictionary unit in English conversion. [Figure 45] FIG. 10 is a diagram showing an example of dictionary data in a predictive conversion dictionary section in English conversion. [Figure 46] FIG. 19 is a diagram showing data displayed in black and white highlighting and color display mode using the color definition data of FIG. 18 in English conversion. [Figure 47] 10A and 10B are diagrams illustrating an example of an operation guide for English conversion and selectable candidates displayed by the operation guide. [Figure 48] 10A and 10B are diagrams illustrating an example of specifying a selection object in the case of English conversion; [Figure 49] FIG. 49 shows an example of displaying operation command candidates based on operation command definition data when there is a handwritten object as shown in FIG. [Figure 50] 10 is a diagram illustrating an example of a method for inputting angle information of 90 degrees in English conversion. [Figure 51] 10A and 10B are diagrams illustrating a method for registering handwritten signature data in English conversion. [Figure 52] FIG. 10 is a diagram showing an example of an operation guide that is displayed when the user handwrites "Bob," which is handwritten signature data that has been registered, in English conversion. [Figure 53] FIG. 10 is a diagram illustrating a method for changing user-defined data in English conversion. [Figure 54] FIG. 10 is a diagram illustrating another configuration example of the display device. [Figure 55] FIG. 10 is a diagram illustrating another configuration example of the display device. [Figure 56] FIG. 10 is a diagram illustrating another configuration example of the display device. [Figure 57] FIG. 10 is a diagram illustrating another configuration example of the display device. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A display device and a display method performed by the display device will be described below as an example of an embodiment of the present invention with reference to the accompanying drawings. [Example]

[0011] <Terminology> An input means can be any means that allows handwriting by specifying coordinates on a touch panel. Examples include a pen, a human finger or hand, or a rod-shaped object. Eye-gaze input is also possible. Handwritten data is data that displays a sequence of coordinate points as a trajectory on a touch panel where a user moves the input means continuously. A stroke is a series of operations in which a user presses the input means against the touch panel, moves it continuously, and then releases it from the touch panel. Data handwritten using strokes is called stroke data. Handwritten data has one or more strokes. Handwritten input refers to handwritten data being input by a user.

[0012] <Comparative Example of a Display Device> In describing the display device of this embodiment, first, a simple description will be given of an example of color display in a monochrome compatible device.

[0013] Figure 1 shows an example of color to black and white conversion displayed by a black and white device. First, Figure 1(a) shows an example of handwritten data displayed by a black and white device. A black and white device is a display device that cannot display color. A user can select a color and input handwritten data on a black and white device, but the black and white device converts the color information to grayscale before displaying it.

[0014] Figures 1(b) and (c) show the correspondence between color and grayscale. Due to limitations in drawing, both are in black and white, but Figure 1(b) is a color chart, and Figure 1(c) is a grayscale color chart. Comparing the color and grayscale at the same position in the circumferential direction of the color charts in Figures 1(b) and 1(c), we can see that red (R) is converted to light gray, and blue (B) is converted to dark gray.

[0015] For this reason, as shown in Figure 1(a), when handwritten data 303 written by a user in red is converted to grayscale by a black-and-white compatible device, it is displayed as light gray. When handwritten data 302 written by a user in blue is converted to grayscale by a black-and-white compatible device, it is displayed as dark gray. Handwritten data 301 written by a user in black is displayed as black. However, it can be seen that red and blue also have difficulty fulfilling the function of making the original information stand out. Blue, in particular, is difficult to distinguish from black.

[0016] <Outline of Color Display of Display Device in This Embodiment> Therefore, the display device of this embodiment associates handwriting-input-derived data with information indicating that the data is handwriting-input-derived data, making it possible to detect that the data is handwriting-input-derived data. Furthermore, when the display device displays handwriting-input-derived data, the display device displays the handwritten data in a format corresponding to the color information based on the color information associated with the handwritten data. Displaying handwritten data in a format corresponding to the color information is called "black and white highlight display." Handwriting-input-derived data is stroke data entered by hand, or text data converted from this stroke data by handwriting recognition.

[0017] 2 is a diagram outlining a method for displaying handwriting-input-derived data using a display device according to this embodiment. The display device will be described as not being capable of color display, such as electronic paper. However, a display device capable of color display may also display in black and white.

[0018] (1) First, the user can select the color to input the handwritten data in using the buttons. In Figure 2, the user can select Bk, R, B, G, M, and C.

[0019] (2) When the user writes by hand, the display device displays the handwritten data in a manner that corresponds to the color. A manner that corresponds to the color means that different line types are displayed depending on the color. By displaying in line types that correspond to the color in this way, black and white highlighting becomes possible. As will be described later, the display device supports handwriting recognition and can convert handwritten data into text (character code). Handwritten data 1 and 2 in Figure 2 have been converted to text, and handwritten data 3 is handwritten data that has not been converted.

[0020] (3) The display device of this embodiment stores color information and information indicating whether each piece of handwritten input data is handwriting-input-derived data. Handwritten data 1 is handwriting-input-derived data and has color information of black, handwritten data 2 is handwriting-input-derived data and has color information of red, and handwritten data 3 is handwriting-input-derived data and has color information of blue. Since text (character codes) converted from handwritten data, such as handwritten data 1 and 2, also stores information indicating whether it is handwriting-input-derived data, the display device can display the handwriting-input-derived data in a line style corresponding to the specified color.

[0021] (4) When a display device converts handwritten data into a PDF file or the like, it sets the color information of the handwritten data or text data according to the PDF format and attaches the color information and a notice that the data is handwritten input-derived data as metadata. When the display device reads this PDF file from a storage medium (such as a USB memory), it can display the handwritten input-derived data in a line style corresponding to the color information based on the metadata. Any other device will ignore the metadata and display it as a normal PDF file, so it can be displayed in color. Metadata is data that describes the meaning of data. In other words, metadata is data that contains information about certain data. Because metadata is not the data itself but data about data, it is called meta (higher-level) data.

[0022] Furthermore, when the display device transmits handwriting input-derived data to an external device, the display device converts the handwriting input-derived data according to the type of device before transmitting the data. Examples of external devices include the following: ·Black and white compatible devices (e.g. monochrome printers) Color-compatible devices (e.g., color printers) Black and white emphasis compatible devices (e.g., the display device of the present application)

[0023] This conversion allows monochrome and color compatible devices to display the data in monochrome or color without determining whether the data is handwritten input or not.

[0024] In this way, the display device of this embodiment associates handwriting input-derived data with a symbol to enable it to detect that the data is handwriting input-derived data. As a result, even when the display device 2 acquires a file from an external device, when displaying the handwriting input-derived data, it can highlight the data in black and white based on the color information associated with the handwriting data. Furthermore, when handwriting input-derived data is transmitted to an external device, it can be displayed without first determining whether the data is handwriting input-derived data.

[0025] <Terminology> Black and white device: A black and white device is a device that can only output black or white, or grayscale. Examples include electronic paper and monochrome printers. It can also be said to be a device that does not support color. Electronic paper is also called E-paper. It is a general term for displays that are about 1 / 10 of a millimeter thick and can display and erase data electrically. The representative technology that realizes electronic paper is the microcapsule electrophoresis method, known as E-ink (registered trademark) technology. Each microcapsule contains white and black particles, with the white particles being positively charged and the black particles being negatively charged.

[0026] Color-compatible devices: Color-compatible devices are devices that can output not only black and white but also color. These include devices with LCD or OLED displays, and color printers. Color-compatible devices are also black and white compatible devices.

[0027] Black and white emphasis device: A black and white emphasis device is a device that can only output black or white, or grayscale, but displays black and white emphasis based on color information associated with handwritten input-derived data. An example is the display device of this embodiment (an example of a specific display device). Note that the display device is capable of handwritten input, and is sometimes called an input device or a handwriting input device.

[0028] Stroke data is a freehand drawn line, which has a set of continuous points and may be interpolated as appropriate.

[0029] Handwriting input-derived data refers to data entered by handwriting on a touch panel. It does not matter whether the data is left handwritten or converted into text data after input. Data acquired from an external device also remains handwriting input-derived data. Handwriting input-derived data may include text data converted through character recognition, as well as data converted based on user operations, such as stamps displayed as fixed characters like "Done" and "Confidential," shapes like circles and stars, and straight lines. In other words, handwriting input-derived data is stroke data entered by handwriting drawn on a display device, or text data converted from this stroke data through handwriting recognition.

[0030] Data that is not derived from handwriting input refers to data other than data entered by handwriting on a touch panel, such as image data, handwritten data entered on a device that does not support black-and-white enhancement, or text data. In other words, data that is not derived from handwriting input is stroke data entered on a device other than the display device, such as a personal computer, or text data converted from strokes by handwriting recognition.

[0031] ColorId is used to determine whether data is handwriting input-derived data or not. ColorId is an identifier used to distinguish between data that is handwriting input-derived data and data that is not handwriting input-derived data. This identifier can also be used to make the determination using a device identifier, model number (a number used to identify the product model), etc.

[0032] The term "color information is emphasized in black and white" refers to displaying data in a manner that allows the user to distinguish different colors, rather than simply converting colors into brightness or density. It is even better if this emphasis reflects the characteristics of the color, such as its attention-grabbing properties.

[0033] <Example of pen appearance> FIG. 3 shows an example of a perspective view of the pen 2500. FIG. 3 shows an example of a multi-function pen 2500. The pen 2500 has a built-in power source and can send commands to the display device 2, which is called an active pen (a pen without a built-in power source is called a passive pen). The pen 2500 in FIG. 3 has one physical switch at the tip, one at the end, and two on the side of the pen; the tip is for writing, the end is for erasing, and the side is for assigning user functions. In this embodiment, the pen also has non-volatile memory, which stores a pen ID that is unique to other pens.

[0034] Furthermore, if the pen has a switch, it is possible to reduce the number of steps the user has to take to operate the display device 2. Pens with switches mainly refer to active pens, but since passive pens that do not have a built-in power supply can generate power using only an LC circuit in the electromagnetic induction method, this includes not only active pens but also passive pens that use the electromagnetic induction method. Pens with switches that use optical, infrared, and capacitive methods other than electromagnetic induction are also active pens.

[0035] The hardware configuration of the pen 2500 is assumed to be similar to that of a general control system equipped with a communication function and a microcomputer. The pen 2500 may be of an electromagnetic induction type, an active electrostatic coupling type, or the like. The pen 2500 may also have functions such as pressure detection, tilt detection, and a hover function (displaying the cursor before the pen touches the screen).

[0036] <Overall configuration of the device> The overall configuration of the display device 2 according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing the overall configuration of the display device 2. Fig. 4(a) shows, as an example of the display device 2, a display device 2 used as a horizontally long electronic whiteboard hung on a wall.

[0037] As shown in Fig. 4(a), a display 220 as an example of a display device is installed on the upper part of the display device 2. A user U can handwrite (also referred to as input or drawing) characters and the like on the display 220 using a pen 2500.

[0038] FIG. 4(b) shows a display device 2 used as a vertically long electronic whiteboard hung on a wall.

[0039] 4(c) shows the display device 2 placed flat on a desk 230. The display device 2 is about 1 cm thick, so there is no need to adjust the height of the desk even when placed flat on a regular desk. In addition, the user can move it easily.

[0040] The display device 2 can automatically detect how it is placed using a tilt sensor.

[0041] <Device hardware configuration> Next, the hardware configuration of the display device 2 will be described with reference to Fig. 5. As shown in the figure, the display device 2 has the configuration of an information processing device or a computer. Fig. 5 is an example of a hardware configuration diagram of the display device 2. As shown in Fig. 5, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an SSD (Solid State Drive) 204.

[0042] Of these, the CPU 201 controls the overall operation of the display device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201. The SSD 204 stores various data, such as programs for the display device 2.

[0043] The display device 2 also includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, a power switch 227, a tilt sensor 217, a serial interface 218, a speaker 219, a microphone 221, a wireless communication device 222, an infrared I / F 223, a power control circuit 224, an AC adapter 225, and a battery 226.

[0044] The display controller 213 controls and manages the screen display in order to output an output image to the display 220, etc. The touch sensor 216 detects that the pen 2500, the user's hand, etc. (the pen and the user's hand are input means) has come into contact with the display 220. The touch sensor 216 also receives a pen ID.

[0045] The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 inputs and detects coordinates. For example, in an optical method, two light-emitting / receiving devices installed at both ends of the upper side of the display 220 emit multiple infrared rays parallel to the display 220. The infrared rays are reflected by a reflective member installed around the periphery of the display 220, and the light-emitting / receiving devices receive the returning light that travels along the same optical path as the light emitted by the light-receiving elements. The touch sensor 216 outputs position information of the infrared rays emitted by the two light-emitting / receiving devices that are blocked by an object to the touch sensor controller 215, and the touch sensor controller 215 identifies the coordinate position, which is the contact position of the object. The touch sensor controller 215 also has a communication unit 215a and can communicate wirelessly with the pen 2500. For example, if communication is based on a standard such as Bluetooth (registered trademark), a commercially available pen can be used. If one or more pens 2500 are registered in advance in the communication unit 215a, the user can communicate with the display device 2 without having to perform connection settings to allow the pen 2500 to communicate with the display device 2.

[0046] The power switch 227 is a switch for switching ON / OFF the power of the display device 2. The tilt sensor 217 is a sensor for detecting the tilt angle of the display device 2. It is mainly used to detect whether the display device 2 is being used in the installation state shown in FIG. 4(a), FIG. 4(b), or FIG. 4(c), and can automatically change the thickness of characters, etc. depending on the installation state.

[0047] The serial interface 218 is an external communication interface such as a USB or LAN interface. The serial interface 218 is used for inputting information from the outside. The speaker 219 is used for audio output, and the microphone 221 is used for audio input. The wireless communication device 222 communicates with a terminal carried by the user, and relays a connection to the Internet, for example. The wireless communication device 222 communicates using Wi-Fi, Bluetooth (registered trademark), or the like, but any communication standard is acceptable. The wireless communication device 222 forms an access point, and when the user sets the SSID (Service Set Identifier) ​​and password they have obtained in the terminal they carry, they can connect to the access point.

[0048] Preferably, the wireless communication device 222 is provided with two access points. a. Access point → Internet b. Access point → Internal network → Internet Access point (a) is for users outside the company, and users cannot access the company network through access point (a), but can use the Internet. Access point (b) is for users inside the company, and users can use the company network and the Internet through access point (b).

[0049] The infrared I / F 223 detects the display device 2 arranged adjacently. The infrared I / F 223 can detect only the display device 2 arranged adjacently by utilizing the linearity of infrared rays. It is preferable that one infrared I / F 223 is provided on each side, and it can detect in which direction another display device 2 is arranged relative to the display device 2. The adjacent display device 2 can display handwritten information that was previously handwritten (handwritten information on another page, with the area of ​​one display 220 being one page).

[0050] The power supply control circuit 224 controls an AC adapter 225 and a battery 226, which are power sources for the display device 2. The AC adapter 225 converts AC common to commercial power sources into DC.

[0051] If the display 220 is so-called electronic paper, it consumes little or no power to maintain an image after it has been drawn, and so can be driven by a battery 226. This makes it possible to use the display device 2 for purposes such as digital signage even in places where it is difficult to connect a power source, such as outdoors.

[0052] The display device 2 further includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201 shown in FIG.

[0053] The touch sensor 216 is not limited to an optical type, and may be a capacitive touch panel that identifies the contact position by detecting changes in capacitance. The touch sensor 216 may be a resistive film touch panel that identifies the contact position by detecting voltage changes across two opposing resistive films. The touch sensor 216 may be an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by an object touching the display unit. Various detection means may be used for the touch sensor 216. The touch sensor 216 may be of a type that does not require an electronic pen to detect whether or not the pen tip is touching. In this case, a fingertip or a pen-shaped stick can be used to perform a touch operation. The pen 2500 does not have to be a long, thin pen-shaped panel.

[0054] <About the device functions> Next, the functions of the display device 2 and the pen 2500 will be described with reference to FIG. 6. FIG. 6(a) is an example of a functional block diagram showing the functions of the display device 2 in blocks. The display device 2 includes a handwriting input unit 21, a display unit 22, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a character string conversion control unit 28, a character string conversion dictionary unit 29, a predictive conversion control unit 30, a predictive conversion dictionary unit 31, an operation command recognition control unit 32, an operation command definition unit 33, a pen ID control data storage unit 36, a file transmission / reception control unit 37, a handwritten signature authentication control unit 38, and a handwritten signature data storage unit 39. Each function of the display device 2 is realized by any of the components shown in FIG. 5 operating in response to an instruction from the CPU 201 in accordance with a program loaded from the SSD 204 onto the RAM 203.

[0055] The handwriting input unit 21 is realized by a touch sensor 216 or the like, and accepts handwriting input by the user and receives a pen ID. The handwriting input unit 21 converts the user's pen input d1 into pen operation data d2 (pen up, pen down, or pen coordinate data) with the pen ID, and transmits it to the handwriting input display control unit 23. The pen coordinate data is periodically transmitted as discrete values, and coordinates between the discrete values ​​are interpolated.

[0056] The display unit 22 is realized by a display 220 or the like, and displays handwritten objects and operation menus. The display unit 22 converts the drawing data d3 written to the video memory by the handwriting input display control unit 23 into data according to the characteristics of the display 220, and transmits the data to the display 220.

[0057] The handwriting input display control unit 23 performs overall control over handwriting input and display. The handwriting input display control unit 23 processes the pen operation data d2 from the handwriting input unit 21 and transmits it to the display unit 22 for display. Details of the processing of the pen operation data d2 and the display of strokes will be described later with reference to Figs. 30 to 37.

[0058] The candidate display timer control unit 24 is a timer for controlling the display of selectable candidates. The candidate display timer control unit 24 starts or stops a timer to generate the timing for starting the display of selectable candidates and the timing for erasing the display. The selectable candidates are handwriting recognition character string / language character string candidates, conversion character string candidates, character string / predictive conversion candidates, and operation command candidates that are selectably displayed in an operation guide described below. The candidate display timer control unit 24 receives a timer start request d4 (which may also be a timer stop request) from the handwriting input display control unit 23, and transmits a timeout event d5 to the handwriting input display control unit 23.

[0059] The handwriting input saving unit 25 has a storage function for saving user data (handwritten objects / character string objects). The handwriting input saving unit 25 receives user data d6-1 from the handwriting input display control unit 23 and saves it in the handwriting input saving unit 25. The handwriting input saving unit 25 receives an acquisition request d6-2 from the handwriting input display control unit 23 and transmits the user data d7 saved in the handwriting input saving unit 25. The handwriting input saving unit 25 transmits position information d36 of the confirmed object to the operation command recognition control unit 32.

[0060] The handwriting recognition control unit 26 is a recognition engine that performs online handwriting recognition. Unlike a typical OCR (Optical Character Reader), it recognizes characters (not only Japanese but also multiple languages ​​such as English), numbers, symbols (%, $, &, etc.), shapes (lines, circles, triangles, etc.) in parallel with the user's pen operation. Various algorithms have been devised for recognition methods, but this embodiment uses known techniques and will not be described in detail.

[0061] The handwriting recognition control unit 26 receives the pen operation data d8-1 from the handwriting input display control unit 23, performs handwriting recognition, and stores handwriting-recognized character string candidates. The handwriting recognition control unit 26 also stores language character string candidates converted from the handwriting-recognized character string candidates d12 using the handwriting recognition dictionary unit 27. Separately, when an acquisition request d8-2 is received from the handwriting input display control unit 23, the handwriting recognition control unit 26 transmits the stored handwriting-recognized character string candidates and language character string candidates d9 to the handwriting input display control unit 23.

[0062] The handwriting recognition dictionary unit 27 stores dictionary data for language conversion in handwriting recognition. The handwriting recognition dictionary unit 27 receives handwriting recognition character string candidates d12 from the handwriting recognition control unit 26, converts them into linguistically probable language character string candidates d13, and transmits them to the handwriting recognition control unit 26. For example, in the case of Japanese, hiragana characters are converted into kanji characters or katakana characters.

[0063] The character string conversion control unit 28 controls the conversion of conversion character string candidates into character strings. A conversion character string is a character string that is likely to be generated by including a handwriting recognition character string or a language character string. The character string conversion control unit 28 receives the handwriting recognition character string and language character string candidates d11 from the handwriting recognition control unit 26, converts them into conversion character string candidates using the character string conversion dictionary unit 29, and stores them. Separately, when an acquisition request d14 is received from the handwriting input display control unit 23, the character string conversion control unit 28 transmits the stored conversion character string candidates d15 to the handwriting input display control unit 23.

[0064] The character string conversion dictionary unit 29 stores dictionary data for character string conversion. The character string conversion dictionary unit 29 receives handwriting-recognized character strings and language character string candidates d17 from the character string conversion control unit 28, and transmits converted character string candidates d18 to the character string conversion control unit 28.

[0065] The predictive conversion control unit 30 receives the handwriting-recognized character string and the language character string candidate d10 from the handwriting recognition control unit 26, and receives the converted character string candidate d16 from the character string conversion control unit 28. The predictive conversion control unit 30 converts each of the handwriting-recognized character string, the language character string candidate d10, and the converted character string candidate d16 into a predicted character string candidate using the predictive conversion dictionary unit 31. A predicted converted character string is a character string that is likely to be generated by including the handwriting-recognized character string, the language character string, or the converted character string. If an acquisition request d19 is separately received from the handwriting input display control unit 23, the predictive conversion control unit 30 transmits the predicted character string candidate d20 to the handwriting input display control unit 23.

[0066] The predictive conversion dictionary unit 31 stores dictionary data for predictive conversion. The predictive conversion dictionary unit 31 receives handwritten recognition character strings, language character string candidates, and conversion character string candidates d21 from the predictive conversion control unit 30, and transmits predicted character string candidates d22 to the predictive conversion control unit 30.

[0067] The operation command recognition control unit 32 receives the handwriting-recognized character string and the language string candidate d30 from the handwriting recognition control unit 26, and receives the conversion character string candidate d28 from the character string conversion control unit 28. The operation command recognition control unit 32 receives the predicted character string candidate d29 from the predictive conversion control unit 30. Then, the operation command recognition control unit 32 transmits an operation command conversion request d26 for each of the handwriting-recognized character string, the language string candidate d30, the conversion character string candidate d28, and the predicted character string candidate d29 to the operation command definition unit 33, and receives the operation command candidate d27 from the operation command definition unit 33. The operation command recognition control unit 32 stores the operation command candidate d27.

[0068] If the operation command conversion request d26 partially matches the operation command definition, the operation command definition unit 33 sends operation command candidates d27 to the operation command recognition control unit 32.

[0069] Furthermore, the operation command recognition control unit 32 receives pen operation data d24-1 from the handwriting input display control unit 23. The operation command recognition control unit 32 transmits a request d23 to acquire position information of a confirmed object that was previously input and confirmed to the handwriting input storage unit 25. The operation command recognition control unit 32 stores the confirmed object designated by the pen operation data as a selected object (including position information). The operation command recognition control unit 32 identifies a selected object that satisfies the position of the pen operation data d24-1 and a predetermined criterion. Separately, when an acquisition request d24-2 is received from the handwriting input display control unit 23, the operation command recognition control unit 32 transmits a selected object d25 identified as a candidate for the stored operation command to the handwriting input display control unit 23.

[0070] The pen ID control data storage unit 36 ​​holds pen ID control data (which may be called storage means). Before the handwriting input display control unit 23 transmits display data to the display unit 22, the pen ID control data storage unit 36 ​​transmits pen ID control data d41 to the handwriting input display control unit 23. The handwriting input display control unit 23 draws the display data under the operating conditions stored in association with the pen ID. Furthermore, before the handwriting recognition control unit 26 performs handwriting recognition, the pen ID control data storage unit 36 ​​transmits angle information d44 of the pen ID control data to the handwriting recognition control unit 26, and the handwriting recognition control unit 26 rotates the stroke using the angle information stored in association with the pen ID and performs handwriting recognition.

[0071] Furthermore, after the handwriting recognition control unit 26 recognizes a straight line for setting angle information when the user writes characters or the like by hand, the handwriting recognition control unit 26 transmits angle information d43 of the pen ID control data to the pen ID control data storage unit 36 ​​and stores the angle information d43 in association with the pen ID. Furthermore, after the handwriting input display control unit 23 executes an operation command for setting angle information, the handwriting input display control unit 23 transmits pen ID control data d42 to the pen ID control data storage unit 36 ​​and stores the execution result of the operation command (angle information set by the user) in association with the pen ID. Thereafter, the stroke of that pen ID is rotated by the set angle information before handwriting recognition is performed.

[0072] Furthermore, the handwriting recognition control unit 26 transmits stroke data d49 rotated clockwise using the angle information of the pen ID control data to the handwritten signature authentication control unit 38. This makes it possible to authenticate a handwritten signature regardless of the user's position (the direction from which the handwriting is made relative to the display device 2).

[0073] The handwritten signature data storage unit 39 holds handwritten signature data. When the handwritten signature data storage unit 39 receives a handwritten signature data acquisition request d45 from the handwritten signature authentication control unit 38, it transmits handwritten signature data d46 to the handwritten signature authentication control unit 38. The format of the handwritten signature data depends on the handwritten signature authentication algorithm of the handwritten signature authentication control unit 38. The data in the handwritten signature data storage unit 39 will be described with reference to FIG. 15.

[0074] When the handwritten signature authentication control unit 38 receives clockwise rotated stroke data d49 from the handwritten signature recognition control unit 26, it sends a handwritten signature data acquisition request d45 to the handwritten signature data storage unit 39, and the handwritten signature data storage unit 39 sends handwritten signature data d46 to the handwritten signature authentication control unit 38.

[0075] The handwritten signature authentication control unit 38 authenticates the user based on the handwritten signature data. While various algorithms have been devised for authenticating users based on handwritten signature data, this embodiment uses a technology that can achieve a practically acceptable recognition rate. For example, a feature vector is generated using elements such as coordinates, writing pressure, and stroke writing time that make up the handwritten signature data, and the elements are weighted appropriately. The feature vector of the registered handwritten signature data is compared with a feature vector of the user name handwritten by the user when signing in. If the degree of match is equal to or greater than a threshold, authentication is deemed successful; if it is less than the threshold, authentication is deemed unsuccessful.

[0076] The handwritten signature authentication control unit 38 stores the handwritten signature authentication result, which is the comparison result between the stroke data d49 and the handwritten signature data d46, and when an acquisition request d48 is separately received from the handwritten input display control unit 23, the handwritten signature authentication control unit 38 transmits the stored handwritten signature authentication result d47 to the handwritten input display control unit 23. The handwritten signature authentication result indicates whether or not the stroke data d49 and the handwritten signature data d46 can be considered to match, and if they do match, includes a SignatureId (described later) that is associated with the matching handwritten signature data d46.

[0077] If the handwriting recognition result of handwriting recognition control unit 26 matches the operation command instructing the execution of handwritten signature registration, handwriting recognition control unit 26 acquires data d52 input into a handwritten signature registration form (a box into which handwritten signature data is input, as will be described later) from handwriting input storage unit 25, and transmits handwritten signature data d50 of the data d52 to handwritten signature authentication control unit 38. Handwritten signature authentication control unit 38 transmits the received handwritten signature data d50 to handwritten signature data storage unit 39 for registration.

[0078] If the handwriting recognition result of the handwriting recognition control unit 26 is an instruction to cancel the handwritten signature, the handwriting recognition control unit 26 sends a request d51 to delete the handwritten signature registration form to the handwriting input storage unit 25 and deletes the handwritten signature registration form from the handwriting input storage unit 25.

[0079] If the handwriting recognition result of handwriting recognition control unit 26 is an instruction to change the user-defined data, handwriting recognition control unit 26 acquires data d53 entered in the user-defined data change form from handwriting input storage unit 25. Handwriting recognition control unit 26 sends change value d54 of data d53 to operation command definition unit 33 to change the user-defined data. User-defined data will be described with reference to FIG. 14.

[0080] If the handwriting recognition result of the handwriting recognition control unit 26 is an instruction to cancel the user-defined data change form, the handwriting recognition control unit 26 sends a request d55 to the handwriting input storage unit 25 to delete the user-defined data change form, and deletes the user-defined data change form from the handwriting input storage unit 25.

[0081] The file transmission / reception control unit 37 saves and acquires handwriting input-derived data or data not originating from handwriting input to a storage medium, communicates with external devices (print requests, display requests, etc.), etc. The file transmission / reception control unit 37 receives a file transmission / reception execution request d64 from the handwriting input display control unit 23. When saving or printing a file, the handwriting input display control unit 23 sends a file transmission request to the file transmission / reception control unit 37, and the file transmission / reception control unit 37 sends a request to acquire handwriting input save data d61 to the handwriting input save unit 25. In the case of data that is not handwritten input-derived data, the handwritten input storage unit 25 transmits the color information held in the data as is. In the case of handwritten input-derived data, if the destination is a color-compatible device (for example, a color printer), the handwritten input saving unit 25 converts the data into color and sends the converted handwritten input saved data d62 to the file transmission / reception control unit 37. In the case of data resulting from handwriting input, if the destination is a black-and-white compatible device (e.g., a monochrome printer), the handwriting input saving unit 25 converts the data to black-and-white and sends the handwriting input saved data d62 to the file transmission / reception control unit 37. Since the black-and-white compatible device may be capable of converting the data to grayscale, the data may be converted to color before being sent. If the destination is a device that supports black and white emphasis, the handwritten input saved data, which will be described later, is sent to the file transmission / reception control unit 37. Furthermore, when the handwriting input saving unit 25 saves handwriting input-derived data in a file, it converts the handwriting input saved data d62 into color in accordance with the file format and attaches the handwriting input saved data as metadata for the file. When data that is not handwriting input-derived data is saved in a file, it converts the handwriting input saved data d62 into color in accordance with the file format.

[0082] Whether a device is color-compatible or monochrome-compatible is stored, for example, in an MIB (Management Information Base) held by the network device, and the file transmission / reception control unit 37 can determine this by acquiring the MIB. Similarly, whether a device is monochrome-emphasis compatible can also be determined from the model name and the like published by the MIB.

[0083] When a file is read, the handwriting input display control unit 23 sends a file list acquisition request d65 to the file transmission / reception control unit 37, which then sends a file list acquisition request to the external device, acquires a file list d63, and sends it to the handwriting input display control unit 23. The handwriting input display control unit 23 displays the file list on the display unit 22, the handwriting input unit 21 sends the display position of the selected file to the handwriting input display control unit 23, and the handwriting input display control unit 23 sends a file reception request d66 to the file transmission / reception control unit 37. The file transmission / reception control unit 37 acquires a file from the external device and sends this file d67 to the handwriting input saving unit 25. The handwriting input saving unit 25 analyzes the metadata of the file to determine whether it is handwriting input-derived data. If it is handwriting input-derived data, it extracts handwriting input saved data (black and white highlight / color convertible data, described below). The handwriting input saving unit 25 converts the handwriting input-derived data to black and white highlight display, and if it is not handwriting input-derived data, it sends the data to the handwriting input display control unit 23 without conversion (displayed in grayscale). The handwriting input display control unit 23 transmits display data of the handwritten object to the display unit 22 .

[0084] 6(b) is a functional block diagram showing the functions of the pen 2500. The pen 2500 has a pen event transmission unit 41. The pen event transmission unit 41 attaches a pen ID to event data of pen up, pen down, and pen coordinates and transmits them to the display device 2.

[0085] <About predefined control data> Next, the predefined control data used by the display device 2 for various processes will be described with reference to Fig. 7. Fig. 7 shows an example of the predefined control data. In the example of Fig. 7, control data is shown for each control item.

[0086] The selectable candidate display timer 401 defines the time until the selectable candidates are displayed. This is because the selectable candidates are not displayed while handwriting is in progress. In FIG. 7, this means that the selectable candidates are displayed if the pen does not come down within TimerValue=500 [ms] after the pen is up. The selectable candidate display timer 401 is held by the candidate display timer control unit 24. The selectable candidate display timer 401 is used when the selectable candidate display timer is started in step S18-2 of FIG. 32, which will be described later.

[0087] The selectable candidate deletion timer 402 defines the time until the displayed selectable candidates are deleted. This is to delete the selectable candidates if the user does not select a selectable candidate. In FIG. 7, this means that if a selectable candidate is not selected within TimerValue=5000 [ms] from the display of the selectable candidates, the selectable candidate display data is deleted. The selectable candidate deletion timer 402 is held by the candidate display timer control unit 24. The selectable candidate deletion timer 402 is used when the selectable candidate display deletion timer starts in step S64 of FIG. 34.

[0088] The handwritten object neighborhood rectangular area 403 defines a rectangular area considered to be near the handwritten object. In the example of FIG. 7, the handwritten object neighborhood rectangular area 403 is a rectangular area obtained by enlarging the handwritten object rectangular area by 50% (Horizontal) of the estimated character size in the horizontal direction and by 80% (Vertical) of the estimated character size in the vertical direction. In the example of FIG. 7, the rectangular area is a percentage of the estimated character size (specified as a %), but it can also be a fixed length by using units such as "mm". The handwritten object neighborhood rectangular area 403 is stored in the handwritten input storage unit 25. The estimated character size 405 is used to determine the overlap state between the handwritten object neighborhood rectangular area and the stroke rectangular area in step S10 of FIG. 31.

[0089] The estimated writing direction / character size determination condition 404 defines constants for determining the writing direction and the measurement direction of the character size. In the example of FIG. 7, if the difference between the time when the first stroke was added to the rectangular area of ​​the handwritten object and the time when the last stroke was added is MinTime=1000 [ms] or more, the difference between the horizontal distance (width) and the vertical distance (height) of the rectangular area of ​​the handwritten object is MinDiff=10 [mm] or more, and the horizontal distance is longer than the vertical distance, the estimated writing direction is determined to be "horizontal writing" and the estimated character size is determined to be the vertical distance. If the horizontal distance is shorter than the vertical distance, the estimated writing direction is determined to be "vertical writing" and the estimated character size is determined to be the horizontal distance. If these conditions are not met, the estimated writing direction is determined to be "horizontal writing" (DefaultDir="Horizontal") and the estimated character size is determined to be the longer of the horizontal distance and the vertical distance. The estimated writing direction / character size determination condition 404 is held in the handwriting input storage unit 25. The estimated writing direction / character size judgment condition 404 is used to obtain the estimated writing direction in step S59 in FIG. 34 and to obtain the character string object font in step S81 in FIG.

[0090] The estimated character size 405 defines data for estimating the size of characters, etc. In the example of FIG. 7, this means that the estimated character size determined by the estimated writing direction / character size determination condition 404 is compared with the small character 405a (hereinafter referred to as the minimum font size) and the large character 405c (hereinafter referred to as the maximum font size) of the estimated character size 405. If the estimated character size is smaller than the minimum font size, the estimated character size is determined to be the minimum font size. If the estimated character size is larger than the maximum font size, the estimated character size is determined to be the maximum font size. Otherwise, the estimated character size is determined to be the character size of the medium character 405b. The estimated character size 405 is stored in the handwriting input storage unit 25. The estimated character size 405 is used to obtain the character string object font in step S81 of FIG. 37.

[0091] Specifically, the handwriting input storage unit 25 compares the estimated character size determined by the estimated writing direction / character size determination condition 404 with the FontSize of the estimated character size 405, and uses the font with the closest size. For example, if the estimated character size is 25 mm (FontSize for small characters) or less, it is determined to be a "small character," if the estimated character size is more than 25 mm but less than 50 mm (FontSize for medium characters), it is determined to be a "medium character," and if the estimated character size is more than 100 mm (FontSize for large characters), it is determined to be a "large character." "Small characters" 405a use a 25 mm Mincho font (FontStyle="Mincho" FontSize="25mm"), "medium characters" 405b use a 50 mm Mincho font (FontStyle="Mincho" FontSize="50mm"), and "large characters" 405c use a 100 mm Gothic font (FontStyle="Gothic" FontSize="100mm"). If you want more font sizes or styles, you can increase the number of estimated font sizes 405.

[0092] The crossing line determination condition 406 defines data used to determine whether multiple objects have been selected. If the handwritten object is a single stroke, and in the example of FIG. 7, there is an object whose long side length is 100 mm or more (MinLenLongSide="100mm"), whose short side length is 50 mm or less (MaxLenShortSide="50mm"), and whose overlap rate with the handwritten object in the long side direction and short side direction is 80% or more (MinOverLapRate="80%"), then it is determined that multiple objects have been selected (selected objects). The crossing line determination condition 406 is held by the operation command recognition control unit 32. The crossing line determination condition 406 is used in the crossing line determination for determining the selected object in step S50 of FIG. 33.

[0093] The encircling line determination condition 407 defines data used to determine whether an object is an encircling line. In the example of Fig. 7, the operation command recognition control unit 32 determines a fixed object, in which the overlap rate of the long side direction and the short side direction of the handwritten object is 100% or more (MinOverLapRate="100%"), as a selected object. The encircling line determination condition 407 is held by the operation command recognition control unit 32. The encircling line determination condition 407 is used in the encircling line determination for determining the selected object in step S50 of Fig. 33.

[0094] Note that either the cross-line determination condition 406 or the enclosing line determination condition 407 may be given priority in the determination. For example, if the cross-line determination condition 406 is set to a relaxed value (to make it easier to select a cross-line) and the enclosing line determination condition 407 is set to a strict value (to make it possible to select only an enclosing line), the operation command recognition control unit 32 may give priority to the enclosing line determination condition 407 in the determination.

[0095] <Example of dictionary data> The dictionary data will be described with reference to Figs. 8 to 10. Fig. 8 is an example of dictionary data in the handwriting recognition dictionary unit 27. Fig. 9 is an example of dictionary data in the character string conversion dictionary unit 29. Fig. 10 is an example of dictionary data in the predictive conversion dictionary unit 31. These dictionary data are used in steps S33 to S41 in Fig. 33, respectively.

[0096] In this embodiment, the conversion results using the dictionary data of the handwriting recognition dictionary unit 27 in FIG. 8 are called language string candidates, the conversion results using the dictionary data of the string conversion dictionary unit 29 in FIG. 9 are called converted string candidates, and the conversion results using the dictionary data of the predictive conversion dictionary unit 31 in FIG. 10 are called predicted string candidates. For each dictionary data, "before conversion" represents the string to be searched for in the dictionary data, "after conversion" represents the converted string corresponding to the string to be searched for, and "probability" represents the probability of selection by the user. The probability is calculated from the results of users selecting each string in the past. Therefore, the probability may be calculated for each user. Various algorithms have been devised for calculating probability, but the calculation may be performed using an appropriate method as appropriate, and details will be omitted. This embodiment is characterized by displaying string candidates in descending order of selection probability based on the estimated writing direction.

[0097] The dictionary data of the handwriting recognition dictionary unit 27 in Figure 8 shows that handwritten "gi" is converted to "gi" with a probability of 0.55 and to "gi" with a probability of 0.4, and handwritten "gishi" is converted to "gishi" with a probability of 0.5 and to "gishi" with a probability of 0.45. The same applies to other "before conversion" character strings. In Figure 8, the "before conversion" character string is handwritten hiragana, but characters other than hiragana may also be registered as "before conversion."

[0098] 9 indicates that the character string "" is converted to "" with a probability of 0.95, and the character string "" is converted to "" with a probability of 0.85. The same applies to other "before conversion" character strings.

[0099] The dictionary data of the predictive conversion dictionary unit 31 in Fig. 10 indicates that the character string "minutes" is converted to "addressee of the minutes" with a probability of 0.65, and the character string "skill test" is converted to "approval of skill test" with a probability of 0.75. In the example of Fig. 10, all character strings before conversion are kanji, but characters other than kanji may also be registered.

[0100] The dictionary data is not language-dependent, and any character strings may be registered before and after conversion.

[0101] <Operation command definition data held by the operation command definition section> Next, the operation command definition data used by the operation command recognition control unit 32 will be described with reference to Fig. 11. Fig. 11 shows an example of the operation command definition data and system definition data held by the operation command definition unit 33.

[0102] Fig. 11(a) shows an example of operation command definition data. The operation command definition data in Fig. 11(a) is an example of operation command definition data when no selected object is selected by a handwritten object, and applies to all operation commands that operate the display device 2. The operation command in Fig. 11(a) has an operation command name (Name), a string (String) that partially matches a string candidate, and an operation command string (Command) to be executed. "%~%" in the operation command string is a variable, and is associated with system definition data as shown in Fig. 11(b). In other words, "%~%" is replaced with the system definition data shown in Fig. 11(b).

[0103] First, the operation command definition data 701 indicates that the operation command name is "Read minutes template," the string that partially matches the string candidate is "minutes" or "template," and the operation command string to be executed is "ReadFile https: / / %username%:%password%@server.com / template / minutes.pdf." In this example, the operation command string to be executed contains the system definition data "%~%," and "%username%" and "%password%" are replaced by the system definition data 704 and 705, respectively. Therefore, the operation command string to be ultimately executed is the string "ReadFile https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf," which indicates that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf" is to be read (ReadFile).

[0104] The operation command definition data 702 indicates that the operation command name is "Save to minutes folder," the strings that partially match the string candidate are "minutes" or "save," and the operation command string to be executed is "WriteFile https: / / %username%:%password%@server.com / minutes / %machinename%_%yyyy-mm-dd%.pdf." As with the operation command definition data 701, "%username%," "%password%," and "%machinename%" in the operation command string are replaced with system definition data 704 to 706, respectively. Note that "%yyyy-mm-dd%" indicates that it should be replaced with the current date. For example, if the current date is September 26, 2018, it should be replaced with "2018-09-26." The final operation command to be executed is "WriteFile https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf", which indicates that the minutes should be saved (WriteFile) in the file "https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf".

[0105] The operation command definition data 703 indicates that the operation command name is "Print", the string that partially matches the string candidate is "Print" or "Print", and the operation command string to be executed is "PrintFile https: / / %username%:%password%@server.com / print / %machinename%-"%yyyy-mm-dd%.pdf". When the operation command string is replaced in the same way as in the operation command definition data 702, the operation command to be executed finally becomes "PrintFile https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf", which indicates that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf" will be printed (PrintFile). In other words, the file is sent to the server. When the user makes the printer communicate with the server and specifies a file, the printer prints the contents of the file on paper.

[0106] In this way, the operation command definition data 701 to 703 can be identified from the character string candidates, and the operation command can be displayed by the user's handwriting. Also, if the user authentication is successful, "%username%", "%password%", etc. in the operation command definition data are replaced with the user information, making it possible to input and output files associated with the user.

[0107] If user authentication is not performed (including the case where authentication fails but the user can still use the display device 2), the display device 2 replaces the username and password with the pre-set password for the display device 2. Therefore, file input and output can be performed in association with the display device 2 without user authentication.

[0108] The operation command definition data 709, 710, 711, 720, 721, and 722 are operation commands for changing the pen color. The pen color is the color of handwritten data input with the pen being used by the user. The operation command names of the operation command definition data 709, 710, 711, 720, 721, and 722 are "black pen," "red pen," "blue pen," "green pen," "magenta pen," and "cyan pen," respectively. For example, in the case of "black pen," the character strings that partially match the character string candidate are "kuro" (black) and "pen" (pen), and if the user writes "kuro," only "black pen" is displayed as an operation command candidate. On the other hand, "pen" also corresponds to a character string (String) that partially matches the character string candidate, such as "red pen," so if the user writes "pen," "black pen" to "cyan pen" are displayed as operation command candidates. When these operation commands are executed, the control data associated with the pen ID of the pen 2500 used by the user for the operation is updated, and the color of the pen with this pen ID (ColorId of the pen ID control data) is changed, as shown in FIG. 17.

[0109] In the operation command definition data 719, the operation command name is "Read File," the character strings (String) that partially match the character string candidates are "File," "Read," and "Read," and the operation command to be executed is "ReadFile https: / / %username%:%password%@server.com / files / ." When substituted with the system definition data in the same way as the operation command definition data 702, the operation command becomes "ReadFile https: / / taro.tokkyo:x2PDHTyS@server.com / files / ," which indicates that the file at this address (folder) is to be read (ReadFile). When this operation command is executed, the display device 2 displays a file selection window such as that shown in FIG. 12. The file selection window in FIG. 12 displays thumbnails of the first page of the files stored at this address (folder).

[0110] FIG. 12 is an example of file list display data displayed on the display unit 22 in step S94 (file list display data) in FIG. 37. FIG. 12(a) is created from the file list information acquired by the file list information acquisition in step S91 in FIG. 37. The file list information has a list of file addresses, and the display unit 22 extracts file names and thumbnails from each file address and displays them in ascending order of file name. In FIG. 12, the display unit 22 displays four thumbnails and file names, and displays the previous and next thumbnails and file names using left and right arrow icons 99. When a displayed thumbnail or file name is pressed with a pen, the display unit 22 clears the screen shown in FIG. 12, and the file transmission / reception control unit 37 receives the file from the file address pressed with the pen.

[0111] When the close button 98 in the upper right corner is pressed with a pen, the display unit 22 erases the screen shown in FIG. 12, and execution of the operation command definition data 719 is canceled. When the PDF file shown in FIGS. 12(a) and (d) is selected, the file transmission / reception control unit 37 receives the PDF file, saves it in the handwriting input saving unit 25, and analyzes it. The file transmission / reception control unit 37 saves the text data (with metadata) that is handwriting input-derived data as handwriting input saved data (black and white emphasis and color convertible data) such as handwriting input saved data 801 to 805 in FIG. 16. The black and white emphasis and color convertible data is data that can be both black and white emphasis and converted to color (mainly handwriting input-derived data). Text data that is not handwriting input-derived data is converted to handwriting input saved data such as handwriting input saved data 806 in FIG. 16, saved, and displayed on the display 220.

[0112] When the JPG file of Figure 12(b)(c) is selected, the file transmission / reception control unit 37 receives the image file, stores it in the handwriting input storage unit 25 as handwriting input storage data such as handwriting input storage data 807 of Figure 16, and displays it on the display 220.

[0113] Returning to FIG. 11, the explanation will be given. The operation command definition data 712 is an operation command for aligning the direction of text data in a certain direction. The operation command name of the operation command definition data 712 is "align text direction", the character string that partially matches the character string candidate is "text", "orientation", or "direction", and the operation command character string is "AlignTextDirection". Text data written by the user in a direction other than upside-down is in various directions, making it difficult to read the entire data from a single direction. When the user executes the operation command definition data 712, the display device 2 aligns the handwritten and recognized character string in the same direction (for example, upside-down). "Aligning" in this case means rotating the text data by the angle information.

[0114] The operation command definition data 713 indicates that the operation command name is "Register handwritten signature", the character strings that partially match the character string candidates are "Sign" and "Register", and the operation command character string is "RegistSignature". When the RegistSignature command is executed, a handwritten signature registration form is added to the handwritten input storage unit 25, and the handwritten signature registration form is displayed on the operation screen 101. An example of the handwritten signature registration form will be described later (see FIG. 27).

[0115] The operation command definition data 714 indicates that the operation command name is "Sign in with handwriting," the character string that partially matches the character string candidate is "%signature%," and the operation command is "Signin." Here, "%signature%" is a reserved word in the system definition data, and indicates that registered handwritten signature data matches stroke data such as a user name. In other words, if there is a match, the operation command 512 based on the operation command definition data 714 is displayed in the operation guide 500 (see FIG. 28).

[0116] When the Signin command is executed, the AccountId of the user who has the SignatureId of the matching handwritten signature data is saved in the pen ID control data of the pen 2500 that handwritten the stroke data such as the user name. This associates the pen ID with the AccountId, and the display device 2 can use the user-defined data identified by this AccountId (see FIG. 17(a)).

[0117] The operation command definition data 715 indicates that the operation command name is "Sign out by handwriting", the character strings that partially match the character string candidates are "Sign" or "Out", and the operation command is "Signout". When the Signout command is executed, the AccountId is deleted from the pen ID control data of the pen 2500 that operated the handwriting signout. This removes the association between the pen ID and the AccountId, allowing any user to use the pen 2500.

[0118] Operation command definition data 716 indicates that the operation command name is "Change settings," the character strings that partially match the character string candidates are "Settings" or "Change," and the operation command is "ConfigSettings." When the ConfigSettings command is executed, a user-defined data change form is added to handwritten input storage unit 25, and the user-defined data change form is displayed on operation screen 101. The user-defined data change form will be described later (see FIG. 29).

[0119] Next, we will explain the operation command definition data when there is a selected object, that is, the operation command definition data for editing and modification. Fig. 13 shows an example of the operation command definition data when there is a selected object selected by a handwritten object. The operation command definition data in Fig. 13 has the operation command name (Name), the group name (Group) of operation command candidates, and the operation command string to be executed (Command).

[0120] The operation command definition data 707 defines operation commands for editing (Group="Edit"), and is an example of definition data for the names of operation commands for editing: "erase," "move," "rotate," and "select." In other words, these operation commands are displayed for the selected object, allowing the user to select the desired operation command.

[0121] The operation command definition data 708 defines operation commands for the decoration system (Group="Decorate"), and is an example of definition data for the names of operation commands for the decoration system: "bold", "thin", "larger", "smaller", and "underline". These operation commands are displayed for the selected object, allowing the user to select the desired operation command. In addition, color operation commands may also be displayed.

[0122] Therefore, when the user selects a selection object with a handwritten object, the operation command definition data 707 and 708 are specified, and the user can display the operation command by handwriting.

[0123] <User-defined data> Next, user-defined data will be described with reference to Fig. 14. Fig. 14 shows an example of user-defined data held by operation command definition unit 33. The user-defined data in Fig. 14 is an example of definition data for one user. In user-defined data 717, AccountId is user identification information that is automatically assigned a number for each user, AccountUsername and AccountPassword are a user name and password, SignatureId is handwritten signature data identification information that is automatically assigned a number when handwritten signature data is registered, and username, password, and machinename are character strings that are set in operation command definition data 701 to 703 instead of system definition data 704 to 706, respectively. This makes it possible to execute operation commands using user-defined data.

[0124] When a user signs in by handwriting a user name or the like, the handwriting input display control unit 23 utilizes the fact that the pen ID and Account Id are associated with the pen ID control data (see FIG. 17(a)). The handwriting input display control unit 23 uses the character string of the user-defined data having the Account Id associated with the pen ID of the pen 2500 used by the user when executing an operation command. After the user signs out, the character string of the system-defined data is used when executing an operation command, even if the pen 2500 used for signing in is used.

[0125] User-defined data 718 is data used in the user-defined data change form. Name is the field name of AccountUsername, AccountPassword, username, password, or machinename in user-defined data 717, and Data is the changed value of AccountUsername, AccountPassword, username, password, or machinename. In this example, the data for "Name" is "%AccountName%". The data for "Password" is "%AccountPassword%". The data for "Folder User Name" is "%username%". The data for "Folder Password" is "%password". The data for "Folder File Name" is "%machinename". Each field corresponds to each field in user-defined data 717. These fields entered in the user-defined data change form are reflected in user-defined data 717.

[0126] It is also possible for the user to register color definition data, which will be described later, in user-defined data 717, and the user can input color definition data that he or she has defined himself or herself.

[0127] <Handwritten signature data> Next, the handwritten signature data will be described with reference to Fig. 15. Fig. 15 shows an example of handwritten signature data stored in the handwritten signature data storage unit 39. The handwritten signature data has Data representing the handwritten signature associated with SignatureId. SignatureId is identification information that is automatically assigned when the handwritten signature data is registered, and Data is data calculated by the handwritten signature authentication algorithm of the handwritten signature authentication control unit 38 from the stroke data received from the handwritten signature authentication control unit 38.

[0128] <Handwritten input data saved by the handwritten input saving unit> Next, the handwritten input saved data will be described with reference to Fig. 16. Fig. 16 shows an example of handwritten input saved data saved in the handwritten input saving unit 25. One line in Fig. 16 represents one stroke, one sentence (text), or one image.

[0129] When Type is Stroke, one piece of saved handwriting input data has the following items: DataId, Type, PenId, ColorId, Angle, StartPoint, StartTime, EndPoint, EndTime, Point, and Pressure. DataId is identification information for the stroke. Type is the type of saved handwriting input data. Types include Stroke, Group, Text, and Image. The type of saved handwriting input data 801 and 802 is Stroke. The type of saved handwriting input data 803 is Group. The type of saved handwriting input data 804, 805, and 806 is Text. The type of saved handwriting input data 807 is Image.

[0130] Group means grouping other strokes, and handwriting input saved data with a type of Group specifies the stroke to be grouped by DataId. PenId, ColorId, and Angle are transcriptions of the pen ID control data described below. StartPoint is the coordinate of the start point of the stroke, and StartTime is the time at which the stroke started. EndPoint is the coordinate of the end point of the stroke, and EndTime is the time at which the stroke ended. Point is the coordinate sequence from the start point to the end point, and Pressure is the pen pressure from the start point to the end point. As shown in Angle, handwriting input saved data 804 and 805 were rotated clockwise by 180 degrees and 270 degrees, respectively, before being recognized as handwriting.

[0131] Furthermore, handwritten input saved data 804, 805, and 806 whose Type is Text have FontName, FontSize, and Text, where FontName is the font name of the text, FontSize is the character size, and Text is the content of the text (character code).

[0132] The handwritten input save data 801-805 may be attached as metadata as it is when the file is saved as handwritten input-derived data. Therefore, the handwritten input save data 801-805 can be acquired by the display device 2 when the file is read. When the display device 2 transmits the handwritten input save data to an external device, the handwritten input save data is converted to color for color-compatible devices, and may be transmitted without or after conversion to color for monochrome-compatible devices. When transmitting to a monochrome emphasis-compatible device, the handwritten input save data is simply transmitted. In this way, the handwritten input save data 801-805 supports both monochrome emphasis and color conversion.

[0133] While handwriting input saved data 801 to 805 are handwriting input-originated data, handwriting input saved data 806 and 807 are not. These are files loaded using a file load command. This is determined by whether a color value defined by the color definition data (described later) is registered in ColorId. For example, the ColorId of handwriting input saved data 801 is "Black," while the ColorId of handwriting input saved data 806 is "#e6001200." This is expressed as # and eight hexadecimal digits, with each pair of digits representing the values ​​of R (red), G (green), B (blue), and A (transparency).

[0134] In this way, in text data that is handwriting input-derived data, ColorId contains a character string representing color information, but in text data that is not handwriting input-derived data, ColorId is "#color value." In this way, handwriting input storage unit 25 can determine whether text data is handwriting input-derived data by focusing on ColorId.

[0135] The handwritten input saving unit 25 saves the color of the text data of the file read by the file transmission / reception control unit 37 as ColorId="#e6001200". Therefore, the handwritten input saved data 806 does not support black and white emphasis, but only supports color conversion.

[0136] The handwriting input display control unit 23 determines whether the data is handwriting input-derived data by ColorId, and if it is not handwriting input-derived data, calculates and displays a grayscale value from RGBA. If the display device 2 is a color-compatible device, it displays the RGBA as is.

[0137] The handwriting input saved data 807 is also an image file read from a storage medium by a file read command. If the Type is Image, the handwriting input saved data 807 has a FileId and a FileName. The FileId is a management number within the handwriting input saved data, and the FileName is the original file name. For example, if the file transmission / reception control unit 37 reads the file "color_chart.pdf" in FIG. 12(a), the RGBCMY text in the center is saved in the format of handwriting input saved data 806, and the ring-shaped color part is saved in the format of handwriting input saved data 807, both of which are saved in the handwriting input saving unit 25. Note that handwriting input saved data with a Type of Image is obviously not handwriting input-derived data. Therefore, the handwriting input display control unit 23 may determine whether the data is handwriting input-derived data based on the Type.

[0138] 16, whether data is handwriting-input-derived data is determined based on the ColorId value, but the handwriting input display control unit 23 may record whether data is handwriting-input-derived data using a dedicated flag or the like. Also, although this is limited to handwritten data whose Type is stroke, it is also possible to determine whether data is handwriting-input-derived data from its shape using machine learning. In this case, the learning device uses deep learning or the like to learn the correspondence between training data for determining whether data is handwriting-input-derived data and the shape of the characters, and outputs whether data is handwriting-input-derived data or not based on the shape of the input characters.

[0139] In addition, data resulting from handwritten input may include not only text data converted from handwritten data through character recognition, but also data converted based on user operations, such as stamps displayed as fixed characters such as "Done" and "Confidential," shapes such as circles and stars, and straight lines.

[0140] The handwritten input saved data is used in step S7 of FIG. 30 (pen coordinates and reception time), step S88 of FIG. 37 (acquisition of handwritten input saved data), step S96 of FIG. 37 (save file), and so on.

[0141] <Pen ID control data stored in the pen ID control data storage unit> Next, the pen ID control data will be described using FIG. 17. FIG. 17 is a diagram illustrating the pen ID control data stored in the pen ID control data storage unit 36. The pen ID control data controls the color of the handwriting input-derived data, etc. FIG. 17 shows four pens 2500. Each line in FIG. 17(a) shows the pen ID control data for one pen. FIG. 17(b) is a diagram illustrating angle information when a user writes handwriting on the display device 2. The angle information can be considered as the angle of the direction in which the user is present, the angle of the direction in which the pen is used, or the angle related to the rotation of characters handwritten by the user. With a predetermined direction of the display device 2 (e.g., the vertical direction) as 0 degrees (reference), the angle information for each user is 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees, and 315 degrees counterclockwise.

[0142] Note that the angle information is not automatically determined based on the user's position, but is input (specified) by each user. The resolution of the angle information that can be specified (45 degrees in Figure 17) is just one example, and a smaller resolution such as 5 to 30 degrees may also be used. However, it is believed that users will be able to read characters rotated at around 45 degrees.

[0143] The pen ID control data has PenId, ColorId, Angle, and AccountId. AccountId is associated with the pen used by the user to sign in. PenId is identification information stored inside the pen 2500. ColorId indicates the color selected by the user using this pen 2500 (can be changed by the user as desired). ColorId is set to the color ID of the color definition data described below. The color definition data defines the details of each color, thickness, etc.

[0144] Angle is stroke angle information set for this pen 2500 (can be changed as desired by the user). In the example of Figure 17(a), the angle information for each pen is 0 degrees, 90 degrees, 180 degrees, and 270 degrees counterclockwise. AccountId is user identification information. By associating AccountId with the pen ID, it is possible to identify the AccountId associated with the pen ID of the pen 2500 used by the user, and to execute operation commands using user-defined data.

[0145] Pen ID control data 901 is pen ID control data with PenId=1, color is black, angle information is 0 degrees, and AccountId=1. The user with AccountId=1 is the user in user-defined data 717 in Fig. 14. This indicates that this user signed in by handwriting a user name and the like with the pen with PenId=1, and selected black. Pen ID control data without an AccountId indicates a signed-out state (not associated with a user).

[0146] The pen ID control data is used in step S5 of Figure 30 (obtain pen ID control data), step S20 of Figure 32 (save angle information of pen ID control data), step S21 of Figure 32 (obtain angle information of pen ID control data), and step S60 of Figure 34 (obtain pen ID control data).

[0147] <Color definition data> Figure 18 shows an example of color definition data. One line of color definition data defines one color. The color definition data defines the black and white highlighting display on a black and white compatible device for ColorId (how to display the black and white pattern, width (Pattern), and edge (Edged)) and the color conversion method (how to display color information (Color) and width (Width) on a color compatible device). Color information is represented by # and an eight-digit hexadecimal number. Every two digits of color information represent #R (red), G (green), B (blue), and A (transparency), and px represents the pixel width. Note that color definition data only applies to data resulting from handwritten input.

[0148] Color definition data 1001 is an example of a definition where ColorId is "Black." Pattern represents the content of the stroke or text border. Edged represents the presence or absence of a border. Color represents RGBA color information. Width represents the line width. Color definition data 1001 indicates that handwritten input saved data is displayed as solid black 5 pixels wide + no border on a black and white emphasis compatible device, and as black with 0% transparency + 5 pixels wide on a color compatible device. Similarly, color definition data 1002 indicates that black and white emphasis compatible device displays as solid black 3 pixels wide + a black border 1 pixel wide from the outside + a white border 2 pixels wide on a black and white emphasis compatible device, and as red with 0% transparency on the color chart + 5 pixels wide on a color compatible device. Color definition data 1003 indicates that black and white emphasis compatible device displays as solid white 4 pixels wide + a black border 1 pixel wide on a black and white emphasis compatible device, and as blue with 0% transparency on the color chart + 5 pixels wide on a color compatible device. Color definition data 1004 indicates that a black and white emphasis-compatible device displays a black dot pattern 4 pixels wide plus a black border 1 pixel wide, and that a color-compatible device displays it with 0% green transparency on the color chart plus a 5-pixel border. Color definition data 1005 indicates that a black and white emphasis-compatible device displays a black upper right diagonal line pattern with 5-pixel transparent white plus no border, and that a color-compatible device displays it with 60% magenta transparency on the color chart. Color definition data 1006 indicates that a black and white emphasis-compatible device displays a 5-pixel transparent black horizontal line pattern with 5-pixel transparent white, and that a color-compatible device displays it with 60% cyan transparency on the color chart.

[0149] In this way, the color definition data includes black and white emphasis and color conversion data. The color definition data is held in the handwritten input storage unit 25 and is used to convert the handwritten input storage data.

[0150] By defining color definition data as system-defined data and user-defined data and adding a form to change the color definition data to the user-defined data, users can change the color definition to their personal preferences after signing in.

[0151] Figure 19 shows data displayed in black and white highlighting and in color display mode using the color definition data of Figure 18. Figure 19(a) shows the black and white highlighting displayed by a black and white highlighting compatible device, while Figure 19(b) shows the color conversion and display by a color compatible device. For ease of drawing, Figure 19(b) is also displayed in black and white (hatched).

[0152] Code C1 is text in "black," "red," "blue," "green," "magenta," and "cyan," which is data resulting from handwritten input. Code C2 is handwritten strokes in "black," "red," "green," "magenta," and "cyan." Code C3 is a spring-like, single-stroke handwritten stroke. Code C4 is a donut-shaped color chart (image). Code C5 is RGBCMY inside the color chart (text that is not data resulting from handwritten input). Below code C3 is the black text "Transparent," which is data resulting from handwritten input.

[0153] The color chart indicated by code C4 and the text RGBCMY indicated by code C5 are data read from a PDF file or the like, and are not handwriting-input-derived data. Therefore, in terms of handwriting-input-derived data stored by the handwriting-input-storage unit 25 shown in FIG. 16, the stroke (handwritten) indicated by code C2 is stored as handwriting-input-derived data 801 and 802 in FIG. 16. The text indicated by code C1, which is handwriting-input-derived data, is stored as handwriting-input-derived data 804 and 805 in FIG. 16. The text indicated by code C5, which is not handwriting-input-derived data, is stored as handwriting-input-derived data 806 in FIG. 16. The image indicated by code C4 is stored as handwriting-input-derived data such as handwriting-input-derived data 807 in FIG. 16. The color information of each handwriting-input-derived data is defined by ColorId.

[0154] As can be seen by comparing Figures 19(a) and (b), the red text "R" inside the color chart of code C10 in Figure 19(b) is not data originating from handwriting input. For this reason, the text "R" is displayed in grayscale in Figure 19(a). In contrast, the text "red" of code C6 is data originating from handwriting input, and is therefore displayed in highlighted black and white in Figure 19(a).

[0155] Also, because magenta and cyan are transparent colors, "Transparent" appears transparent in Figure 19(b). In Figure 19(a), magenta and cyan also appear transparent because they use a transparent white pattern. In this way, a device that supports black-and-white emphasis can highlight colors to make them stand out, and on a color device, they can be converted to color and displayed.

[0156] The color definition data is used in step S6 of FIG. 30 (coordinate complement display data), step S82 of FIG. 37 (character string object display data), and step S100 of FIG. 37 (object display data).

[0157] <Pen color selection button definition data> Next, a method for a user to select a pen color will be described with reference to Figures 20 and 21. Figure 20 is an example of pen color selection button definition data. The pen color selection button definition data is data that associates a pen button displayed on the display 220 with a ColorId. The pen color selection button definition data has a pen button ID (PenButtonId), an icon file (Icon), a color ID (ColorId), and a font name (FontName).

[0158] 21 shows an example of pen buttons 81 to 86 displayed on the display. The pen buttons 81 to 86 are displayed using the icon files described above. The pen buttons 81 to 86 are displayed in order from left to right in ascending order of pen button ID.

[0159] Pen color selection button definition data 1101 is definition data for pen button 81, which is displayed on the far left. When the user presses pen button 81 with pen 2500, the stroke data handwritten with that pen 2500 will have a ColorId of "Black" and the font of the text after handwriting recognition will be "Mincho." Pen color selection button definition data 1101 is for pen button 82, which is displayed second from the left, and the ColorId of the stroke data will be "Red" and the font of the text after handwriting recognition will be "Pop." Similarly, pen color selection button definition data 1103 to 1106 define the button display position, ColorId, and text font of pen buttons 83 to 86.

[0160] Figure 21(a) shows an example of a text font display that is highlighted in black and white based on the pen color selection button definition data. The text font display example is for illustrative purposes only, but may be displayed when the user hovers the pen 2500. Figure 21(b) shows an example of a text font display that is displayed in color on a color-capable device based on the pen color selection button definition data (it is actually in color, but is displayed in black and white for drawing purposes).

[0161] As shown in FIG. 21(a), by changing the font in addition to the black and white emphasis, it is possible to achieve a more effective black and white emphasis display.

[0162] If the pen color selection button definition data is also defined in system-defined data and user-defined data, and a form for changing the pen color selection button definition data is added to the user-defined data, the pen color selection button can be changed to suit individual preferences after handwritten signature authentication. Therefore, the correspondence between ColorId and font is just one example.

[0163] The pen color selection button is used in step S6 of FIG. 30 (coordinate complement display data), step S82 of FIG. 37 (character string object display data), and step S100 of FIG. 37 (object display data).

[0164] <Example of selectable candidates> FIG. 22 shows an example of an operation guide and selectable candidates 530 displayed by the operation guide. When the user draws a handwritten object 504 by hand (when the selectable candidate display timer times out), the operation guide 500 is displayed. The operation guide 500 has an operation header 520, operation command candidates 510, handwriting-recognized character string candidates 506, conversion character string candidates 507, character string / predictive conversion candidates 508, and a handwritten object rectangular area display 503. The selectable candidates 530 are the operation command candidates 510, handwriting-recognized character string candidates 506, conversion character string candidates 507, and character string / predictive conversion candidates 508. In this example, "No language conversion string" may be displayed. The selectable candidates 530 excluding the operation command candidates 510 are referred to as character string candidates 539.

[0165] Operation header 520 has buttons 501, 509, 502, and 505. Button 501 accepts an operation to switch between predictive conversion and kana conversion. In the example of FIG. 22, when the user presses button 509 displaying "Predictive," handwriting input unit 21 accepts the press and notifies handwriting input display control unit 23, and display unit 22 changes the display to button 509 displaying "Kana." After the change, character string candidates 539 are arranged in descending order of the probability of "Kana conversion."

[0166] Button 502 operates the page of the candidate display. In the example of FIG. 22, there are three candidate display pages, and the first page is currently displayed. Button 505 accepts erasure of the operation guide 500. When the user presses button 505, the handwriting input unit 21 accepts it and notifies the handwriting input display control unit 23 of that fact, and the display unit 22 erases the display except for the handwritten objects. Button 509 accepts erasure of all displays. When the user presses button 509, the handwriting input unit 21 accepts it. The handwriting input unit 21 notifies the handwriting input display control unit 23 of that fact. The display unit 22 erases all displays shown in FIG. 22, including the handwritten objects, allowing the user to start handwriting again from the beginning.

[0167] A handwritten object 504 is the character "gi" handwritten by the user. A handwritten object rectangular area display 503 is displayed surrounding the handwritten object 504. The display procedure will be explained with reference to the sequence diagrams of Figs. 30 to 37. In the example of Fig. 22, the handwritten object rectangular area display 503 is displayed in a dotted line frame.

[0168] The character string candidates are arranged in descending order of probability in handwriting recognition character string candidate 506, conversion character string candidate 507, and character string / predictive conversion candidate 508. "gi" in handwriting recognition character string candidate 506 is a candidate for the recognition result. In this example, the display device 2 correctly recognizes "gi."

[0169] Conversion string candidate 507 is a conversion string candidate converted from a language string candidate. In this example, "Technical trial" is an abbreviation for "Technical mass production trial." Character string / predictive conversion candidate 508 is a predicted character string candidate converted from a language string candidate or a conversion string candidate. In this example, "Approve the technical trial" and "Address to send the minutes" are displayed.

[0170] The operation command candidate 510 is a candidate operation command selected based on the operation command definition data 701 to 703 and 709 to 716 in Fig. 11(a). In the example of Fig. 22, the first character of the line "》" 511 is shown to be a candidate operation command. In Fig. 22, there is no selected object for the handwritten object 504 "ぎ" to select, and the character string candidate for "ぎ" "meiji" (minutes) partially matches the operation command definition data 701 and 702 in Fig. 11(a), so it is displayed as the operation command candidate 510.

[0171] When the user selects "Load minutes template," the operation command defined in operation command definition data 701 is executed, and when the user selects "Save in minutes folder," the operation command defined in operation command definition data 702 is executed. In this way, operation command candidates are displayed only when operation command definition data containing the converted character string is found, so they may not always be displayed.

[0172] As shown in FIG. 22, character string candidates and operation command candidates are displayed simultaneously (together), so the user can freely select either the character string candidate or the operation command that he or she intended to input.

[0173] <Example of specifying a selected object> In the display device 2 of this embodiment, the user can designate a selected object by selecting a fixed object by handwriting. The selected object becomes a target for editing or modification.

[0174] Fig. 23 is an example of a diagram for explaining an example of designating a selected object. In Fig. 23, a handwritten object 11 is displayed with a solid black line, a handwritten object rectangular area 12 is displayed with gray shading, a confirmed object 13 is displayed with a black line, and a selected object rectangular area 14 is displayed with a dotted line. Lowercase letters added to the symbols are used to distinguish them from one another. Furthermore, the crossing line determination condition 406 or the enclosing line determination condition 407 of the predefined control data shown in Fig. 7 is used as the determination condition for determining whether a confirmed object is a selected object (whether or not there is a predetermined relationship).

[0175] 23(a) shows an example in which the user has specified two horizontally written fixed objects 13a and 13b with a crossover line (hand-drawn object 11a). In this example, the length H1 of the short side and the length W1 of the long side of the hand-drawn object rectangular area 12a satisfy the crossover line determination condition 406, and the overlap rate with the fixed objects 13a and 13b also satisfies the crossover line determination condition 406. Therefore, both the fixed objects 13a and 13b, "minutes" and "giji," are specified as selected objects.

[0176] 23(b) shows an example in which the user specifies a horizontally written fixed object 13c by enclosing the handwritten object 11b. In this example, only the fixed object 13c, "minutes," whose overlap rate with the handwritten object rectangular area 12c satisfies the enclosing line determination condition 407, is specified as the selected object.

[0177] 23(c) is an example in which multiple vertically written fixed objects 13d and 13e are specified with a crossing line (hand-drawn object 11c). In this example, similar to FIG. 23(a), the length H1 of the short side and the length W1 of the long side of the hand-drawn object rectangular area 12d satisfy the condition of the crossing line determination condition 406, and the overlap rate with each of the two fixed objects 13d and 13e, "minutes" and "giji", satisfy the condition of the crossing line determination condition 406. Therefore, both the fixed objects 13d and 13e, "minutes" and "giji", are specified as selected objects.

[0178] 23(d) shows an example in which the user specifies a vertically written fixed object 13f by enclosing it (handwritten object 11d). In this example, just like in FIG. 23(b), only the fixed object 13f, "minutes," is specified as the selected object.

[0179] <Example of displaying operation command candidates> Figure 24 shows an example of displaying operation command candidates based on the operation command definition data when there is the handwritten object shown in Figure 13. Figure 24(a) shows candidates for editing operation commands, and Figure 24(b) shows candidates for modification operation commands. Figure 24(a) also shows an example in which a selected object is specified by the handwritten object 11a in Figure 23(a).

[0180] As shown in Figures 24(a) and 24(b), the operation command candidates displayed following the first character "»" 511 are a main menu 550. The name of the operation command last executed or the name of the first operation command in the operation command definition data is displayed in the main menu 550. The first character "»" 511a on the first line is a candidate for an editing-related operation command, and the first character "»" 511b on the second line is a candidate for a modification-related operation command.

[0181] The ">" 512 at the end of the line indicates that there is a sub - menu (an example of a sub - menu button). The ">" 512a in the first line displays the sub - menu (the last - selected one) which is a candidate for editing - related operation commands. The ">" 512b in the second line displays the remaining sub - menus which are candidates for formatting - related operation commands. When the user presses the ">" 512, a sub - menu 560 is displayed on its right side. All the operation commands defined in the operation command definition data are displayed in the sub - menu 560. In the display example of FIG. 24(a), the sub - menu 560 corresponding to the ">" 512a in the first line is also displayed since the main menu is displayed. The sub - menu 560 may be displayed by pressing the ">" 512a in the first line.

[0182] When the user presses any operation command name with the pen, the Command of the operation command definition data associated with the operation command name is executed by the handwritten input display control unit 23 for the selected object. That is, when "Delete" 521 is selected, "Delete" is executed; when "Move" 522 is selected, "Move" is executed; when "Rotate" 523 is selected, "Rotate" is executed; when "Select" 524 is selected, "Select" is executed respectively.

[0183] For example, when the user presses "Delete" 521 with the pen, "Proceedings" and "Draft" can be deleted. When "Move" 522, "Rotate" 523, or "Select" 524 is pressed, a bounding box (the circumscribed rectangle of the selected object) is displayed. When the user presses "Move" 522 and "Rotate" 523, the bounding box can be moved or rotated by the drag operation of the pen. When the user presses "Select" 524, other operations on the bounding box can be executed.

[0184] The strings "—" 541, "—," 542, "~" 543, "→" 544, "⇒" 545 which are string candidates other than operation command candidates are the recognition results of the strikethrough line (handwritten object 11a). When the user intends to input a string rather than an operation command, the string candidate can be selected.

[0185] In FIG. 24(b), pressing ">" 512b on the second line displays the submenu of FIG. 24(b). In the display example of FIG. 24(b), a main menu 550 and a submenu 560 are displayed, similar to FIG. 24(a). Based on the operation command definition data of FIG. 13, when "thicken" 531 is selected, the handwriting input display control unit 23 executes "Thick" on the selected object. When "thin" 532 is selected, the handwriting input display control unit 23 executes "Thin" on the selected object. When "larger" 533 is selected, the handwriting input display control unit 23 executes "Large" on the selected object. When "smaller" 534 is selected, the handwriting input display control unit 23 executes "Small" on the selected object. When underline 535 is selected, the handwriting input display control unit 23 executes "Underline" on the selected object.

[0186] Note that fixed values ​​are separately defined for how thick to make the line when "Thick" 531 is selected, how thin to make the line when "Thin" 532 is selected, how large to make the line when "Larger" 533 is selected, how small to make the line when "Smaller" 534 is selected, and the type of line when underline 535 is selected. Alternatively, it would be better if a separate selection menu is opened when the submenu in Figure 24(b) is selected, allowing the user to make adjustments.

[0187] When the user presses "thicken" 531 with the pen, the handwriting input display control unit 23 thickens the lines that make up the final objects 13a and 13b, "minutes" and "giji." When the user presses "thinner" 532 with the pen, the handwriting input display control unit 23 can thin the lines that make up "minutes" and "giji." When the user presses "larger" 533 with the pen, the handwriting input display control unit 23 can make the lines larger. When the user presses "smaller" 534 with the pen, the handwriting input display control unit 23 can make the lines smaller. When the user presses "underline" 535 with the pen, the handwriting input display control unit 23 can add an underline.

[0188] FIG. 25 shows an example of displaying operation command candidates based on the operation command definition data when there is a handwritten object as shown in FIG. 13. The difference from FIGS. 24(a) and (b) is that FIGS. 25(a) and (b) show an example in which a selection object is specified by the handwritten object 11b (encircling line) of FIG. 23(b). As can be seen by comparing FIGS. 24 and 25, there is no difference in the operation command candidates displayed depending on whether the handwritten object is a line or an encircling line. This is because the handwritten input display control unit 23 displays operation command candidates on the display unit 22 when a selection object is specified. However, the handwritten input display control unit 23 may recognize the handwritten object and change the operation command candidates to be displayed depending on the handwritten object. In this case, the recognized handwritten object (such as a one or a circle) is associated with operation command definition data as shown in FIG. 13.

[0189] In FIG. 25, the character string candidates other than the operation command candidates, “○” 551, “∞” 552, “0” 553, “00” 554, and “ロ” 555, are the recognition results of the enclosing line (handwritten object 11b), and if the user intends to input a character string instead of an operation command, the user can select a character string candidate.

[0190] <Example of angle information input> Next, a method for inputting angle information will be described with reference to Fig. 26. Fig. 26 is an example of a diagram for explaining a method for inputting angle information. Fig. 26 explains a case where a user positioned at the 3 o'clock position of the display device 2 inputs angle information. Characters, etc. handwritten from the 3 o'clock position are correctly recognized when rotated 90 degrees clockwise, so it is preferable to input angle information of 90 degrees.

[0191] 26(a) shows a state in which the angle information of the pen ID control data is 0 degrees (initial value) and the user standing at the 3 o'clock position of the display device 2 has handwritten "ぎ", causing the operation guide 500 to be displayed. Because the display device 2 recognizes the character "ぎ" handwritten from the 3 o'clock position with the angle information remaining at 0 degrees, a selectable candidate 530 that is different from the expected one is displayed.

[0192] When inputting angle information, the user draws a straight line in the operation guide 500 from top to bottom as seen by the user. FIG. 26(b) shows an example of this straight line 521. The angle α formed counterclockwise between the 6 o'clock direction where the angle information is 0 degrees and this straight line 521 is the angle information. That is, the angle α formed counterclockwise between the straight line 522 drawn downward from the starting point S in the 6 o'clock direction and the straight line 521 input by the user is the angle information. Briefly, the direction of the end point of the straight line 521 is the angle information. Therefore, the angle information input by the user in FIG. 26(b) is 90 degrees.

[0193] Regarding the detection of a straight line, for example, there is a method of converting the coordinates from the starting point S to the end point E into a straight line by the least squares method and comparing the obtained correlation coefficient with a threshold value to determine whether it is a straight line.

[0194] Immediately after the user starts writing the straight line 521 (immediately after the pen 2500 touches the starting point S of the straight line 521), the display device 2 erases the operation guide 500. Also, immediately after finishing writing the straight line 521 (immediately after the pen 2500 leaves the end point E of the straight line 521), the display device 2 searches for the value closest to the above-mentioned formed angle α from 45 degrees, 90 degrees, 135 degrees, 180 degrees, 215 degrees, 270 degrees, 315 degrees, 360 degrees and determines it as the angle information. The formed angle α itself may also be the angle information. The determined angle information is set in the Angle of the pen ID control data. When the pen tip of the pen 2500 is pressed for handwriting or the like, the pen event transmission unit 41 of the pen 2500 transmits the pen ID to the display device 2, so the display device 2 can associate the angle information with the pen ID control data.

[0195] Note that the user can input angle information only by handwriting a straight line in the operation guide 500. As a result, when the user handwrites a straight line outside the operation guide 500, it is recognized as "1" or "—", etc., and when handwriting a straight line in the operation guide 500, angle information can be input. That is, the handwriting recognition control unit 26 detects a straight line from a predetermined range and converts the stroke data handwritten outside the predetermined range into text data.

[0196] Figure 26(c) shows the operation guide 500 immediately after the operation in Figure 26(b). Since 90 degrees is set as the angle information (Angle) in the pen ID control data, the handwritten object (stroke data) is internally rotated 90 degrees clockwise for handwriting recognition, and the operation guide 500 is displayed rotated 90 degrees counterclockwise. Note that the user may also manually input the angle information from a menu.

[0197] <Example of handwritten signature data registration> Next, an example of registering handwritten signature data will be described with reference to Fig. 27. Fig. 27 is a diagram illustrating a method for registering handwritten signature data. First, Fig. 27(a) is an example of selectable candidates 530 that are displayed when the user handwrites "signature." Two operation commands 513, 514, "Register handwritten signature" and "Sign out handwritten signature," based on operation command definition data 713, 715, which partially match the character string "signature," as well as character string candidates "sign," "autograph session," and "signed" are displayed. The two operation commands 513, 514 are displayed because the String in the operation command definition data 713, 715 in Fig. 11 includes "signature."

[0198] When the user presses "Register handwritten signature" with the pen 2500, the handwritten signature registration form 561 shown in FIG. 27(b) is added to the handwritten input storage unit 25 and displayed on the operation screen 101. For example, the operation guide 500 shown in FIG. 27(a) is erased, and the handwritten signature registration form 561 is displayed in the same location as the operation guide 500. The handwritten signature registration form 561 has, from the top, a name input field 561a, signature input fields 561b to 561d, and a registration confirmation field 561e. The user enters the name text in the name input field 561a, the first, second, and third handwritten signatures in the signature input fields 561b to 561d, and a check mark or cancel mark in the registration confirmation field 561e. The name text is the user's display name and is converted into text data. The handwritten signature is entered three times because the features are registered on the assumption that handwritten signatures are different each time a user writes them and will not exactly match.

[0199] A handwritten signature is generally a user name or other characters related to the user, and may be a user name, numbers such as an employee number, a nickname, or a portrait. A handwritten signature is not limited to characters related to the user, but may be any handwritten object. For example, a handwritten signature may be a circle, triangle, square, symbol, or a combination thereof. Since coordinates are not the only feature data for a handwritten signature, it can be correctly authenticated even if two users with the same surname (for example, Mr. Suzuki) both register the handwritten signature "Suzuki."

[0200] When the user writes his / her handwriting in the handwritten signature registration form 561 as instructed, the handwritten signature registration form 561 shown in FIG. 27(c) appears. When the user writes a check mark in the registration confirmation field 561e, the handwritten signature data is registered in the handwritten signature data storage unit 39, and the handwritten signature registration form 561 is deleted. A Signature ID is assigned as a result of the registration, and the similarly assigned Account ID and the name text in the name input field 561a are associated with the Signature ID and registered in the user-defined data. When the user signs in by handwriting a user name or the like, the Signature ID associated with the Account ID in the user-defined data is acquired and registered in the pen ID control data in association with the pen ID of the pen 2500 used for the handwritten sign-in. Thereafter, whenever the user uses the pen 2500, the pen ID is sent to the display device 2, and the Account ID associated with the pen ID in the pen ID control data can be identified, making it possible to execute operation commands using the user-defined data without the user even being aware of it.

[0201] If the user handwrites an "X" in the registration confirmation field 561e, the handwritten signature registration is canceled and the handwritten signature registration form 561 is deleted. If any error occurs in the registration, a message to that effect is displayed in a system reserved area on the operation screen 101 or the like.

[0202] In this way, the handwriting input display control unit 23 can accept handwriting input without distinguishing between handwriting input into a form and handwriting input outside of a form.

[0203] <Example of handwritten sign-in> Next, a method for a user to sign in after registering handwritten signature data will be described with reference to Figure 28. Figure 28 shows an example of operation guide 500 that is displayed when the user handwrites "Suzuki," which is registered handwritten signature data. Since "Suzuki" is registered as handwritten signature data in handwritten signature data storage unit 39, "Suzuki" matches the handwritten signature data. Therefore, operation command 512, "Sign in by handwriting," is displayed.

[0204] Furthermore, since the handwritten signature data matches, the SignatureId representing "Suzuki" is identified, and the user-defined data having the AccountId associated with this SignatureId is identified.

[0205] When the user selects the operation command 512 "Sign in by hand," the AccountId of "Suzuki" is associated with the pen ID of the pen 2500 being used and added to the pen ID control data, and the user-defined data of "Suzuki" is used when the operation command is used.

[0206] 27 is controlled in the same way as normal handwritten input of characters, etc. Therefore, the handwritten signature registration form 561 is displayed on the same operation screen as the operation screen on which characters, etc. are handwritten. There is no difference between handwriting operations inside and outside the handwritten signature registration form 561, and the user can complete input into the handwritten signature registration form 561 simply by handwriting input into the areas separated by ruled lines on the handwritten signature registration form 561.

[0207] <Example of changing user-defined data> Next, a method for changing user-defined data will be described with reference to Fig. 29. Fig. 29 is a diagram for explaining a method for changing user-defined data. Fig. 29(a) is an example of operation guide 500 that is displayed when the user handwrites "se". In the operation command definition data 716 shown in Fig. 11, "settings" is defined in the String, and "settings" was included in the predicted character string for "se", so the operation command "change settings" was displayed.

[0208] When a user selects "Change settings" using the pen 2500 with which they have signed in by handwriting, the Account ID of the user associated with the pen ID of this pen 2500 is identified in the pen ID control data. This identifies the user-defined data of the signed-in user, and a user-defined data change form 562 shown in FIG. 29(b) is added to the handwriting input storage unit 25. The user-defined data change form 562 is displayed on the operation screen 101. In the example of FIG. 29, the user-defined data change form 562 is created in accordance with the user-defined data 718 shown in FIG. 14. The user-defined data change form 562 has a name field 562a, a password field 562b, a folder user name field 562c, a folder password field 562d, a folder file name field 562e, and a register or cancel field 562f.

[0209] If the user has not signed in by handwriting in advance, the display device 2 will be unable to identify the user's Account ID, resulting in an error, and an error message will be displayed in a system reserved area on the operation screen 101 or the like.

[0210] In the user-defined data change form 562 of Figure 29 (b), the user handwrites a password in the password field 562b. The user handwrites a folder user name in the folder user name field 562c. The user handwrites a folder password in the folder password field 562d. The user handwrites a folder file name in the folder file name field 562e. The user handwrites a "check mark" or "x" in the register or cancel field 562f. As a result, the user-defined data change is executed and the user-defined data change form 562 is deleted.

[0211] In this way, the user can display user-defined data change form 562 and change the user-defined data as desired by handwriting stroke data that calls user-defined data change form 562. Handwriting input display control unit 23 accepts handwriting input without distinguishing between handwriting input into a form and handwriting input outside of a form.

[0212] The AccountUsername of the user-defined data is automatically displayed in the name field 562a. The user-defined data change form 562 can be used not only for changes but also for registration.

[0213] Changing user-defined data using user-defined data change form 562 in Figure 29 is controlled in the same way as normal handwritten input of characters, etc. Therefore, user-defined data change form 562 is displayed on the same operation screen as the operation screen on which characters, etc. are handwritten. There is no difference between handwriting operations inside and outside user-defined data change form 562, and the user can complete input into user-defined data change form 562 simply by handwriting input into the ruled areas of user-defined data change form 562.

[0214] <Operation procedure> The operation of the display device 2 will be described using the above configuration and Figs. 30 to 37. Figs. 30 to 37 are sequence diagrams illustrating an example of the process in which the display device 2 displays character string candidates and operation command candidates. The process in Fig. 30 starts when the display device 2 is started (when the application is started). Due to space limitations, Figs. 30 to 37 use symbols to indicate the functions of Fig. 6.

[0215] Before starting to input handwritten data, the user has already selected one of the pen buttons 81 to 86 (the PenId has been specified). a. The pen button ID, Color ID and font are specified in the pen color selection button definition data, b. Pen ID and Color ID are registered in the pen ID control data.

[0216] S1: First, the handwriting input display control unit 23 sends a handwriting object start signal to the handwriting input storage unit 25. The handwriting input storage unit 25 allocates a handwriting object area (a memory area for storing handwritten objects). The handwriting object area may be allocated after the user brings the pen into contact with the handwriting input unit 21.

[0217] S2: Next, the user touches the pen to the handwriting input unit 21. The handwriting input unit 21 detects the pen down and transmits it to the handwriting input display control unit 23.

[0218] S3: The handwritten input display control unit 23 transmits a stroke start signal to the handwritten input storage unit 25, and the handwritten input storage unit 25 reserves a stroke area.

[0219] S4: When the user moves the pen while keeping it in contact with the handwriting input unit 21, the handwriting input unit 21 transmits the pen coordinates to the handwriting input display control unit 23.

[0220] S5: The handwriting input display control unit 23 specifies the pen ID received from the pen 2500 at the same time as the coordinates are input, and acquires the current pen ID control data stored in the pen ID control data storage unit 36. The pen ID is sent when the coordinates are input, so the stroke and the pen ID are associated. The pen ID control data storage unit 36 ​​sends the pen ID control data (ColorId (angle information is still set to zero, and there is no AccountId yet)) to the handwriting input display control unit 23. Note that the angle information is still at the initial value of zero. Also, there is no AccountId because the user has not signed in.

[0221] S6: The handwriting input display control unit 23 transmits pen coordinate complement display data (data for interpolating discrete pen coordinates) to the display unit 22. The display unit 22 interpolates the pen coordinates using the pen coordinate complement display data, specifies the line type and thickness from the color definition data based on ColorId, and displays the stroke.

[0222] S7: The handwriting input display control unit 23 transmits the pen coordinates, the reception time thereof, ColorId, and angle information to the handwriting input saving unit 25. The handwriting input saving unit 25 adds the pen coordinates to the stroke. While the user is moving the pen, the handwriting input unit 21 periodically repeats transmitting the pen coordinates to the handwriting input display control unit 23, and therefore, the processes of steps S4 to S7 are repeated until the pen is lifted.

[0223] S8: When the user removes the pen from the handwriting input unit 21, the handwriting input unit 21 transmits a pen-up signal to the handwriting input display control unit 23.

[0224] S9: The handwriting input display control unit 23 sends a stroke end signal to the handwriting input saving unit 25, and the handwriting input saving unit 25 confirms the pen coordinates of the stroke. Once the pen coordinates of the stroke have been confirmed, it becomes impossible to add pen coordinates to the stroke.

[0225] S10: Next, the handwriting input display control unit 23 transmits an overlap status acquisition of the rectangular area near the handwritten object and the stroke rectangular area based on the rectangular area near the handwritten object 403 to the handwriting input saving unit 25. The handwriting input saving unit 25 calculates the overlap status and transmits the overlap status to the handwriting input display control unit 23.

[0226] The subsequent steps S11 to S17 are executed if the handwritten object vicinity rectangular area and the stroke rectangular area do not overlap.

[0227] S11: If the rectangular area near the handwritten object and the rectangular area of ​​the stroke do not overlap, one handwritten object is determined, and the handwritten input / display control unit 23 transmits a command to clear the stored data to the handwriting recognition control unit 26.

[0228] S12 to S14: The handwriting recognition control unit 26 sends a retained data clear command to the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32. These clear the data relating to character string candidates and operation command candidates that have been retained by the handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32. At the time of clearing, the last handwritten stroke has not been added to the handwritten object.

[0229] S15: The handwriting input display control unit 23 transmits a handwriting object end notification to the handwriting input saving unit 25. The handwriting input saving unit 25 finalizes the handwritten object. Finalizing a handwritten object means that one handwritten object has been completed (no more strokes can be added).

[0230] S16: The handwriting input display control unit 23 transmits a handwriting object start signal to the handwriting input saving unit 25. In preparation for the start of handwriting of the next handwritten object (pen down), the handwriting input saving unit 25 reserves a new handwritten object area.

[0231] S17: Next, the handwriting input display control unit 23 transmits a stroke addition for the stroke completed in step S9 to the handwriting input saving unit 25. If steps S11 to S17 have been executed, the added stroke is the first stroke of the handwritten object, and the handwriting input saving unit 25 adds the stroke data to the currently started handwritten object. If steps S11 to S17 have not been executed, the added stroke is added to the currently handwritten object.

[0232] S18: Next, the handwriting input display control unit 23 transmits a stroke addition to the handwriting recognition control unit 26. The handwriting recognition control unit 26 adds the stroke data to a stroke data storage area (area where stroke data is temporarily stored) where character string candidates are stored.

[0233] S19: The handwriting recognition control unit 26 performs gesture handwriting recognition on the stroke data storage area. Gesture handwriting recognition refers to recognizing angle information from a straight line. Since gesture handwriting recognition is performed inside the operation guide 500, the handwriting recognition control unit 26 detects straight lines inside the operation guide 500. The position information of the operation guide 500 is sent to the handwriting recognition control unit 26 in step S67, which will be described later.

[0234] S20: If a straight line is detected in the operation guide 500, the angle α formed counterclockwise between a straight line 522 drawn from the starting point of the straight line in the 6 o'clock direction and the straight line 521 input by the user is determined in 45-degree increments. Then, the handwriting recognition control unit 26 associates the determined angle information with the pen ID contained in the stroke data of the straight line 521 and stores it in the pen ID control data storage unit 36. Note that step S20 is executed when a straight line is detected in the operation guide 500.

[0235] S20-2: The handwriting recognition control unit 26 clears the selectable candidate display rectangle.

[0236] S21: Next, the handwriting recognition control unit 26 acquires the angle information of the current pen ID control data from the pen ID control data storage unit 36, specifying the pen ID received from the handwriting input unit 21.

[0237] S22: The handwriting recognition control unit 26 rotates the stroke data in the stroke data storage area clockwise using the acquired angle information.

[0238] S23: The handwriting recognition control unit 26 transmits the rotated stroke data to the handwritten signature authentication control unit 38. In this way, the stroke data is always transmitted to the handwritten signature authentication control unit 38 in a state where it is unclear whether it is a handwritten signature or not.

[0239] S24: The handwritten signature authentication control unit 38 receives the stroke data and receives the registered handwritten signature data from the handwritten signature data storage unit 39. The handwritten signature authentication control unit 38 then compares (matches) the stroke data with the handwritten signature data and stores the authentication result of the handwritten signature so that it can be obtained in subsequent step S61. If the authentication is successful, the AccountId is registered in the pen ID control data.

[0240] S25: Next, the handwriting recognition control unit 26 performs handwriting recognition on the stroke data, and if there is a "check mark" or "x" in the registration or cancellation column of the form, processes the form; otherwise, it performs normal handwriting recognition processing.

[0241] S26: If there is a check mark in the registration or cancellation field of the handwritten signature data registration form, the handwritten signature data (stroke data) entered by the user into the handwritten signature registration form is sent by the handwritten signature recognition control unit 26 to the handwritten signature authentication control unit 38. The handwritten signature registration form is generated in the handwritten input storage unit 25 by the handwritten input display control unit 23 in step S86, which will be described later.

[0242] S27: The handwritten signature authentication control unit 38 registers the received handwritten signature data (stroke data) in the handwritten signature data storage unit 39. As a result, a SignatureId is assigned. The SignatureId is returned to the handwriting recognition control unit 26. If the SignatureId and the name entered in the name input field 561a of the handwritten signature registration form 561 are not present in the user-defined data, the handwriting recognition control unit 26 adds new user-defined data. The handwriting recognition control unit 26 also assigns a number to an AccountId and saves the SignatureId in the user-defined data. If the name entered in the name input field 561a is present in the user-defined data, the SignatureId is saved in the user-defined data. This process links the AccountId and SignatureId. Note that when new user-defined data is added, no other values ​​are set, but the user can register and change them using the user-defined data change form 562.

[0243] S28: By registering the handwritten signature data, the handwriting recognition control unit 26 deletes the handwritten signature registration form 561 from the handwritten input storage unit 25.

[0244] S29: If there is a check mark in the registration or cancellation field of the user-defined data change form, the handwriting recognition control unit 26 sends the change value entered in the user-defined data change form 562 to the operation command definition unit 33. The user-defined data change form 562 is generated in the handwriting input storage unit 25 by the handwriting input display control unit 23 in step S86, which will be described later.

[0245] S30: When the user-defined data change is executed, the handwriting recognition control unit 26 deletes the user-defined data change form 562 from the handwriting input storage unit 25.

[0246] S31: If the registration or cancellation column of the form added in step S86 (to be described later) contains "x", the handwriting recognition control unit 26 deletes the form added in step S86 from the handwriting input storage unit 25.

[0247] S33: If the processing is not form processing, the handwriting recognition control unit 26 transmits the handwriting recognition character string candidates that are the execution results to the handwriting recognition dictionary unit 27. The handwriting recognition dictionary unit 27 transmits the language character string candidates that are linguistically likely to the handwriting recognition control unit 26.

[0248] S34: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidate and the received language character string candidate to the character string conversion control unit 28.

[0249] S35: The character string conversion control unit 28 transmits the handwriting recognition character string candidates and the language character string candidates to the character string conversion dictionary unit 29. The character string conversion dictionary unit 29 transmits the converted character string candidates to the character string conversion control unit 28.

[0250] S36: The character string conversion control unit 28 transmits the received converted character string candidate to the predictive conversion control unit 30.

[0251] S37: The predictive conversion control unit 30 transmits the received converted character string candidates to the predictive conversion dictionary unit 31. The predictive conversion dictionary unit 31 transmits the predicted character string candidates to the predictive conversion control unit 30.

[0252] S38: The predictive conversion control unit 30 transmits the received predicted character string candidate to the operation command recognition control unit 32.

[0253] S39: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) matching the predicted character string candidate.

[0254] Thereafter, the display device 2 performs the same processing up to the transmission of operation command candidates, which will be explained in steps S40 to S47.

[0255] S40: The character string conversion control unit 28 transmits the received converted character string candidate to the operation command recognition control unit 32.

[0256] S41: The operation command recognition control unit 32 transmits the received conversion character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) that matches the conversion character string candidate.

[0257] S42: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidates and the language character string candidates to the predictive conversion control unit 30.

[0258] S43: The predictive conversion control unit 30 transmits the handwritten recognition character string candidate and the received language character string candidate to the predictive conversion dictionary unit 31. The predictive conversion dictionary unit 31 transmits the predicted character string candidate to the predictive conversion control unit 30.

[0259] S44: The predictive conversion control unit 30 transmits the received predicted character string candidates to the operation command recognition control unit 32.

[0260] S45: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) matching the predicted character string candidate.

[0261] S46: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidate and the received language character string candidate to the operation command recognition control unit 32.

[0262] S47: The operation command recognition control unit 32 transmits the handwriting recognition character string candidate and the received language character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) that matches the language character string candidate.

[0263] S48: Next, the handwriting recognition control unit 26 transmits the stroke addition to the operation command recognition control unit 32.

[0264] S49: The operation command recognition control unit 32 transmits the position information of the confirmed object to the handwriting input storage unit 25. The handwriting input storage unit 25 transmits the position information of the confirmed object to the operation command recognition control unit 32.

[0265] S50: The operation command recognition control unit 32 determines whether the position information of the stroke received from the handwriting recognition control unit 26 when adding the stroke in step S48 has a predetermined relationship with the position information of the confirmed object received from the handwriting input storage unit 25, based on the crossing line determination condition 406 and the encircling line determination condition 407. In this way, the operation command recognition control unit 32 determines the selected object. The operation command recognition control unit 32 stores the confirmed object that can be determined to be selected as the selected object. In addition, in this case, since the selected object is identified, candidates for input / output operation commands are obtained from the operation command definition unit 33.

[0266] In addition, the handwriting recognition control unit 26, the string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 each store handwriting recognition string candidates, language string candidates, conversion string candidates, predicted string candidates, operation command candidates, and data related to the selected object so that they can be acquired in subsequent steps S55 to S58.

[0267] S18-2: Immediately after transmitting the stroke addition command to the handwriting recognition control unit 26 in step S18, the handwriting input display control unit 23 transmits a selectable candidate display timer start command to the candidate display timer control unit 24. The candidate display timer control unit 24 starts this timer.

[0268] The following steps S51 to S53 are executed if a pen-down occurs before a certain time has elapsed (before the timer times out).

[0269] S51: If the user touches the pen to the handwriting input unit 21 before the timer times out, the handwriting input unit 21 transmits a pen-down signal (the same event as in step S2) to the handwriting input display control unit 23.

[0270] S52: The handwriting input display control unit 23 transmits a stroke start (same as step S3) to the handwriting input saving unit 25. The subsequent sequence is the same as step S3 and after.

[0271] S53: Furthermore, the handwriting input display control unit 23 transmits a selectable candidate display timer stop command to the candidate display timer control unit 24. The candidate display timer control unit 24 stops the timer because the timer is no longer necessary since pen-down has been detected.

[0272] Steps S54 to S103 are executed if the pen does not go down before a certain time has elapsed (before the timer times out), and therefore the operation guide 500 shown in FIG.

[0273] S54: If the user does not bring the pen into contact with the handwriting input unit 21 while the selectable candidate display timer is in progress, the candidate display timer control unit 24 transmits a timeout to the handwriting input display control unit 23.

[0274] S55: The handwriting input display control unit 23 transmits a request to acquire handwriting recognition character string / language character string candidates to the handwriting recognition control unit 26. The handwriting recognition control unit 26 transmits the handwriting recognition character string / language character string candidates currently held to the handwriting input display control unit 23.

[0275] S56: The handwriting input display control unit 23 transmits a request to acquire a conversion character string candidate to the character string conversion control unit 28. The character string conversion control unit 28 transmits the conversion character string candidate currently held to the handwriting input display control unit 23.

[0276] S57: The handwriting input display control unit 23 transmits a predicted character string candidate acquisition request to the predictive conversion control unit 30. The predictive conversion control unit 30 transmits the predicted character string candidates currently held to the handwriting input display control unit 23.

[0277] S58: The handwriting input display control unit 23 transmits the acquired operation command candidates to the operation command recognition control unit 32. The operation command recognition control unit 32 transmits the operation command candidates and the selected object currently held to the handwriting input display control unit 23.

[0278] S59: Furthermore, the handwriting input display control unit 23 transmits the estimated writing direction acquisition to the handwriting input saving unit 25. The handwriting input saving unit 25 determines the estimated writing direction from the stroke addition time, horizontal distance, and vertical distance of the handwritten object rectangular area, and transmits the estimated writing direction to the handwriting input display control unit 23.

[0279] S60: Next, the handwriting input display control unit 23 specifies the pen ID received from the handwriting input unit 21 and acquires the ColorId, AccountId, and angle information of the current pen ID control data from the pen ID control data storage unit .

[0280] S61: The handwriting input display control unit 23 obtains the authentication result of the handwritten signature from the handwritten signature authentication control unit 38. This provides the user's SignatureId, and when an operation command (described later) is executed, the AccountId is associated with the pen ID and registered in the pen ID control data. When the user signs in, color definition data associated with the user-defined data identified by the AccountId is identified. The display device 2 can display handwritten data in which the color information of the user-defined data is highlighted in black and white, or the text data converted from the handwritten data.

[0281] S62: The handwriting input display control unit 23 creates selectable candidate display data as shown in Figure 22 from these handwriting recognition character string candidates ("gi" in Figure 22), language character string candidates (for example, "gi" although not shown in Figure 22), converted character string candidates ("minutes" and "skill test" in Figure 22), predicted character string candidates ("Settle the skill test" and "Destination for minutes" in Figure 22), operation command candidates ("Load minutes template" and "Save in minutes folder" in Figure 22), each selection probability, and estimated writing direction. In addition, the handwriting input display control unit 23 rotates the selectable candidate display data (operation guide 500) counterclockwise using the angle information acquired in step S60, and sends the rotated selectable candidate display data (operation guide 500) to the display unit 22 for display.

[0282] S63: The handwriting input display control unit 23 also rotates the rectangular area display data (rectangular frame) of the handwritten object and the selected object (handwritten object rectangular area display 503 in FIG. 22) counterclockwise using the angle information acquired in step S60, and transmits it to the display unit 22 for display.

[0283] S64: The handwriting input display control unit 23 transmits a selectable candidate display deletion timer start signal to the candidate display timer control unit 24 to delete the selectable candidate display data after a certain time has elapsed since its display. The candidate display timer control unit 24 starts this timer.

[0284] Steps S65 to S70 are executed when the user erases the selectable candidate display displayed on the display unit 22 while the selectable candidate deletion timer is running, or when a change occurs in the handwritten object (i.e., when a stroke of the handwritten object is added, deleted, moved, deformed, or divided), or when no candidate is selected before the timer times out.

[0285] Furthermore, steps S65 to S67 are executed when the candidate display is erased or a change occurs in the handwritten object.

[0286] S65: The handwriting input unit 21 transmits to the handwriting input display control unit 23 a notice that the selectable candidate display has been deleted or that the handwritten object has been changed.

[0287] S66: The handwriting input display control unit 23 sends a command to stop the selectable candidate deletion timer. The candidate display timer control unit 24 stops the timer. This is because an operation has been performed on the handwritten object within a certain period of time, making the timer unnecessary.

[0288] S67: The handwriting input display control unit 23 stores the position information of the operation guide 500 in the handwriting recognition control unit 26 so that it can be used in the gesture determination for the gesture handwriting recognition in step S19. The position information is, for example, the coordinates of the upper left corner and the lower right corner or coordinates equivalent thereto. This allows the handwriting recognition control unit 26 to determine whether the straight line used to input the angle information is within the operation guide 500.

[0289] S69: The handwriting input display control unit 23 transmits a command to erase the selectable candidate display data to the display unit 22, thereby erasing the display.

[0290] S70: The handwriting input display control unit 23 erases the display by transmitting a command to erase the rectangular area display data of the handwritten object and the selected object to the display unit 22. Therefore, when the display of the operation command candidate is erased under conditions other than the selection of the operation command candidate, the handwritten object remains displayed as is.

[0291] S68: On the other hand, if the selectable candidate display is not erased or the handwritten object is not changed while the selectable candidate deletion timer is started (if the user does not perform a pen operation), the candidate display timer control unit 24 sends a timeout to the handwritten input display control unit 23.

[0292] Similarly, after the selectable candidate display deletion timer times out, the handwriting input display control unit 23 executes steps S69 and S70 because the selectable candidate display data and the rectangular area display data of the handwritten object and the selected object may be deleted after a certain period of time has elapsed.

[0293] If the user selects a selectable candidate while the selectable candidate deletion timer is in progress, steps S71 to S103 are executed.

[0294] S71: When the user selects a selectable candidate while the selectable candidate deletion timer is starting, the handwriting input unit 21 transmits the character string candidate or operation command candidate selection to the handwriting input display control unit 23.

[0295] S71-2: The handwriting input display control unit 23 transmits a selectable candidate display deletion timer stop command to the candidate display timer control unit 24. The candidate display timer control unit 24 stops this timer.

[0296] S72: Next, the handwriting input / display control unit 23 transmits a command to clear the stored data to the handwriting recognition control unit 26.

[0297] S73: The handwriting recognition control unit 26 sends a command to clear the stored data to the character string conversion control unit 28.

[0298] S74: The handwriting recognition control unit 26 transmits a command to clear the stored data to the predictive conversion control unit 30.

[0299] S75: The handwriting recognition control unit 26 sends a retained data clear command to the operation command recognition control unit 32. The handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 clear the data relating to character string candidates and operation command candidates that have been retained up to this point.

[0300] S76: Next, the handwriting input display control unit 23 transmits a command to erase the selectable candidate display data to the display unit 22, thereby erasing the display.

[0301] S77: The handwriting input display control unit 23 transmits a command to erase the rectangular area display data of the handwritten object and the selected object to the display unit 22, thereby erasing the display.

[0302] S78: The handwriting input display control unit 23 erases the display by transmitting to the display unit 22 a command to erase the handwritten object display data and the pen coordinate complement display data transmitted in step S6. This is because the handwritten object, etc., is no longer necessary since a character string candidate or an operation command candidate has been selected.

[0303] S79: The handwriting input display control unit 23 transmits a request to delete the handwritten object to the handwriting input storage unit 25.

[0304] If a character string candidate is selected, steps S80 to S82 are executed.

[0305] S80: When a character string candidate is selected, the handwriting input display control unit 23 transmits a character string object addition to the handwriting input saving unit 25.

[0306] S81: Furthermore, the handwriting input display control unit 23 transmits a character string object font acquisition to the handwriting input saving unit 25. The handwriting input saving unit 25 identifies the font associated with the PenId of the pen ID control data from the pen color selection button definition data, and also transmits an estimated character size of the handwritten object to the handwriting input display control unit 23.

[0307] S82: Next, the handwriting input display control unit 23 displays the character string object by transmitting character string object display data to the display unit 22 to be displayed in the same position as the handwritten object using the defined font received from the handwriting input storage unit 25. The handwriting input display control unit 23 identifies the line type and thickness from the color definition data based on ColorId of the pen ID control data, and displays the text (character string object) rotated using the angle information.

[0308] If a candidate for an operation command is selected, steps S83 to S101 are executed, and if there is a selected object, steps S83 to S85 are executed.

[0309] S83: When a candidate for an operation command for a selected object is selected (when a selected object exists), the handwriting input display control unit 23 erases the display by sending a command to erase the display data of the selected object to the display unit 22. This is to erase the original selected object once.

[0310] S84: Next, the handwriting input display control unit 23 transmits an operation command execution for the selected object to the handwriting input saving unit 25. The handwriting input saving unit 25 transmits display data of the newly selected object (display data after editing or modification) to the handwriting input display control unit 23.

[0311] S85: Next, the handwriting input display control unit 23 transmits the selected object display data to the display unit 22, thereby re-displaying the selected object after the operation command is executed.

[0312] S86: If “Register handwritten signature” in the operation command definition data 713 or “Change settings” in the operation command definition data 716 is specified, the handwritten input display control unit 23 adds the handwritten signature registration form 561 or the user-defined data change form to the handwritten input storage unit 25.

[0313] S87: If the operation command candidate “save file” or “print” is selected, the handwriting input display control unit 23 transmits a file transmission request to the file transmission / reception control unit 37.

[0314] S88: The file transmission / reception control unit 37 transmits to the handwritten input storage unit 25 a request to acquire the handwritten input storage data to be transmitted.

[0315] S89: The handwritten input storage unit 25 determines whether the destination is a color-compatible device using the MIB or the like. In the case of a color-compatible device, the handwriting input saving unit 25 transmits the handwriting input saving data converted to color to the file transmission / reception control unit 37. That is, the handwriting input saving unit 25 determines whether the handwriting input saving data is handwriting input-derived data based on the ColorId of the handwriting input saving data, obtains color information associated with the ColorId from the color definition data, and converts the handwriting input-derived data to this color information. Since data that is not handwriting input-derived data originally contains color information, the handwriting input saving unit 25 transmits the color information. It is sufficient for the handwriting input saving unit 25 to determine whether the data is handwriting input-derived data (handwritten data drawn on the display device 2) without using the ColorId. Furthermore, the information indicating whether the data is handwriting input-derived data may be a value indicating a device, indicating whether the data is handwritten input-derived data drawn on the display device 2. If the destination is a monochrome device, the handwriting input saving unit 25 may ignore the color information associated with the ColorId of the handwriting input saving data and simply send a sequence of coordinate points or text. The handwriting input saving unit 25 may also obtain color information associated with the ColorId from the color definition data and convert the handwriting input-derived data to this color information. However, the handwriting input saving unit 25 may also obtain color information associated with the ColorId from the color definition data and convert it into monochrome highlighting (line type and thickness) before sending. Since data that is not handwriting input-derived data originally contains color information, the handwriting input saving unit 25 sends this color information. When the destination is a black-and-white emphasis-compatible device, the handwriting input saving unit 25 sends only the handwriting input saved data, or the handwriting input saved data and color definition data to the file transmission / reception control unit 37. When the handwriting input saving unit 25 sends only the handwriting input saved data, the line type and thickness are determined according to the color definition data held by the destination black-and-white emphasis-compatible device. When the handwriting input saving unit 25 sends both the handwriting input saved data and color definition data, the line type and thickness are determined according to the color definition data held by the source black-and-white emphasis-compatible device. Note that the source black-and-white emphasis-compatible device may convert the data to black-and-white emphasis display before the handwriting input saving unit 25 sends it. Furthermore, when the file transmission / reception control unit 37 writes data to a file, such as a PDF file, it obtains color information associated with ColorId from the color definition data and sets the handwriting-input-derived data in the PDF file according to the format of the PDF file. The same applies to fonts, character sizes, etc. The file transmission / reception control unit 37 also saves all or part of the saved handwriting-input data (e.g., ColorId, FontName, etc.) in the metadata of the PDF file. Because PDF file readers ignore metadata, there is no effect on the display, and when the display device 2 reads this PDF file, the saved handwriting-input data can be reproduced from the metadata. In the case of data that is not handwriting-input-derived data, the file transmission / reception control unit 37 sets the color information, font, character size, etc. contained in the saved handwriting-input data in the PDF file according to the format of the PDF file.

[0316] S90: The file transmission / reception control unit 37 transmits the handwritten input data received from the handwritten input storage unit to the destination or writes it to a file.

[0317] S91: When the operation command candidate “read file” is selected, the handwriting input display control unit 23 transmits a file list information acquisition request to the file transmission / reception control unit 37.

[0318] S92: The file transmission / reception control unit 37 receives file list information from a storage medium such as a USB memory, a network storage, or a Web server, or from an external device.

[0319] S93: The file transmission / reception control unit 37 transmits the file list information to the handwriting input display control unit 23.

[0320] S94: The handwriting input display control unit 23 transmits file list display data to the display unit 22. As a result, the display unit 22 displays a file list on the display as shown in FIG.

[0321] S95: When the user selects a file and the handwriting input unit 21 accepts it, the handwriting input unit 21 transmits the file selection to the handwriting input display control unit 23.

[0322] S96: The handwriting input display control unit 23 transmits a file reception request for the selected file to the file transmission / reception control unit 37.

[0323] S97: The file transmission / reception control unit 37 acquires the file from the external device.

[0324] S98: The file transmission / reception control unit 37 stores the file in the handwritten input storage unit 25.

[0325] S99: The handwriting input saving unit 25 analyzes the file received from the file transmission / reception control unit 37 and converts the handwriting input-derived data into handwriting input saved data (black and white highlighting / color convertible data). That is, the handwriting input saving unit 25 determines whether metadata is present. If metadata is present, the handwriting input saving unit 25 determines whether conversion to handwriting input saved data is possible (whether a Color ID, etc. is present), and saves the data as handwriting input saved data. The handwriting input saving unit 25 then reads the Color ID of the handwriting input-derived data, refers to the color definition data, and converts the data to black and white highlighting associated with the Color ID. The handwriting input saving unit 25 transmits display data of the handwritten object to the handwriting input display control unit 23. If the data is not handwriting input-derived data, the handwriting input saving unit 25 reads color information, font, character size, etc. according to the file format and saves it as handwriting input saved data.

[0326] S100: The handwriting input display control unit 23 displays the display data of the handwriting input data on the display unit 22. As a result, the display unit 22 displays the handwriting input-derived data as handwritten data with black and white emphasis, and displays other data in black and white using conventional brightness conversion.

[0327] When the operation command 512 for signing in is executed, the handwriting input display control unit 23 acquires the pen ID received by the display device 2 when the operation command 512 is executed. The handwriting input display control unit 23 identifies the user-defined data having the Signature ID acquired in step S61 and acquires the Account ID from the user-defined data. The handwriting input display control unit 23 then registers the Account ID in the pen ID control data in association with the pen ID. This associates the pen 2500 with the user, and the display device 2 becomes able to perform processing using the user-defined data.

[0328] When the user writes by hand or loads a file after signing in, the handwriting input display control unit 23 obtains from the pen ID control data the Account Id associated with the pen ID received by the display device 2 when the operation command is executed. The handwriting input display control unit 23 identifies the user-defined data by this Account Id, and sets color definition data, etc. in % to % of the operation command and executes it.

[0329] S101: When an input / output operation command is selected, the handwriting input display control unit 23 executes the operation command character string (Command) of the operation command definition data corresponding to the operation command selected by the user.

[0330] S102: If the user manually inputs angle information, the handwriting input display control unit 23 stores the received angle information in the pen ID control data storage unit 36 ​​in association with the pen ID received from the pen 2500 when the rotation operation button 511 is pressed.

[0331] S103: For the next handwritten object, the handwritten input display control unit 23 transmits a handwritten object start signal to the handwritten input saving unit 25. The handwritten input saving unit 25 reserves a handwritten object area. After this, the processes of steps S2 to S103 are repeated.

[0332] <Summary> As described above, the display device 2 of this embodiment associates handwriting-input-derived data with a notation to enable it to detect that the data is handwriting-input-derived data. As a result, even when the display device 2 acquires a file from an external device, when displaying the handwriting-input-derived data, the display device 2 can highlight the data in black and white based on the color information associated with the handwriting data. Furthermore, when the display device 2 transmits handwriting-input-derived data to an external device, the display device 2 can display the data without determining whether the data is handwriting-input-derived data. [Example]

[0333] In this embodiment, a display device 2 that converts handwritten data into English will be described. The configuration of the display device 2 is the same as in the first embodiment except that the conversion dictionary and operation command definition data correspond to English. Therefore, differences from the first embodiment when the display device 2 converts handwritten data into English (hereinafter referred to as English conversion) will be described.

[0334] 38 is a diagram for explaining an outline of a method for displaying handwriting input-derived data by the display device 2 of this embodiment in the case of English conversion. Note that the explanation of FIG. 38 will mainly focus on the differences from FIG. 2.

[0335] (1) First, the user can select the color to input the handwritten data in using the buttons. In Figure 2, the user can select the colors Bk, R, B, G, M, and C.

[0336] (2) When the user writes, for example, "hand-written," the display device 2 displays the handwritten data in a manner that corresponds to the color. A manner that corresponds to the color means that different line types are displayed depending on the color. By displaying in line types that correspond to the color in this way, black and white highlighting becomes possible. The display device 2 can convert the handwritten data into text (character code). Handwritten data 4 and 5 in Figure 2 have been converted to text, and handwritten data 6 is handwritten data that has not been converted.

[0337] (3) The display device 2 of this embodiment stores color information and information indicating whether each piece of handwritten input data is handwriting-input-derived data. Handwritten data 4 is handwriting-input-derived data with color information Bk, handwritten data 5 is handwriting-input-derived data with color information R, and handwritten data 6 is handwriting-input-derived data with color information B. Since the display device 2 also stores information indicating whether text (character codes) converted from handwritten data, such as handwritten data 4 and 5, is handwriting-input-derived data, it can display the handwriting-input-derived data in a line style corresponding to the specified color.

[0338] (4) When the display device 2 converts handwritten data into a PDF file or the like, the display device 2 sets the color information of the handwritten data or text data according to the PDF format, and attaches the fact that it is data resulting from handwritten input and the color information as metadata. When the display device 2 reads this PDF file from a storage medium (such as a USB memory), the data resulting from handwritten input can be displayed in a line type corresponding to the color information based on the metadata.

[0339] Thus, when the display device 2 of this embodiment displays data resulting from handwritten input even when the user writes in English, it can perform black-and-white highlighting based on the color information associated with the handwritten data.

[0340] Note that FIG. 39 is a diagram for explaining an outline of a method for displaying data resulting from handwritten input by the display device 2 of this embodiment in the case of Chinese conversion. "手写的" means "handwritten". Thus, it can be similarly converted in the case of Chinese. Also, the display device 2 of this embodiment is not language-dependent and can similarly handle other languages.

[0341] <An example of operation command definition data> FIG. 40 is an example of operation command definition data when there is no selection object. In the description of FIG. 40, mainly the differences from FIG. 11 will be explained. The content of each operation command is the same as that in FIG. 11, but English expressions are associated with Name and string. Therefore, the user can write operation commands in English and select English operation commands.

[0342] FIG. 41 is an example of operation command definition data when there is a selection object. In the description of FIG. 41, mainly the differences from FIG. 13 will be explained. The content of each operation command is the same as that in FIG. 13, but an English expression is associated with Name. Therefore, the user can select an English operation command.

[0343] Figure 42 is an example of user-defined data. The explanation of Figure 42 will mainly focus on the differences from Figure 14. The content of the user-defined data is the same as in Figure 14, but the AccountUsername is set to "Bob". Also, an English expression is associated with the Name. Therefore, the user can select user-defined data in English.

[0344] The handwritten signature data stored in the handwritten signature data storage unit 39 may be the same as that shown in Fig. 15, and the handwritten input storage data may be the same as that shown in Fig. 16. The pen ID control data may be the same as that shown in Fig. 17, and the color definition data may be the same as that shown in Fig. 18.

[0345] <Example of dictionary data> Dictionary data for English conversion will be described using Figures 43 to 45. In the explanation of Figures 43 to 45, differences from Figures 8 to 10 will be mainly explained. Figure 43 is an example of dictionary data in handwriting recognition dictionary unit 27. The dictionary data in handwriting recognition dictionary unit 27 in Figure 43 indicates that handwritten "a (stroke data state)" is converted to "a" with a probability of 0.90 and to "o" with a probability of 0.10.

[0346] Fig. 44 is an example of dictionary data in the character string conversion dictionary unit 29. The dictionary data in the character string conversion dictionary unit 29 in Fig. 44 indicates that the character string "a" is converted to "ab" with a probability of 0.55, and the character string "a" is converted to "AI" with a probability of 0.45. The same applies to other "before conversion" character strings.

[0347] Fig. 45 is an example of dictionary data of the predictive conversion dictionary unit 31. The dictionary data of the predictive conversion dictionary unit 31 in Fig. 45 indicates that the character string "agenda" is converted to "agenda list" with a probability of 0.55 and to "agenda template" with a probability of 0.30.

[0348] The dictionary data is not language-dependent, and any character strings may be registered before and after conversion.

[0349] <Data displayed in black and white and color> Figure 46 shows data displayed in black and white highlighting and in color display mode using the color definition data of Figure 18. The explanation of Figure 46 will mainly focus on the differences from Figure 19.

[0350] Code C21 is the text "Bk", "R", "B", "G", "magenta", and "cyan", which is data originating from handwriting input. Code C22 is a handwritten stroke of "Bk", "R", "B", "G", "magenta", and "cyan". Code C23 is a spring-like, single-stroke handwritten stroke. Code C24 is a donut-shaped color chart (image). Code C25 is the RGBCMY inside the color chart (text that is not data originating from handwriting input). Below code C23 is the black text "Transparent", which is data originating from handwriting input.

[0351] Therefore, the Japanese words black, red, blue, green, magenta, and cyan are "Bk," "R," "B," "G," "magenta," and "cyan," but the rest are the same as in Figure 19. The same is true for Figure 46(b).

[0352] As can be seen by comparing Figures 46(a) and (b), the red text "R" inside the color chart of code C30 in Figure 46(b) is not data originating from handwriting input. For this reason, the text "R" is displayed in grayscale in Figure 46(a). In contrast, the text "R" of code C26 is data originating from handwriting input, so it is displayed in highlighted black and white in Figure 46(a).

[0353] Also, because magenta and cyan are transparent colors, "Transparent" appears transparent in Figure 46(b). In Figure 46(a), magenta and cyan also appear transparent because they use a transparent white pattern. In this way, a device that supports black-and-white emphasis can highlight colors to make them stand out, and on a color device, they can be converted to color and displayed.

[0354] <Example of selectable candidates> Figure 47 shows an example of an operation guide 500 for English conversion and selectable candidates 530 displayed by the operation guide 500. The explanation of Figure 47 will mainly focus on the differences from Figure 22. In Figure 47, the user has handwritten "a" as handwritten object 504. Based on "a", operation command candidates 510, handwritten recognition character string candidates 506, conversion character string candidates 507, and character string / predictive conversion candidates 508 are displayed. Therefore, the same as Figure 22 can be displayed except that the Japanese in Figure 22 is changed to English.

[0355] The operation command candidate 510 is, for example, an operation command candidate having "agenda" in the string in the operation command definition data of FIG. 40(a) (operation command definition data 701, 702).

[0356] In this way, the user can have the operation guide 500 displayed in the same way in the case of English conversion.

[0357] <Example of specifying a selected object> 48 is an example of a diagram for explaining an example of designating a selection object in the case of English conversion. Note that the explanation of FIG. 48 will mainly focus on the differences from FIG.

[0358] 48(a) shows an example in which the user has specified two horizontally written determined objects 13a2 and 13b2 with a crossover line (hand-drawn object 11a2). In this example, the length H1 of the short side and the length W1 of the long side of the hand-drawn object rectangular area 12a2 satisfy the crossover line determination condition 406, and the overlap rate with the determined objects 13a2 and 13b2 also satisfies the crossover line determination condition 406. Therefore, both the determined objects 13a2 and 13b2 for "agenda" and "ag" are specified as selected objects.

[0359] 48(b) shows an example in which the user specifies a horizontally written fixed object 13c2 with an encircling line (hand-drawn object 11b2). In this example, only the fixed object 13c2 called "agenda" is specified as the selected object because the overlap rate between the fixed object 13c2 and the hand-drawn object rectangular area 12c2 satisfies the encircling line determination condition 407.

[0360] In this way, in the case of English conversion, the user can select the selected object in the same way.

[0361] <Example of displaying operation command candidates> Fig. 49 shows an example of displaying operation command candidates based on operation command definition data when there is a handwritten object as shown in Fig. 48. In the explanation of Fig. 49, differences from Fig. 24 will be mainly explained.

[0362] Figure 49(a) shows candidate edit-related operation commands, and Figure 49(b) shows candidate modification-related operation commands. Figure 49(a) also shows an example in which a selected object is specified using the handwritten object 11a2 in Figure 48(a). As shown in Figures 49(a) and (b), the candidate operation commands displayed following the first character "»" 511 is a main menu 550.

[0363] In FIG. 49(a), pressing ">" 512a on the first line displays submenu 560 in FIG. 49(a). When the user presses any of the operation command names with the pen, the handwriting input display control unit 23 executes the Command of the operation command definition data associated with the operation command name on the selected object. That is, when "Delete" 521b is selected, the handwriting input display control unit 23 executes "Delete." When "Move" 522b is selected, the handwriting input display control unit 23 executes "Move." When "Rotate" 523b is selected, the handwriting input display control unit 23 executes "Rotate." When "Select" 524b is selected, the handwriting input display control unit 23 executes "Select."

[0364] When the user presses "Delete" 511b with the pen, the handwriting input display control unit 23 deletes the confirmed objects 13a2 and 13b2, "agenda" and "ag." When the user presses "Move" 522b with the pen, the handwriting input display control unit 23 accepts the movement of the confirmed objects 13a2 and 13b2, "agenda" and "ag." When the user presses "Rotate" 523b with the pen, the handwriting input display control unit 23 rotates the confirmed objects 13a2 and 13b2, "agenda" and "ag," by a certain angle. When the user presses "Select" 524b with the pen, the handwriting input display control unit 23 accepts the selection of the confirmed objects 13a2 and 13b2, "agenda" and "ag."

[0365] The character string candidates other than the operation command candidates, "-" 541b, "-" 542b, "~" 543b, "→" 544b, and "⇒" 545b, are the recognition results of the crossing line (handwritten object 11a2), and if the user intends to input a character string instead of an operation command, the user can select a character string candidate.

[0366] In FIG. 49(b), pressing ">" 512b on the second line displays the submenu of FIG. 49(b). In the display example of FIG. 49(b), a main menu 550 and a submenu 560 are displayed, similar to FIG. 49(a). Based on the operation command definition data of FIG. 41, when "Thick" 531b is selected, the handwriting input display control unit 23 executes "Thick" on the selected object. When "Thin" 532b is selected, the handwriting input display control unit 23 executes "Thin" on the selected object. When "Large" 533b is selected, the handwriting input display control unit 23 executes "Large" on the selected object. When "Small" 534b is selected, the handwriting input display control unit 23 executes "Small" on the selected object. When "Underline" 535b is selected, the handwriting input display control unit 23 executes "Underline" on the selected object.

[0367] When the user presses "Thick" 531b with the pen, the handwriting input display control unit 23 thickens the lines that make up the determined objects 13a2, 13b2, "agenda" and "ag." When the user presses "Thin" 532b with the pen, the handwriting input display control unit 23 thins the lines that make up "agenda" and "ag." When the user presses "Large" 533b with the pen, the handwriting input display control unit 23 can increase the size. When the user presses "Small" 534b with the pen, the handwriting input display control unit 23 can decrease the size. When the user presses "Underline" 535b with the pen, the handwriting input display control unit 23 can add an underline.

[0368] In this way, the user can display operation commands when there is a handwritten object even in the case of English conversion.

[0369] <Example of angle information input> Next, Figure 50 is an example of a diagram explaining how to input 90-degree angle information. The method for inputting angle information in the case of English conversion is the same as that in the case of Japanese conversion. The explanation of Figure 50 will mainly focus on the differences from Figure 26.

[0370] 50(a) shows a state in which the angle information of the pen ID control data is 0 degrees (initial value), and the user standing at the 3 o'clock position of the display device 2 has handwritten the letter "a," resulting in the display of the operation guide 500. The display device 2 recognizes the handwritten "a" from the 3 o'clock position as a character while the angle information remains at 0 degrees, and therefore displays a selectable candidate 530 that is different from the expected one.

[0371] When inputting angle information, the user handwrites a straight line from top to bottom as seen from the user's perspective within operation guide 500. Figure 50(b) shows an example of this straight line 521b. The angle information is the angle α formed by the counterclockwise direction between this line 521b and the 6 o'clock direction, which has angle information of 0 degrees.

[0372] Figure 50(c) shows the operation guide 500 immediately after the operation in Figure 50(b). Since 90 degrees is set as the angle information (Angle) in the pen ID control data, the handwritten object (stroke data) is internally rotated 90 degrees clockwise for handwriting recognition, and the operation guide 500 is displayed rotated 90 degrees counterclockwise. Note that the user may also manually input the angle information from a menu.

[0373] In this way, users can input angle information even when converting to English.

[0374] <Example of handwritten signature data registration> Next, Figure 51 is a diagram explaining a method for registering handwritten signature data. The explanation of Figure 51 will mainly focus on the differences from Figure 27. First, Figure 51(a) is an example of selectable candidates 530 that are displayed when the user handwrites "Sign." Two operation commands 513b, 514b, "Register handwritten signature" and "Hand sign out," based on operation command definition data 713, 715, which partially match the character string "Sign," as well as character string candidates "Sign," "Signature session," and "Signed" are displayed.

[0375] When the user presses "Register handwritten signature" with the pen 2500, a handwritten signature registration form 561 shown in Fig. 51(b) is added to the handwritten input storage unit 25 and displayed on the operation screen 101. For example, the operation guide 500 shown in Fig. 51(a) is erased, and the handwritten signature registration form 561 is displayed in the same location as the operation guide 500.

[0376] The user inputs the text of the name in the name input field 561a, the first, second and third handwritten signatures in the signature input fields 561b to 561d, and a check mark or a cancel mark in the registration confirmation field 561e.

[0377] When the user writes his / her handwriting on the handwritten signature registration form 561 as instructed, the handwritten signature registration form 561 shown in FIG. 51(c) appears.

[0378] In this way, the handwritten input display control unit 23 can accept handwritten input without distinguishing between handwritten input into a form and handwritten input outside of a form. The user can register handwritten signature data in English.

[0379] <Example of handwritten sign-in> 52 is an example of an operation guide 500 that is displayed when the user handwrites "Bob," which is registered handwritten signature data, in English conversion. The explanation of FIG. 52 will mainly focus on the differences from FIG. 28.

[0380] "Bob" matches the handwritten signature data because it is registered as handwritten signature data in handwritten signature data storage unit 39. Therefore, operation command 512 "Hand Sign in" is displayed.

[0381] Furthermore, since the handwritten signature data matches, the SignatureId representing "Bob" is identified, and the user-defined data having the AccountId associated with this SignatureId is identified.

[0382] When the user selects the operation command 512 "Hand Sign in," the AccountId of "Bob" is associated with the pen ID of the pen 2500 being used and added to the pen ID control data, and the user-defined data of "Bob" is used when the operation command is used.

[0383] In this way, users can sign in using English.

[0384] <Example of changing user-defined data> Figure 53 is a diagram explaining how to change user-defined data in English conversion. The explanation of Figure 53 will mainly focus on the differences from Figure 29. Figure 53(a) is an example of operation guide 500 that is displayed when the user writes "set" by hand. "set" is defined as a String in the operation command definition data 716, and the operation command "Change settings" is displayed.

[0385] When a user selects "Change settings" using the pen 2500 with which they have signed in by handwriting, the pen ID control data identifies the AccountId of the user associated with the pen ID of this pen 2500. This identifies the user-defined data of the signed-in user, and a user-defined data change form 562 in FIG. 53(b) is added to the handwriting input storage unit 25 and displayed on the operation screen 101. The items in the user-defined data change form 562 are the same as those in FIG. 29(b).

[0386] In this way, the user can change the user definition for English translation just as they can for Japanese translation.

[0387] The operation procedure may be the same as that shown in FIGS. 30 to 37 in the first embodiment. [Example]

[0388] In the first embodiment, the display device 2 is described as having a large touch panel, but the display device is not limited to having a touch panel. In this embodiment, a projector-type display device will be described.

[0389] <<Another configuration example 1 of the display device>> Fig. 54 is a diagram showing another example of the configuration of a display device. In Fig. 54, a projector 411 is installed on the upper side of a regular whiteboard 413. This projector 411 corresponds to the display device. Regular whiteboard 413 is not a flat panel display integrated with a touch panel, but a whiteboard on which a user directly writes by hand with a marker. Note that the whiteboard may also be a blackboard, as long as it is flat and large enough to project an image.

[0390] The projector 411 has an ultra-short focus optical system and can project an image with little distortion from a distance of about 10 cm onto the whiteboard 413. This image may be transmitted from the PC 400-1 connected wirelessly or by wire, or may be stored in the projector 411.

[0391] The user writes by hand on the whiteboard 413 using a dedicated electronic pen 2501. The electronic pen 2501 has a light-emitting part, for example at its tip, that is switched on and emits light when the user presses it against the whiteboard 413 to write. The light has a wavelength of near-infrared or infrared, so it is invisible to the user's eyes. The projector 411 has a camera that captures an image of the light-emitting part and analyzes the image to identify the direction of the electronic pen 2501. The electronic pen 2501 also emits sound waves along with the light emission, and the projector 411 calculates the distance based on the time it takes for the sound waves to reach the screen. The projector 411 can identify the position of the electronic pen 2501 based on the direction and distance. A stroke is drawn (projected) at the position of the electronic pen 2501.

[0392] The projector 411 projects the menu 430, so when the user presses a button with the electronic pen 2501, the projector 411 identifies the pressed button based on the position of the electronic pen 2501 and the ON signal of the switch. For example, when the save button 431 is pressed, the strokes (a set of coordinates) handwritten by the user are saved in the projector 411. The projector 411 saves the handwritten information in a predetermined server 412, USB memory 2600, or the like. The handwritten information is saved for each page. Since the coordinates are saved as they are, rather than as image data, the user can re-edit the information. However, in this embodiment, the menu 430 does not need to be displayed because operation commands can be called up by handwriting.

[0393] <<Another configuration example 2 of the display device>> 55 is a diagram showing another example of the configuration of the display device 2. In the example of FIG.

[0394] The terminal device 600 is connected by wire to the image projection device 700A and the pen motion detection device 810. The image projection device 700A projects image data input by the terminal device 600 onto the screen 800.

[0395] The pen operation detection device 810 communicates with the electronic pen 820 and detects the operation of the electronic pen 820 in the vicinity of the screen 800. Specifically, the electronic pen 820 detects coordinate information indicating a point indicated by the electronic pen 820 on the screen 800 and transmits the coordinate information to the terminal device 600.

[0396] The terminal device 600 generates image data of a stroke image input by the electronic pen 820 based on the coordinate information received from the pen operation detection device 810. The terminal device 600 causes the image projection device 700A to draw the stroke image on the screen 800.

[0397] Furthermore, the terminal device 600 generates superimposed image data that indicates a superimposed image obtained by combining the background image projected by the image projection device 700A and the stroke image input by the electronic pen 820.

[0398] <<Another Configuration Example 3 of the Display Device>> 56 is a diagram showing an example of the configuration of a display device 2. In the example of Fig. 56, the display device 2 includes a terminal device 600, a display 800A, and a pen motion detection device 810.

[0399] The pen motion detection device 810 is placed near the display 800A. The pen motion detection device 810 detects coordinate information indicating a point indicated by the electronic pen 820A on the display 800A, and transmits the coordinate information to the terminal device 600. In the example of Fig. 56, the electronic pen 820A may be charged by the terminal device 600 via a USB connector.

[0400] The terminal device 600 generates image data of a stroke image input by the electronic pen 820A based on the coordinate information received from the pen operation detection device 810. The terminal device 600 displays the stroke image on the display 800A.

[0401] <<Another Configuration Example 4 of the Display Device>> Fig. 57 is a diagram showing a configuration example of a display device 2. In the example of Fig. 57, the display device 2 includes a terminal device 600 and an image projection device 700A.

[0402] The terminal device 600 performs wireless communication (Bluetooth (registered trademark) or the like) with the electronic pen 820B and receives coordinate information of a point indicated by the electronic pen 820B on the screen 800. Then, based on the received coordinate information, the terminal device 600 generates image data of a stroke image input by the electronic pen 820B. The terminal device 600 causes the image projection device 700A to project the stroke image.

[0403] Furthermore, the terminal device 600 generates superimposed image data that indicates a superimposed image obtained by combining the background image projected by the image projection device 700A and the stroke image input by the electronic pen 820.

[0404] As described above, the above-described embodiments can be applied to various system configurations.

[0405] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0406] For example, the display method of the present embodiment can be suitably applied to any information processing device having a touch panel. Furthermore, devices having the same functions as a display device are also referred to as electronic blackboards, electronic whiteboards, electronic information boards, interactive boards, etc. Examples of information processing devices equipped with a touch panel include output devices such as PJs (Projectors), digital signage, HUDs (Head Up Display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, network appliances, notebook PCs (Personal Computers), mobile phones, smartphones, tablet devices, game consoles, PDAs (Personal Digital Assistants), digital cameras, wearable PCs, and desktop PCs.

[0407] In this embodiment, the server may perform part of the processing performed by the display device 2. For example, the display device transmits stroke information to the server, and acquires and displays information to be displayed on the operation guide 500 from the server.

[0408] In this embodiment, the display device 2 detects the coordinates of the pen by detecting the coordinates of the pen tip with a touch panel, but the coordinates of the pen tip may also be detected by ultrasonic waves. The pen emits light and also emits ultrasonic waves, and the display device 2 calculates the distance based on the arrival time of the ultrasonic waves. The display device 2 can identify the position of the pen based on the direction and distance. The display device 2 draws (projects) the trajectory of the pen as a stroke.

[0409] In this embodiment, candidates for editing and modification operation commands are displayed when there is a selected object, and candidates for input / output operation commands are displayed when there is no selected object. However, the display device 2 may simultaneously display candidates for editing and modification operation commands and candidates for input / output operation commands.

[0410] Furthermore, the user's handwritten signature data does not have to be stored in the display device 2. It may be stored on the cloud or in an in-house information processing device.

[0411] In addition, the configuration examples in Figure 6 and the like are divided according to main functions to make it easier to understand the processing by the display device 2. The method of dividing the processing units and their names do not limit the present invention. The processing by the display device 2 can be divided into even more processing units depending on the processing content. Also, it can be divided so that one processing unit includes even more processes.

[0412] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions. [Explanation of symbols]

[0413] 2 Display device [Prior art documents] [Patent documents]

[0414] [Non-Patent Document 1] Japanese Patent Application Laid-Open No. 2017-139678

Claims

1. A display device for displaying data, A display device characterized in that it records, in association with the handwritten data, information indicating that the data is handwritten on the display device.

2. 2. The display device according to claim 1, wherein when the data to be displayed is stroke data handwritten on the display device or data resulting from handwritten input, which is text data converted from the stroke data by recognition, the data is displayed with color information contained in the data emphasized in black and white.

3. 3. The display device according to claim 2, wherein the display device is an electronic paper that displays an image in black or white.

4. a handwriting recognition control unit that performs handwriting recognition on handwritten data based on the position of the input means in contact with the touch panel and converts the data into text data; a handwriting input unit that accepts a color selection; 4. The display device according to claim 2, wherein when the handwriting recognition control unit displays the converted text data, the color information received by the handwriting input unit is highlighted in black and white.

5. the handwriting input unit accepts a font designation; 5. The display device according to claim 4, wherein when the handwriting recognition control unit displays the converted text data, the text data is displayed in the font accepted by the handwriting input unit with the color information emphasized in black and white.

6. The system stores definition data in which color information is associated with user identification information, The display device according to claim 4 or 5, characterized in that when a user signs in, the display device displays handwritten data in which the color information registered in the definition data is highlighted in black and white, or the text data converted from the handwritten data.

7. When handwritten data written on the touch panel or text data converted by the handwriting recognition control unit is saved in a file, 7. The display device according to claim 4, wherein the color information received by the handwriting input unit is converted into color information conforming to a file format and saved.

8. When handwritten data written on the touch panel or text data converted by the handwriting recognition control unit is saved in a file, 8. The display device according to claim 7, wherein the color information received by the handwriting input unit is converted into color information conforming to a file format and saved, and the color information received by the handwriting input unit is attached to metadata of the file.

9. A display device according to any one of claims 4 to 8, characterized in that when the handwriting input-derived data is transmitted to a device that outputs the data in which color information is emphasized in black and white, the handwriting input-derived data is transmitted.

10. A display device according to any one of claims 4 to 9, characterized in that when the handwriting input-derived data is transmitted to a color-compatible device, the color of the handwriting input-derived data is converted to the color information accepted by the handwriting input unit and transmitted.

11. A display device according to any one of claims 4 to 9, characterized in that when the handwriting input-derived data is transmitted to a monochrome compatible device, the color of the handwriting input-derived data is converted to the color information accepted by the handwriting input unit, or the handwriting input-derived data is transmitted in monochrome.

12. 9. The display device according to claim 7, wherein, when the data contained in the read file is the handwritten input-derived data, the data is displayed with the color information contained in the metadata attached to the file highlighted in black and white.

13. 13. The display device according to claim 12, wherein the handwritten input originating data is text data.

14. 13. The display device according to claim 12, wherein, when the data included in the read file is not the handwritten input-derived data, color information contained in the data is converted into grayscale and the data is displayed.

15. A color-capable device that displays a file stored by the display device according to any one of claims 1 to 14 or a file transmitted from the display device, A color-capable device that displays the handwritten input-derived data in color.

16. A display method performed by a display device that displays data, A display method comprising: recording, in association with the handwritten data, a message indicating that the data is handwritten on the display device.

17. On the display device that displays the data, A program for recording, in association with the handwritten data, that the data is handwritten data drawn on the display device.

Images