Display device, color correspondence device, display method, program

Abstract

To provide a display device capable of identifying handwritten data drawn on a specific display device.SOLUTION: A display device for displaying data is provided, the display device being configured to record information showing that handwritten data was drawn on the display device in association with the handwritten data.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a display device, a color corresponding device, a display method, and a program.

Background Art

[0002] A display device that displays data handwritten on a touch panel with a pen or a finger is known. A display device equipped with a relatively large touch panel is placed in a conference room or the like and used as an electronic blackboard or the like by a plurality of users.

[0003] In a general display device, a user can switch the color, thickness, etc. of characters with a predetermined button or the like, and the display device can display handwritten data in the color and thickness selected by the user.

[0004] On the other hand, there have been attempts to adopt electronic paper or the like for a display device, but general electronic paper cannot display in color. For this reason, when a black-and-white corresponding device such as electronic paper reads a color file from the outside, the black-and-white corresponding device converts the color information into black-and-white density and displays it (see, for example, Patent Document 1). Patent Document 1 discloses a conversion method for performing appropriate correction according to the characteristics of a color image and performing image data conversion.

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the conventional technology has a problem that it is impossible to identify handwritten data drawn on a specific display device.

[0006] In view of the above problems, an object of the present invention is to provide a display device capable of identifying handwritten data drawn on a specific display device.

Means for Solving the Problems

[0007] In view of the above problems, the present invention It has a display unit and is handwritten data On the display unitrepresentation To cause a display device, A display control unit that displays the handwritten data in black and white in a manner corresponding to the color information indicating the color selected from among a plurality of colors, and that displays, on the display unit, buttons that can accept the selection of one color from among the plurality of colors so as to display the handwritten data in black and white in different manners for each of the plurality of colors; and a handwriting input unit that accepts a handwriting input and converts the accepted handwriting input into the handwritten data. The display control unit displays the handwritten data in black and white on the display unit in a manner corresponding to the color information of the color selected by the button associates with the handwritten data To save and A storage unit Based on a user's request, a transmission unit that transmits the stored handwritten data and the data based on the color information to an external device is characterized by this.

Effect of the Invention

[0008] It is possible to provide a display device that can identify handwritten data drawn on a specific display device.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

Figure 44

Figure 45

Figure 46

Figure 47

Figure 48

Figure 49

Figure 50

Figure 51

Figure 52

Figure 53

Figure 54

Figure 55

Figure 56

Figure 57

Embodiments for Carrying Out the Invention

[0010] Hereinafter, as an example of an embodiment for implementing the present invention, a display device and a display method performed by the display device will be described with reference to the drawings.

Example

[0011] <Regarding Terms> The input means may be any means that can specify coordinates on the touch panel and enable handwriting. For example, there are a pen, a person's finger or hand, a rod-shaped member, etc. Also, eye gaze input may be possible. Handwriting data is data obtained by displaying, as a trajectory, a sequence of coordinate points where the user continuously moves the input means on the touch panel. Also, a series of operations where the user presses the input means against the touch panel and then continuously moves it and then releases it from the touch panel is called a stroke. The data written by a stroke is called stroke data. Handwriting data has one or more stroke data. Handwriting input indicates that handwriting data is input by the user.

[0012] <Regarding a Comparative Example of the Display Device> In explaining the display device of the present embodiment, first, a color display example of a black-and-white corresponding device will be briefly described.

[0013] FIG. 1 shows an example of conversion from color to black-and-white displayed by a black-and-white corresponding device. First, FIG. 1(a) shows an example of handwriting data displayed by a black-and-white corresponding device. The black-and-white corresponding device is a display device that cannot perform color display. Even for the black-and-white corresponding device, the user can select a color and perform handwriting input, but the black-and-white corresponding device converts the color information to grayscale and displays it.

[0014] FIGS. 1(b) and (c) are diagrams showing the correspondence between color and grayscale. Due to the limitations of drawing, both are in black-and-white, but FIG. 1(b) is a color chart and FIG. 1(c) is a grayscale-converted color chart. Comparing the colors and grayscales at the same positions in the circumferential direction of the color charts in FIGS. 1(b) and (c), it can be seen that red (R) is converted to a light gray and blue (B) is converted to a dark gray.

[0015] Therefore, as shown in Fig. 1(a), when the black-and-white conversion device converts the handwritten data 303 written in red by the user into grayscale, it is displayed in a light gray. When the black-and-white conversion device converts the handwritten data 302 written in blue by the user into grayscale, it is displayed in a dark gray. The handwritten data 301 written in black by the user is displayed in black. However, it can be seen that it is difficult for red and blue to fulfill the function of making the original information prominent. In particular, it is difficult to distinguish blue from black.

[0016] <Overview of color display of the display device in this embodiment> Therefore, the display device of this embodiment associates that fact (information indicating that it is handwritten input-derived data) with the handwritten input-derived data so that it can detect that it is handwritten input-derived data. Also, when the display device displays handwritten input-derived data, it displays the handwritten data in a manner corresponding to the color information according to the color information associated with the handwritten data. Displaying the handwritten data in a manner corresponding to the color information is called "black-and-white emphasized display". The handwritten input-derived data is stroke data input by handwriting or text data obtained by converting this stroke data by handwriting recognition.

[0017] Fig. 2 is a diagram for schematically explaining the display method of handwritten input-derived data by the display device of this embodiment. Note that the display device will be described as not supporting color display, such as an electronic paper. However, a display device capable of color display may perform black-and-white display.

[0018] (1) First, the user can select in what color to input the handwritten data from the button. In Fig. 2, the user can select each color of Bk, R, B, G, M, and C.

[0019] (2) When the user writes by hand, the display device displays the handwritten data in a manner corresponding to the color. The manner corresponding to the color means displaying with different line types or the like depending on the color. By displaying with line types corresponding to the color in this way, black-and-white emphasized display becomes possible. The display device supports handwriting recognition as described later and can convert handwritten data into text (character code). The handwritten data 1 and 2 in FIG. 2 have been converted into text, and the handwritten data 3 is handwritten data that has not been converted.

[0020] (3) The display device of this embodiment has information on whether the data is handwriting input-derived data and color information for each handwritten data. The handwritten data 1 is handwriting input-derived data and the color information is black, the handwritten data 2 is handwriting input-derived data and the color information is red, and the handwritten data 3 is handwriting input-derived data and the color information is blue. Since the text (character code) converted from the handwritten data like the handwritten data 1 and 2 also holds information on whether it is handwriting input-derived data, the display device can display the handwriting input-derived data in a line type corresponding to the specified color when displaying it.

[0021] (4) When the display device converts the handwritten data into a PDF file or the like, the display device sets the color information of the handwritten data or text data according to the PDF format and attaches information indicating that it is handwriting input-derived data and the color information as metadata. When the display device reads this PDF file from a storage medium (such as a USB memory), it can display the handwriting input-derived data in a line type corresponding to the color information based on the metadata. Other arbitrary devices ignore the metadata and display it as a normal PDF file, so it can be displayed in color. Note that metadata is data that describes the meaning of data. In other words, metadata is data that has information about a certain data. Since metadata is not the data itself but data about the data, it is called meta (higher-level) data.

[0022] Also, when the display device transmits data caused by handwritten input to an external device, the display device converts and transmits the data caused by handwritten input according to the type of the device. Examples of external devices include the following. · Black-and-white compatible device (e.g., monochrome printer, etc.) · Color compatible device (e.g., color printer, etc.) · Black-and-white emphasis compatible device (e.g., the display device of the present application, etc.)

[0023] By this conversion, the black-and-white compatible device and the color compatible device can display in black-and-white or color without determining whether the data is caused by handwritten input.

[0024] As described above, the display device of the present embodiment associates the fact with the data caused by handwritten input so that it can detect that the data is caused by handwritten input. Thereby, even when the display device 2 acquires a file from the outside, when displaying the data caused by handwritten input, black-and-white emphasis display can be performed based on the color information associated with the handwritten data. Also, when transmitting the data caused by handwritten input to an external device, it can be displayed without determining whether it is the data caused by handwritten input.

[0025] <Regarding terms> Black-and-white compatible device... A black-and-white compatible device is a device that can output only white or black, or grayscale. For example, there are electronic paper, monochrome printers, etc. It can also be said that it is a device that does not support color. Electronic paper is also called E-paper. It is a general term for displays that are about 1 / 10 mm thick and can display and erase data by electrical means. A typical technology for realizing electronic paper is the microcapsule type electrophoresis method, which is called E Ink (registered trademark) technology. Each microcapsule contains white and black fine particles, and the white is charged positively and the black is charged negatively.

[0026] Color-compatible device... A color-compatible device is a device that can output not only black and white but also color. Examples include devices having a liquid crystal or organic EL display, and color printers. A color-compatible device is also a black-and-white-compatible device.

[0027] Black-and-white emphasis-compatible device... A black-and-white emphasis-compatible device is a device that can output only white, black, or grayscale, but performs black-and-white emphasis display based on color information associated with handwritten data for data resulting from handwritten input. An example is the display device of the present embodiment (an example of a specific display device). Note that the display device can accept handwritten input and may also be called an input device or a handwritten input device.

[0028] Stroke data is a freely handwritten line. Stroke data has a set of continuous points and may be interpolated as appropriate.

[0029] Data resulting from handwritten input refers to data input by handwriting on a touch panel. Whether it remains handwritten or is converted to text data after input is not relevant. Also, data acquired from an external device does not lose its status as data resulting from handwritten input. Data resulting from handwritten input may include, in addition to text data obtained by character recognition and conversion, stamps displayed as fixed characters or marks such as "completed" or "confidential", figures such as circles and stars, and lines, i.e., data converted based on user operations. That is, data resulting from handwritten input is stroke data input by handwriting drawn within the display device, or text data obtained by converting this stroke data through handwriting recognition.

[0030] Data that is not data resulting from handwritten input refers to data other than data input by handwriting on a touch panel, for example, image (picture) data, handwritten data or text data input by a device that is not a black-and-white emphasis-compatible device. That is, data that is not data resulting from handwritten input is stroke data input by a personal computer or the like outside the display device, or text data obtained by converting the stroke through handwriting recognition.

[0031] ColorId is used to determine whether the data is caused by handwritten input. ColorId is an identifier for distinguishing between data caused by handwritten input and data not caused by handwritten input. As this identifier, it is also possible to discriminate using a terminal identifier, a model number (a number for identifying the type of a product), etc.

[0032] That the color information is emphasized in black and white means not simply converting the color into luminance or density, but displaying the data in a manner that enables discrimination of different colors. It is even better if this emphasis reflects properties of the color such as attention-grabbing property.

[0033] <An example of the appearance of the pen> FIG. 3 shows an example of a perspective view of the pen 2500. FIG. 3 shows an example of the multifunctional pen 2500. The pen 2500 that incorporates a power source and can send commands to the display device 2 is called an active pen (a pen without a built-in power source is called a passive pen). The pen 2500 in FIG. 3 has one physical switch at the tip of the pen, one at the butt of the pen, and two on the side of the pen. The tip of the pen is for writing, the butt of the pen is for erasing, and the side of the pen is for user function assignment. In the present embodiment, further, it has a non-volatile memory, and this memory stores a pen ID that does not overlap with other pens.

[0034] Note that if the pen has a switch, it is also possible to reduce the operation procedure of the user's display device 2. The pen with a switch mainly refers to an active pen, but in the electromagnetic induction method, even a passive pen without a built-in power source can generate power only with an LC circuit, so it includes not only active pens but also passive pens of the electromagnetic induction method. Pens with switches of optical methods, infrared methods, and capacitance methods other than the electromagnetic induction method are active pens.

[0035] Note that it is assumed that the hardware configuration of the pen 2500 is the same as a general control method including a communication function and a microcomputer. The pen 2500 has an electromagnetic induction method, an active electrostatic coupling method, etc. Also, the pen 2500 may have functions such as pen pressure detection, tilt detection, and hover function (displaying a cursor before the pen touches).

[0036] <Overall Configuration of the Device> Using FIG. 4, the overall configuration of the display device 2 according to the present embodiment will be described. FIG. 4 is a diagram showing the overall configuration diagram of the display device 2. FIG. 4(a) shows the display device 2 used as a horizontally long electronic blackboard suspended on a wall as an example of the display device 2.

[0037] As shown in FIG. 4(a), a display 220 as an example of the display device is installed at the upper part of the display device 2. The user U can write (also referred to as input or drawing) characters and the like on the display 220 using the pen 2500.

[0038] FIG. 4(b) shows the display device 2 used as a vertically long electronic blackboard suspended on a wall.

[0039] FIG. 4(c) shows the display device 2 placed flat on the desk 230. Since the display device 2 has a thickness of about 1 cm, there is no need to adjust the height of the desk even when it is placed flat on a general desk. Also, the user can move it easily.

[0040] Note that the display device 2 can automatically detect how it is being used by the tilt sensor.

[0041] <Hardware Configuration of the Device> Subsequently, using FIG. 5, the hardware configuration of the display device 2 will be described. The display device 2 has the configuration of an information processing device or a computer as shown in the figure. FIG. 5 is an example of the hardware configuration diagram of the display device 2. As shown in FIG. 5, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an SSD (Solid State Drive) 204.

[0042] Among these, the CPU 201 controls the operation of the entire display device 2. The ROM 202 stores programs used for driving the CPU 201 such as the CPU 201 and IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201. The SSD 204 stores various data such as programs for the display device 2.

[0043] Also, the display device 2 includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, a power switch 227, a tilt sensor 217, a serial interface 218, a speaker 219, a microphone 221, a wireless communication device 222, an infrared I / F 223, a power control circuit 224, an AC adapter 225, and a battery 226.

[0044] The display controller 213 controls and manages screen display in order to output the output image to the display 220 and the like. The touch sensor 216 detects that a pen 2500, a user's hand, etc. (the pen and the user's hand serve as input means) have touched the display 220. Also, the touch sensor 216 receives the pen ID.

[0045] The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 performs coordinate input and coordinate detection. For example, in the case of an optical method, the method of coordinate input and coordinate detection is such that two light emitting and receiving devices installed at both upper ends of the display 220 emit a plurality of infrared rays parallel to the display 220. The infrared rays are reflected by a reflecting member provided around the display 220, and the light emitting and receiving device receives the light that returns on the same optical path as the optical path of the light emitted by the light receiving element. The touch sensor 216 outputs the position information of the infrared rays emitted by the two light emitting and receiving devices blocked by an object to the touch sensor controller 215, and the touch sensor controller 215 specifies the coordinate position that is the contact position of the object. In addition, the touch sensor controller 215 has a communication unit 215a and can communicate wirelessly with the pen 2500. For example, when communicating according to a standard such as Bluetooth (registered trademark), a commercially available pen can be used. If one or more pens 2500 are registered in advance in the communication unit 215a, the user can communicate without performing a connection setting for communicating the pen 2500 with the display device 2.

[0046] The power switch 227 is a switch for switching the ON / OFF of the power of the display device 2. The tilt sensor 217 is a sensor for detecting the tilt angle of the display device 2. It is mainly used to detect whether the display device 2 is being used in any of the installation states shown in FIGS. 4(a), 4(b), or 4(c), and the thickness of characters and the like can be automatically changed according to the installation state.

[0047] The serial interface 218 is an interface for communication with the outside, such as a USB or LAN interface. The serial interface 218 is used for input of information from the outside. The speaker 219 is used for output of sound, and the microphone 221 is used for input of sound. The wireless communication device 222 communicates with a terminal carried by the user and relays, for example, a connection to the Internet. The wireless communication device 222 communicates via Wi-Fi, Bluetooth (registered trademark), etc., and the communication standard is not limited. The wireless communication device 222 forms an access point. When the user sets the SSID (Service Set Identifier) and password obtained by the user in the terminal carried by the user, the user can connect to the access point.

[0048] Note that it is preferable that two access points are provided for the wireless communication device 222. a. Access point → Internet b. Access point → Corporate network → Internet The access point in (a) is for external users. The user cannot access the corporate network via the access point in (a), but can use the Internet. The access point in (b) is for internal users. The user can use the corporate network and the Internet via the access point in (b).

[0049] The infrared I / F 223 detects the display device 2 arranged adjacent thereto. The infrared I / F 223 can detect only the display device 2 arranged adjacent thereto by utilizing the straight-line propagation of infrared rays. It is preferable that the infrared I / F 223 is provided one by one on each side, and it can detect in which direction of the display device 2 another display device 2 is arranged. The adjacent display device 2 can display handwritten information (handwritten information on another page with the area of one display 220 as one page) handwritten in the past.

[0050] The power control circuit 224 controls the AC adapter 225 and the battery 226, which are the power sources of the display device 2. The AC adapter 225 converts the alternating current shared by the commercial power supply into direct current.

[0051] When the display 220 is so-called electronic paper, since it consumes little or no power to maintain the image after the image is drawn, it can also be driven by the battery 226. As a result, it becomes possible to use the display device 2 for applications such as digital signage even in places where it is difficult to connect a power source, such as outdoors.

[0052] Furthermore, the display device 2 includes a bus line 210. The bus line 210 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 201 shown in FIG. 5.

[0053] Note that the touch sensor 216 is not limited to the optical type, and may be a capacitive touch panel that identifies the contact position by detecting a change in capacitance. The touch sensor 216 may be a resistive touch panel that identifies the contact position by the voltage change of two opposing resistive films. The touch sensor 216 may be an electromagnetic induction touch panel that detects the electromagnetic induction generated when the contact object touches the display unit and identifies the contact position. Various detection means may be used for the touch sensor 216. The touch sensor 216 may be a method that does not require an electronic pen to detect the presence or absence of a touch at the pen tip. In this case, a fingertip or a pen-shaped rod can be used for the touch operation. Note that the pen 2500 does not need to be an elongated pen shape.

[0054] <Regarding the functions of the device> Next, with reference to FIG. 6, the functions of the display device 2 and the pen 2500 will be described. FIG. 6(a) is an example of a functional block diagram showing the functions of the display device 2 in a block form. The display device 2 includes a handwriting input unit 21, a display unit 22, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a character string conversion control unit 28, a character string conversion dictionary unit 29, a prediction conversion control unit 30, a prediction conversion dictionary unit 31, an operation command recognition control unit 32, an operation command definition unit 33, a pen ID control data storage unit 36, a file transmission / reception control unit 37, a handwritten signature authentication control unit 38, and a handwritten signature data storage unit 39. Each function of the display device 2 is a function or means realized by any of the components shown in FIG. 5 operating according to instructions from the CPU 201 based on a program expanded from the SSD 204 onto the RAM 203.

[0055] The handwriting input unit 21 is realized by a touch sensor 216 or the like, receives handwriting input from the user, and receives a pen ID. The handwriting input unit 21 converts the pen input d1 of the user into pen operation data d2 with a pen ID (pen up, pen down, or pen coordinate data), and transmits it to the handwriting input display control unit 23. The pen coordinate data is transmitted periodically as discrete values, and the coordinates between the discrete values are calculated by interpolation.

[0056] The display unit 22 is realized by a display 220 or the like, and displays a handwritten object or an operation menu. The display unit 22 converts the drawing data d3 written by the handwriting input display control unit 23 into the video memory into data according to the characteristics of the display 220, and transmits it to the display 220.

[0057] The handwriting input display control unit 23 performs overall control related to handwriting input and display. The handwriting input display control unit 23 processes the pen operation data d2 from the handwriting input unit 21 and transmits it to the display unit 22 for display. Details of the processing of the pen operation data d2 and the display of the stroke will be described with reference to FIGS. 30 to 37 below.

[0058] The candidate display timer control unit 24 is a display control timer for selectable candidates. The candidate display timer control unit 24 generates the timing to start or stop the timer to start the display of selectable candidates and the timing to erase the display. The selectable candidates are handwritten recognition character string / language string candidates, conversion string candidates, string / predicted conversion candidates, and operation command candidates that are displayed selectably in the operation guide described later. The candidate display timer control unit 24 receives a timer start request d4 (which may also be a timer stop request) from the handwritten input display control unit 23 and transmits a timeout event d5 to the handwritten input display control unit 23.

[0059] The handwritten input storage unit 25 has a storage function for storing user data (handwritten object / string object). The handwritten input storage unit 25 receives user data d6-1 from the handwritten input display control unit 23 and stores it in the handwritten input storage unit 25. The handwritten input storage unit 25 receives an acquisition request d6-2 from the handwritten input display control unit 23 and transmits the user data d7 stored in the handwritten input storage unit 25. The handwritten input storage unit 25 transmits the position information d36 of the confirmed object to the operation command recognition control unit 32.

[0060] The handwritten recognition control unit 26 is a recognition engine that performs online handwritten recognition. Different from general OCR (Optical Character Reader), it recognizes characters (not only Japanese but also multiple languages such as English), numbers, symbols (%$, &, etc.), figures (lines, circles, triangles, etc.) in parallel with the user's pen operation. Although various algorithms have been devised for the recognition method, the details are omitted in this embodiment assuming that known technologies can be used.

[0061] The handwritten recognition control unit 26 receives pen operation data d8-1 from the handwritten input display control unit 23, performs handwritten recognition, and holds handwritten recognition character string candidates. Further, the handwritten recognition control unit 26 holds language character string candidates converted from the handwritten recognition character string candidates d12 using the handwritten recognition dictionary unit 27. Separately, when receiving an acquisition request d8-2 from the handwritten input display control unit 23, the handwritten recognition control unit 26 transmits the held handwritten recognition character string candidates and language character string candidates d9 to the handwritten input display control unit 23.

[0062] The handwritten recognition dictionary unit 27 is dictionary data for language conversion of handwritten recognition. The handwritten recognition dictionary unit 27 receives handwritten recognition character string candidates d12 from the handwritten recognition control unit 26, converts them into linguistically reliable language character string candidates d13, and transmits them to the handwritten recognition control unit 26. For example, in the case of Japanese, it converts hiragana into kanji or katakana.

[0063] The character string conversion control unit 28 controls the conversion of conversion character string candidates into character strings. The conversion character string is a character string that is likely to be generated including a handwritten recognition character string or a language character string. The character string conversion control unit 28 receives handwritten recognition character strings and language character string candidates d11 from the handwritten recognition control unit 26, converts them into conversion character string candidates using the character string conversion dictionary unit 29, and holds them. Separately, when receiving an acquisition request d14 from the handwritten input display control unit 23, the character string conversion control unit 28 transmits the held conversion character string candidates d15 to the handwritten input display control unit 23.

[0064] The character string conversion dictionary unit 29 is dictionary data for character string conversion. The character string conversion dictionary unit 29 receives handwritten recognition character strings and language character string candidates d17 from the character string conversion control unit 28, and transmits conversion character string candidates d18 to the character string conversion control unit 28.

[0065] The prediction conversion control unit 30 receives the handwritten recognition string and the language string candidate d10 from the handwritten recognition control unit 26, and receives the conversion string candidate d16 from the string conversion control unit 28. The prediction conversion control unit 30 converts each of the handwritten recognition string, the language string candidate d10, and the conversion string candidate d16 into a prediction string candidate using the prediction conversion dictionary unit 31. The prediction conversion string is a string that is highly likely to be generated including the handwritten recognition string, the language string, or the conversion string. Separately, when receiving the acquisition request d19 from the handwritten input display control unit 23, the prediction conversion control unit 30 transmits the prediction string candidate d20 to the handwritten input display control unit 23.

[0066] The prediction conversion dictionary unit 31 is dictionary data for prediction conversion. The prediction conversion dictionary unit 31 receives the handwritten recognition string, the language string candidate, and the conversion string candidate d21 from the prediction conversion control unit 30, and transmits the prediction string candidate d22 to the prediction conversion control unit 30.

[0067] The operation command recognition control unit 32 receives the handwritten recognition string and the language string candidate d30 from the handwritten recognition control unit 26, and receives the conversion string candidate d28 from the string conversion control unit 28. The operation command recognition control unit 32 receives the prediction string candidate d29 from the prediction conversion control unit 30. Then, the operation command recognition control unit 32 transmits the operation command conversion request d26 to the operation command definition unit 33 for each of the handwritten recognition string, the language string candidate d30, the conversion string candidate d28, and the prediction string candidate d29, and receives the operation command candidate d27 from the operation command definition unit 33. The operation command recognition control unit 32 holds the operation command candidate d27.

[0068] When the operation command conversion request d26 partially matches the operation command definition, the operation command definition unit 33 transmits the operation command candidate d27 to the operation command recognition control unit 32.

[0069] In addition, the operation command recognition control unit 32 receives pen operation data d24-1 from the handwritten input display control unit 23. The operation command recognition control unit 32 transmits a position information acquisition request d23 for a confirmed object that has been input and confirmed in the past to the handwritten input storage unit 25. The operation command recognition control unit 32 holds the confirmed object specified by the pen operation data as a selected object (including position information). The operation command recognition control unit 32 identifies a selected object that satisfies a predetermined criterion with the position of the pen operation data d24-1. Separately, when receiving an acquisition request d24-2 from the handwritten input display control unit 23, the operation command recognition control unit 32 transmits the held operation command candidate and the identified selected object d25 to the handwritten input display control unit 23.

[0070] The pen ID control data storage unit 36 holds pen ID control data (which may also be referred to as storage means). Before the handwritten input display control unit 23 transmits display data to the display unit 22, the pen ID control data storage unit 36 transmits pen ID control data d41 to the handwritten input display control unit 23. The handwritten input display control unit 23 draws the display data under the operation conditions stored in association with the pen ID. Also, before the handwriting recognition control unit 26 executes handwriting recognition, the pen ID control data storage unit 36 transmits the angle information d44 of the pen ID control data to the handwriting recognition control unit 26, and the handwriting recognition control unit 26 rotates the stroke with the angle information stored in association with the pen ID to execute handwriting recognition.

[0071] In addition, after the handwriting recognition control unit 26 recognizes a straight line for setting the angle information when the user handwrites characters or the like, the handwriting recognition control unit 26 transmits the angle information d43 of the pen ID control data to the pen ID control data storage unit 36 and stores the angle information d43 in association with the pen ID. Also, after the handwritten input display control unit 23 executes an operation command for setting the angle information, the handwritten input display control unit 23 transmits the pen ID control data d42 to the pen ID control data storage unit 36 and stores the execution result of the operation command (the angle information set by the user) in association with the pen ID. Thereafter, the stroke of the pen ID is rotated by the set angle information before handwriting recognition is executed.

[0072] Also, the handwritten recognition control unit 26 transmits the stroke data d49 rotated clockwise by the angle information of the pen ID control data to the handwritten signature authentication control unit 38. Thereby, it becomes possible to authenticate the handwritten signature regardless of the position of the user (from which direction the user writes with respect to the display device 2).

[0073] The handwritten signature data storage unit 39 holds the handwritten signature data. When the handwritten signature data storage unit 39 receives a handwritten signature data acquisition request d45 from the handwritten signature authentication control unit 38, it transmits the handwritten signature data d46 to the handwritten signature authentication control unit 38. The format of the handwritten signature data is assumed to depend on the handwritten signature authentication algorithm of the handwritten signature authentication control unit 38. The data of the handwritten signature data storage unit 39 will be described with reference to FIG. 15.

[0074] When the handwritten signature authentication control unit 38 receives the stroke data d49 rotated clockwise from the handwritten recognition control unit 26, it transmits a handwritten signature data acquisition request d45 to the handwritten signature data storage unit 39, and the handwritten signature data storage unit 39 transmits the handwritten signature data d46 to the handwritten signature authentication control unit 38.

[0075] The handwritten signature authentication control unit 38 authenticates the user based on the handwritten signature data. Although various algorithms have been devised for user authentication based on handwritten signature data, in this embodiment, a technique that can be recognized with a recognition rate that does not cause practical problems is used. For example, a feature vector is generated using elements such as the coordinates, pen pressure, and time taken to write the stroke that make up the handwritten signature data, and the elements are appropriately weighted, and the feature vector of the registered handwritten signature data is compared with the feature vector of the user name or the like handwritten by the user at the time of signing in. If the degree of coincidence is equal to or higher than the threshold value, it is determined that the authentication is successful, and if it is less than the threshold value, it is determined that the authentication is failed.

[0076] The handwritten signature authentication control unit 38 holds the authentication result of the handwritten signature, which is the comparison result between the stroke data d49 and the handwritten signature data d46. Separately, when receiving an acquisition request d48 from the handwritten input display control unit 23, the handwritten signature authentication control unit 38 transmits the held authentication result d47 of the handwritten signature to the handwritten input display control unit 23. The authentication result of the handwritten signature indicates whether the stroke data d49 and the handwritten signature data d46 can be regarded as matching, and if so, it has a SignatureId (described later) associated with the matching handwritten signature data d46.

[0077] When the handwritten recognition result of the handwritten recognition control unit 26 conforms to an operation command instructing the execution of handwritten signature registration, the handwritten recognition control unit 26 acquires the data d52 input into the handwritten signature registration form (a frame into which handwritten signature data is input, as described later) from the handwritten input storage unit 25, and transmits the handwritten signature data d50 among the data d52 to the handwritten signature authentication control unit 38. The handwritten signature authentication control unit 38 transmits the received handwritten signature data d50 to the handwritten signature data storage unit 39 for registration.

[0078] When the handwritten recognition result of the handwritten recognition control unit 26 is an instruction to cancel the handwritten signature, the handwritten recognition control unit 26 transmits a deletion request d51 for the handwritten signature registration form to the handwritten input storage unit 25 to delete the handwritten signature registration form from the handwritten input storage unit 25.

[0079] When the handwritten recognition result of the handwritten recognition control unit 26 is an instruction to execute user-defined data modification, the handwritten recognition control unit 26 acquires the data d53 input into the user-defined data modification form from the handwritten input storage unit 25. The handwritten recognition control unit 26 transmits the modification value d54 among the data d53 to the operation command definition unit 33 to modify the user-defined data. The user-defined data will be described with reference to FIG. 14.

[0080] When the handwritten recognition result of the handwritten recognition control unit 26 is a cancellation instruction for the user-defined data change form, the handwritten recognition control unit 26 sends a deletion request d55 for the user-defined data change form to the handwritten input storage unit 25, and deletes the user-defined data change form from the handwritten input storage unit 25.

[0081] The file transmission / reception control unit 37 performs storage to and acquisition from a storage medium of data caused by handwritten input or data that is not data caused by handwritten input, communication with an external device (printing request, display request, etc.). The file transmission / reception control unit 37 receives an execution request d64 for file transmission / reception from the handwritten input display control unit 23. When saving or printing a file, the handwritten input display control unit 23 sends a file transmission request to the file transmission / reception control unit 37, and the file transmission / reception control unit 37 sends an acquisition request for the handwritten input stored data d61 to the handwritten input storage unit 25. · In the case of data that is not data caused by handwritten input, the handwritten input storage unit 25 transmits the data while maintaining the color information held by the data. · In the case of data caused by handwritten input, when the transmission destination is a color-compatible device (e.g., a color printer), the handwritten input storage unit 25 transmits the handwritten input stored data d62 converted to color to the file transmission / reception control unit 37. · In the case of data caused by handwritten input, when the transmission destination is a black-and-white-compatible device (e.g., a monochrome printer), the handwritten input storage unit 25 transmits the handwritten input stored data d62 converted to black and white to the file transmission / reception control unit 37. Since the black-and-white-compatible device may be able to convert to grayscale, it may also be transmitted after conversion to color. · When the transmission destination is a black-and-white emphasis-compatible device, the handwritten input stored data described later is transmitted to the file transmission / reception control unit 37. · Also, when the handwritten input storage unit 25 saves data caused by handwritten input to a file, it converts the handwritten input stored data d62 to color according to the file format and attaches the handwritten input stored data as metadata of the file. When saving data that is not data caused by handwritten input to a file, it converts the handwritten input stored data d62 to color according to the file format.

[0082] Whether it is a color correspondence device or a black-and-white correspondence device is stored, for example, in the MIB (Management Information Base) held by the network device, and the file transmission / reception control unit 37 can determine by acquiring the MIB. Similarly, it is also possible to determine whether it is a black-and-white emphasis correspondence device or not from the model name and the like published by the MIB.

[0083] When a file is read, the handwritten input display control unit 23 transmits a file list acquisition request d65 to the file transmission / reception control unit 37. The file transmission / reception control unit 37 transmits a file list acquisition request to an external device, acquires a file list d63, and transmits it to the handwritten input display control unit 23. The handwritten input display control unit 23 displays the file list on the display unit 22. The handwritten input unit 21 transmits the display position of the selected file to the handwritten input display control unit 23. The handwritten input display control unit 23 transmits a file reception request d66 to the file transmission / reception control unit 37. The file transmission / reception control unit 37 acquires a file from an external device and transmits this file d67 to the handwritten input storage unit 25. The handwritten input storage unit 25 analyzes the metadata of the file to determine whether it is data caused by handwritten input. If it is data caused by handwritten input, it extracts handwritten input storage data (black-and-white emphasis / color-convertible data described later). The handwritten input storage unit 25 converts the data caused by handwritten input into black-and-white emphasis display. If it is not data caused by handwritten input, it transmits it to the handwritten input display control unit 23 without conversion (displayed in grayscale). The handwritten input display control unit 23 transmits the display data of the handwritten object to the display unit 22.

[0084] FIG. 6(b) is a functional block diagram showing the functions of the pen 2500 in a block form. The pen 2500 has a pen event transmission unit 41. The pen event transmission unit 41 attaches a pen ID to the event data of pen up, pen down, and pen coordinates and transmits it to the display device 2.

[0085] <Regarding defined control data> Next, with reference to FIG. 7, the defined control data used by the display device 2 for various processes will be described. FIG. 7 shows an example of the defined control data. In the example of FIG. 7, the control data is shown for each control item.

[0086] The selectable candidate display timer 401 defines the time until the selectable candidates are displayed. This is to prevent the selectable candidates from being displayed during handwriting. In FIG. 7, it means that if a pen down does not occur within TimerValue = 500 [ms] from pen up, the selectable candidates will be displayed. The selectable candidate display timer 401 is held by the candidate display timer control unit 24. The selectable candidate display timer 401 is used at the start of the selectable candidate display timer in step S18-2 of FIG. 32 described later.

[0087] The selectable candidate erasure timer 402 defines the time until the displayed selectable candidates are erased. This is to erase the selectable candidates when the user does not select them. In FIG. 7, it means that if the selectable candidates are not selected within TimerValue = 5000 [ms] from the display of the selectable candidates, the selectable candidate display data will be erased. The selectable candidate erasure timer 402 is held by the candidate display timer control unit 24. The selectable candidate erasure timer 402 is used at the start of the selectable candidate display erasure timer in step S64 of FIG. 34.

[0088] The handwritten object vicinity rectangular area 403 defines a rectangular area regarded as the vicinity of the handwritten object. In the example of FIG. 7, the handwritten object vicinity rectangular area 403 is a rectangular area obtained by expanding the rectangular area of the handwritten object by 50% (Horizontal) of the estimated character size in the horizontal direction and 80% (Vertical) of the estimated character size in the vertical direction. In the example of FIG. 7, it is a ratio (%) of the estimated character size, but it can also be made a fixed length if the unit is "mm" or the like. The handwritten object vicinity rectangular area 403 is held by the handwritten input storage unit 25. The estimated character size 405 is used in step S10 of FIG. 31 to determine the overlapping situation between the handwritten object vicinity rectangular area and the stroke rectangular area.

[0089] The handwriting direction / character size determination condition 404 defines constants for determining the handwriting direction and the measurement direction of the character size. In the example of FIG. 7, when the difference between the time when the first stroke was added and the time when the last stroke was added in the handwritten object rectangular area is MinTime = 1000 [ms] or more, and the difference between the horizontal distance (width) and the vertical distance (height) of the handwritten object rectangular area is MinDiff = 10 [mm] or more, and the horizontal distance is longer than the vertical distance, it means that the estimated handwriting direction is "horizontal writing" and the estimated character size is determined as the vertical distance. When the horizontal distance is shorter than the vertical distance, it means that the estimated handwriting direction is "vertical writing" and the estimated character size is determined as the horizontal distance. If the above conditions are not met, the estimated handwriting direction is "horizontal writing" (DefaultDir = "Horizontal"), and the estimated character size is determined as the longer distance between the horizontal distance and the vertical distance. The handwriting direction / character size determination condition 404 is held by the handwritten input storage unit 25. The handwriting direction / character size determination condition 404 is used for obtaining the estimated handwriting direction in step S59 of FIG. 34 and for obtaining the character string object font in step S81 of FIG. 37.

[0090] The estimated character size 405 defines data for estimating the size of characters and the like. In the example of FIG. 7, it means that the estimated character size determined by the handwriting direction / character size determination condition 404 is compared with the smaller character 405a (hereinafter referred to as the minimum font size) and the larger character 405c (hereinafter referred to as the maximum font size) of the estimated character size 405. When the estimated character size is smaller than the minimum font size, the estimated character size is determined as the minimum font size. When the estimated character size is larger than the maximum font size, the estimated character size is determined as the maximum font size. Otherwise, it is determined as the character size of the medium-sized character 405b. The estimated character size 405 is held by the handwritten input storage unit 25. The estimated character size 405 is used for obtaining the character string object font in step S81 of FIG. 37.

[0091] Specifically, the handwritten input storage unit 25 compares the estimated character size determined by the estimated writing direction / character size determination condition 404 with the FontSize of the estimated character size 405 and uses the font with the closest size. For example, when the estimated character size is 25 [mm] (FontSize of smaller characters) or less, it is determined as "smaller characters"; when the estimated character size is more than 25 mm and 50 mm (FontSize of medium-sized characters) or less, it is determined as "medium-sized characters"; when the estimated character size is more than 100 mm (FontSize of larger characters), it is determined as "larger characters". "Smaller characters" 405a uses a Mincho font of 25 mm (FontStyle = "Mincho", FontSize = "25 mm"), "medium-sized characters" 405b uses a Mincho font of 50 mm (FontStyle = "Mincho", FontSize = "50 mm"), and "larger characters" 405c uses a Gothic font of 100 mm (FontStyle = "Gothic", FontSize = "100 mm"). If you want to increase the number of font sizes or types of styles, you can increase the types of the estimated character size 405.

[0092] The strikethrough determination condition 406 defines data used for determining whether a plurality of objects are selected. If the handwritten object is a single stroke, in the example of FIG. 7, if there is an object whose length of the long side of the handwritten object is 100 [mm] or more (MinLenLongSide = "100 mm"), the length of the short side is 50 [mm] or less (MaxLenShortSide = "50 mm"), and the overlap rate in the long side direction and the short side direction with the handwritten object is 80 [%] or more (MinOverLapRate = "80%"), it is determined that a plurality of objects are selected (selected objects). The strikethrough determination condition 406 is held by the operation command recognition control unit 32. The strikethrough determination condition 406 is used for the strikethrough determination of the selection object determination in step S50 of FIG. 33.

[0093] The surrounding line determination condition 407 defines data used to determine whether an object is a surrounding line. In the example of FIG. 7, the operation command recognition control unit 32 determines a confirmed object with an overlap rate of 100% or more (MinOverLapRate = "100%") in the long side direction and the short side direction of the handwritten object as the selected object. The surrounding line determination condition 407 is held by the operation command recognition control unit 32. The surrounding line determination condition 407 is used for the surrounding line determination of the selected object determination in step S50 of FIG. 33.

[0094] Note that either the crossing line determination condition 406 or the surrounding line determination condition 407 may be preferentially determined. For example, when the crossing line determination condition 406 is relaxed (when it is easier to select a crossing line) and the surrounding line determination condition 407 is made strict (when set to a value that allows only the surrounding line to be selected), the operation command recognition control unit 32 should preferentially determine according to the surrounding line determination condition 407.

[0095] <An example of dictionary data> The dictionary data will be described with reference to FIGS. 8 to 10. FIG. 8 is an example of the dictionary data of the handwritten recognition dictionary unit 27. FIG. 9 is an example of the dictionary data of the character string conversion dictionary unit 29. FIG. 10 is an example of the dictionary data of the prediction conversion dictionary unit 31. Note that these dictionary data are respectively used in steps S33 to S41 of FIG. 33.

[0096] In this embodiment, the conversion result by the dictionary data of the handwriting recognition dictionary unit 27 in FIG. 8 is called a language string candidate, the conversion result by the dictionary data of the string conversion dictionary unit 29 in FIG. 9 is called a converted string candidate, and the conversion result by the dictionary data of the predictive conversion dictionary unit 31 in FIG. 10 is called a predicted string candidate. "Before conversion" of each dictionary data is a string to search the dictionary data, "after conversion" is a converted string corresponding to the string to be searched, and "probability" represents the probability of selection by the user. The probability is calculated from the result of the user selecting each string in the past. Therefore, the probability may be calculated for each user. Various algorithms have been devised as a method of calculating the probability, but it is sufficient to calculate it by an appropriate method as appropriate, and details will be omitted. This embodiment is characterized in that the string candidates are displayed in descending order of selection probability from the estimated writing direction.

[0097] The dictionary data of the handwriting recognition dictionary unit 27 in Fig. 8 indicates that handwritten "gi" is converted to "gi" with a probability of 0.55 and to "gi" with a probability of 0.4, and handwritten "gishi" is converted to "gishi" with a probability of 0.5 and to "gishi" with a probability of 0.45. The same is true for other "before conversion" character strings. In Fig. 8, the "before conversion" character string is handwritten hiragana, but characters other than hiragana may be registered as "before conversion".

[0098] 9 indicates that the character string "" is converted to "" with a probability of 0.95, and the character string "" is converted to "" with a probability of 0.85. The same applies to the other "pre-conversion" character strings.

[0099] The dictionary data of the predictive conversion dictionary unit 31 in Fig. 10 indicates that the character string "Minutes" is converted to "Destination of minutes" with a probability of 0.65, and the character string "Skill test" is converted to "Approve skill test" with a probability of 0.75. In the example of Fig. 10, all the character strings before conversion are kanji, but characters other than kanji may be registered.

[0100] The dictionary data is not language-dependent, and any character string may be registered before and after conversion.

[0101] <Operation command definition data held by the operation command definition unit> Next, the operation command definition data used by the operation command recognition control unit 32 will be described with reference to FIG. 11. FIG. 11 shows an example of the operation command definition data and system definition data held by the operation command definition unit 33.

[0102] FIG. 11(a) shows an example of the operation command definition data. The operation command definition data in FIG. 11(a) is an example of the operation command definition data when there is no selected object selected by the handwritten object, and all operation commands for operating the display device 2 are targeted. The operation commands in FIG. 11(a) have an operation command name (Name), a string (String) that partially matches the string candidate, and an operation command string (Command) to be executed. The “%~%” in the operation command string is a variable, and as shown in FIG. 11(b), it is associated with the system definition data. That is, “%~%” is replaced with the system definition data shown in FIG. 11(b).

[0103] First, the operation command definition data 701 indicates that the operation command name is “Load the minutes template”, the string that partially matches the string candidate is “minutes” or “template”, and the operation command string to be executed is “ReadFile https: / / %username%:%password%@server.com / template / minutes.pdf”. In this example, the system definition data of “%~%” is included in the operation command string to be executed, indicating that “%username%” and “%password%” are replaced with the system definition data 704 and 705, respectively. Therefore, the operation command string finally executed becomes the string “ReadFile https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf”, indicating that the file “https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf” is read (ReadFile).

[0104] The operation command definition data 702 indicates that the operation command name is "Save to the Minutes Folder", the string candidates that partially match the string are "Minutes" or "Save", and the operation command string to be executed is "WriteFile https: / / %username%:%password%@server.com / minutes / %machinename%_%yyyy-mm-dd%.pdf". Similar to the operation command definition data 701, the "%username%", "%password%", and "%machinename%" in the operation command string are replaced by the system definition data 704 to 706 respectively. Note that "%yyyy-mm-dd%" indicates that it should be replaced with the current date. For example, if the current date is September 26, 2018, it indicates that it should be replaced with "2018-09-26". The finally executed operation command becomes "WriteFile https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf", indicating that the minutes are saved (WriteFile) to the file "https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf".

[0105] The operation command definition data 703 indicates that the operation command name is "Print", the strings that partially match the string candidates are "Print" or "Print", and the operation command string to be executed is "PrintFile https: / / %username%:%password%@server.com / print / %machinename%-"%yyyy - mm - dd%.pdf". Similar to the operation command definition data 702, when replacing the operation command string, the finally executed operation command becomes "PrintFile https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My - Machine_2018 - 09 - 26.pdf", indicating to print (PrintFile) the file "https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My - Machine_2018 - 09 - 26.pdf". That is, the file is sent to the server. When the user makes the printer communicate with the server and specifies a file, the printer prints the content of the file on paper.

[0106] In this way, since the operation command definition data 701 to 703 can be specified from the string candidates, the user can display the operation command by handwriting. Also, when the user's authentication is successful, the " %username%", " %password%", etc. in the operation command definition data are replaced with the user information, so that file input and output can be performed in association with the user.

[0107] When the user's authentication is not performed (including the case of authentication failure, but when the user can use the display device 2, the case of authentication failure is also included), the display device 2 replaces with the preset " %username%", " %password%", etc. of the display device 2. Therefore, even without user authentication, file input and output can be performed in association with the display device 2.

[0108] The operation command definition data 709, 710, 711, 720, 721, 722 are operation commands for changing the pen color. The pen color is the color of the handwritten data input with the pen used by the user. The operation command names of the operation command definition data 709, 710, 711, 720, 721, 722 are "black pen", "red pen", "blue pen", "green pen", "magenta pen", "cyan pen" respectively. For example, in the case of "black pen", the strings that partially match the string candidate are "kuro" or "pen". If the user writes "kuro", only "black pen" will be displayed in the operation command candidates. On the other hand, since "pen" also corresponds to the string (String) that partially matches the string candidate in "red pen" etc., if the user writes "pen", "black pen" to "cyan pen" will be displayed in the operation command candidates. When these operation commands are executed, as shown in Fig. 17, the control data associated with the pen ID of the pen 2500 used by the user for the operation is updated, and the color of the pen with this pen ID (ColorId of the pen ID control data) is changed.

[0109] The operation command definition data 719 has an operation command name of "load file", the strings that partially match the string candidate are "file", "load", "loading", and the operation command to be executed is "ReadFile https: / / %username%:%password%@server.com / files / ". When replacing with system definition data in the same way as the operation command definition data 702, the operation command becomes "ReadFile https: / / taro.tokkyo:x2PDHTyS@server.com / files / ", indicating that the files in this address (folder) are to be read (ReadFile). When this operation command is executed, the display device 2 displays a file selection window as shown in Fig. 12. The file selection window in Fig. 12 displays the first page etc. of the files stored in this address (folder) as thumbnails.

[0110] FIG. 12 shows an example of file list display data displayed on the display unit 22 in step S94 (file list display data) of FIG. 37. FIG. 12(a) is created from the file list information obtained in the file list information acquisition in step S91 of FIG. 37. The file list information has a list of file addresses, and the display unit 22 extracts the file name and thumbnail from each file address, sorts them in ascending order of file name, and displays them. In FIG. 12, the display unit 22 displays four thumbnails and file names, and the left and right arrow icons 99 display the previous and next thumbnails and file names. When the displayed thumbnail or file name is pressed with a pen, the display unit 22 erases the screen of FIG. 12, and the file transmission / reception control unit 37 receives the file from the file address where the pen was pressed.

[0111] When the close button 98 in the upper right corner is pressed with a pen, the display unit 22 erases the screen of FIG. 12, and the execution of this operation command definition data 719 is canceled. When the PDF files in FIGS. 12(a) and (d) are selected, the file transmission / reception control unit 37 receives the PDF files, saves them in the handwritten input storage unit 25, and analyzes them. The file transmission / reception control unit 37 saves text data (with metadata), which is handwritten input-derived data, as handwritten input storage data (data capable of black-and-white highlighting and color conversion), such as the handwritten input storage data 801 to 805 in FIG. 16. The data capable of black-and-white highlighting and color conversion is data that can be both black-and-white highlighted and color-converted (mainly handwritten input-derived data). Text data that is not handwritten input-derived data is converted and saved as handwritten input storage data, such as the handwritten input storage data 806 in FIG. 16, and is displayed on the display 220.

[0112] When the JPG files in FIGS. 12(b) and (c) are selected, the file transmission / reception control unit 37 receives the image files, saves them as handwritten input storage data, such as the handwritten input storage data 807 in FIG. 16, in the handwritten input storage unit 25, and displays them on the display 220.

[0113] Returning to FIG. 11 for explanation. The operation command definition data 712 is an operation command for aligning the direction of text data in a certain direction. The operation command name of the operation command definition data 712 is "Align Text Direction", the strings that partially match the string candidates are "text", "orientation", or "direction", and the operation command string is "AlignTextDirection". Since the text data written by the user from directions other than the vertical and horizontal directions has various orientations, it is difficult to read all of them from a single direction. When the user executes the operation command definition data 712, the display device 2 aligns the handwritten recognized strings in the same direction (for example, the vertical and horizontal directions). In this case, aligning means rotating the text data only by the angle information.

[0114] The operation command definition data 713 indicates that the operation command name is "Register Handwritten Signature", the strings that partially match the string candidates are "signature" and "register", and the operation command string is "RegistSignature". When the RegistSignature command is executed, a handwritten signature registration form is added to the handwritten input storage unit 25, and the handwritten signature registration form is displayed on the operation screen 101. An example of the handwritten signature registration form will be described later (see FIG. 27).

[0115] The operation command definition data 714 indicates that the operation command name is "Sign in with Handwritten Signature", the string that partially matches the string candidate is "%signature%", and the operation command is "Signin". Here, "%signature%" is a reserved word in the system definition data, and it is assumed to represent the fact that the registered handwritten signature data and the stroke data such as the user name match. That is, when they match, the operation command 512 based on the operation command definition data 714 is displayed on the operation guide 500 (see FIG. 28).

[0116] When the Signin command is executed, the AccountId of the user with the SignatureId of the handwritten signature data that matches the pen ID control data of the pen 2500 on which the stroke data such as the user name is handwritten is saved. As a result, the pen ID and the AccountId are associated with each other, and the display device 2 can use the user-defined data specified by this AccountId (see Fig. 17(a)).

[0117] The operation command definition data 715 indicates that the operation command name is "sign out by handwritten signature", the strings that partially match the string candidates are "sign" or "out", and the operation command is "Signout". When the Signout command is executed, the AccountId is deleted from the pen ID control data of the pen 2500 that has performed the handwritten signature out operation. As a result, the association between the pen ID and the AccountId is lost, and any user can use the pen 2500.

[0118] The operation command definition data 716 indicates that the operation command name is "change settings", the strings that partially match the string candidates are "settings" or "change", and the operation command is "ConfigSettings". When the ConfigSettings command is executed, a user-defined data change form is added to the handwritten input storage unit 25, and the user-defined data change form is displayed on the operation screen 101. The user-defined data change form will be described later (see Fig. 29).

[0119] Subsequently, the operation command definition data in the case where there is a selected object, that is, the operation command definition data of the editing system and the decoration system, will be described. Fig. 13 shows an example of the operation command definition data in the case where there is a selected object selected by the handwritten object. The operation command definition data in Fig. 13 has an operation command name (Name), a group name (Group) of the operation command candidates, and an operation command string (Command) to be executed.

[0120] The operation command definition data 707 defines the operation commands of the editing system (Group = "Edit"), and is an example of the definition data of the operation command names "Delete", "Move", "Rotate", and "Select" in the editing system. That is, these operation commands are displayed for the selected object, and the user can select the desired operation command.

[0121] The operation command definition data 708 defines the operation commands of the decoration system (Group = "Decorate"), and is an example of the definition data of the operation command names "Thicken", "Thin", "Enlarge", "Reduce", and "Underline" in the decoration system. These operation commands are displayed for the selected object, and the user can select the desired operation command. In addition, operation commands for colors may also be displayed.

[0122] Therefore, when the user selects the selected object with the handwritten object, the operation command definition data 707 and 708 are specified, so that the operation commands can be displayed by the user's handwriting.

[0123] <User-defined data> Next, the user-defined data will be described with reference to FIG. 14. FIG. 14 shows an example of the user-defined data held by the operation command definition unit 33. The user-defined data in FIG. 14 is an example of the definition data of one user. The AccountId of the user-defined data 717 is the identification information of the user automatically numbered for each user, AccountUsername and AccountPassword are the username and password, SignatureId is the identification information of the handwritten signature data automatically numbered when registering the handwritten signature data, and username, password, and machinename are strings set in the operation command definition data 701 to 703 instead of the system definition data 704 to 706 respectively. Thereby, it becomes possible to execute the operation commands using the user-defined data.

[0124] When the user signs in by handwriting a username or the like, the handwritten input display control unit 23 utilizes the fact that the pen ID and the AccountId are associated with the pen ID control data (see Fig. 17(a)). The handwritten input display control unit 23 uses the character string of the user-defined data having the AccountId associated with the pen ID of the pen 2500 used by the user when executing the operation command. After the user signs out, even if the pen 2500 used by the user for signing in is used, the character string of the system-defined data is used when executing the operation command.

[0125] The user-defined data 718 is the data used in the user-defined data change form. Name is the item name of AccountUsername, AccountPassword, username, password, or machinename in the user-defined data 717, and Data is the changed value of AccountUsername, AccountPassword, username, password, or machinename. In this example, the data of "Name" is "%AccountName%". The data of "Password" is "%AccountPassword%". The data of "Folder Username" is "%username%". The data of "Folder Password" is "%password". The data of "Folder File Name" is "%machinename". Each item corresponds to each item of the user-defined data 717. These items input in the user-defined data change form are reflected in the user-defined data 717.

[0126] It is also possible for the user to register the color definition data described later in the user-defined data 717, and the user can input with the color definition data defined by himself / herself.

[0127] <Handwritten signature data> Next, the handwritten signature data will be described with reference to FIG. 15. FIG. 15 shows an example of the handwritten signature data held by the handwritten signature data storage unit 39. The handwritten signature data has Data representing a handwritten signature associated with SignatureId. SignatureId is identification information automatically numbered at the time of handwritten signature data registration, and Data is data calculated by the handwritten signature authentication algorithm of the handwritten signature authentication control unit 38 from the stroke data received from the handwritten signature authentication control unit 38.

[0128] <Handwritten input storage data stored by the handwritten input storage unit> Next, the handwritten input storage data will be described with reference to FIG. 16. FIG. 16 shows an example of the handwritten input storage data stored by the handwritten input storage unit 25. One line in FIG. 16 represents one stroke, one sentence (text), or an image.

[0129] When Type is a stroke, one handwritten input storage data has items such as DataId, Type, PenId, ColorId, Angle, StartPoint, StartTime, EndPoint, EndTime, Point, and Pressure. DataId is the identification information of the stroke. Type is the type of the handwritten input storage data. The types include stroke (Stroke), group (Group), text (Text), and image (Image). The types of the handwritten input storage data 801 and 802 are Stroke. The type of the handwritten input storage data 803 is Group. The types of the handwritten input storage data 804, 805, and 806 are Text. The type of the handwritten input storage data 807 is Image.

[0130] "Group" means grouping other strokes. The handwritten input saved data of the type "Group" specifies the strokes to be grouped by DataId. PenId, ColorId, and Angle are the ones to which the following pen ID control data is transferred. StartPoint is the starting point coordinates of the stroke, and StartTime is the starting time of the stroke. EndPoint is the ending point coordinates of the stroke, and EndTime is the ending time of the stroke. Point is the coordinate sequence from the starting point to the ending point, and Pressure is the pen pressure from the starting point to the ending point. As shown in Angle, the handwritten input saved data 804 and 805 indicate that they were rotated 180 degrees and 270 degrees clockwise respectively before being recognized by handwriting recognition.

[0131] Also, the handwritten input saved data 804, 805, and 806 with the type "Text" have FontName, FontSize, and Text. FontName is the font name of the text, FontSize is the character size, and Text is the content (character code) of the text.

[0132] The handwritten input saved data 801 to 805 may be attached as metadata as they are when saving the file as handwritten input origin data. Therefore, the display device 2 can also acquire the handwritten input saved data 801 to 805 when reading the file. When the display device 2 transmits the handwritten input saved data to an external device, in the case of a color-compatible device, the handwritten input saved data can be converted to color, and in the case of a black-and-white-compatible device, it may be transmitted without conversion to color or after conversion to color. When transmitting to a black-and-white emphasis-compatible device, the handwritten input saved data may be transmitted. In this way, the handwritten input saved data 801 to 805 is compatible with both black-and-white emphasis and color conversion.

[0133] The handwritten input saved data 801 to 805 are data caused by handwritten input, while the handwritten input saved data 806 and 807 are not data caused by handwritten input. These are files read by the file read command. This is determined by whether the value of the color defined by the color definition data described later is registered in ColorId. For example, the ColorId of the handwritten input saved data 801 is "Black", while the ColorId of the handwritten input saved data 806 is "#e6001200". This is represented by # and eight digits in hexadecimal, and represents the values of R (red), G (green), B (blue), and A (opacity) every two digits.

[0134] In this way, in the text data that is data caused by handwritten input, a string representing color information is contained in ColorId, while in the text data that is not data caused by handwritten input, ColorId is "#color value". In this way, the handwritten input storage unit 25 can determine whether it is text data caused by handwritten input by paying attention to ColorId.

[0135] The handwritten input storage unit 25 saves the color of the text data of the file read by the file transmission / reception control unit 37 as ColorId = "#e6001200". Therefore, the handwritten input saved data 806 does not support black and white highlighting and only supports color conversion.

[0136] The handwritten input display control unit 23 determines whether it is data caused by handwritten input using ColorId, and when it is not data caused by handwritten input, it calculates and displays the grayscale value from RGBA. When the display device 2 is a color-compatible device, RGBA is displayed as it is.

[0137] The handwritten input saved data 807 is also an image file read from a storage medium by a file read command. When the Type is Image, the handwritten input saved data 807 has a FileId and a FileName. The FileId is the management number within the handwritten input saved data, and the FileName is the original file name. For example, when the file transmission / reception control unit 37 reads the file "color_chart.pdf" in Fig. 12(a), the RGBCMY text in the central part is in the format of the handwritten input saved data 806, and the ring-shaped color part is in the format of the handwritten input saved data 807, and they are respectively saved in the handwritten input storage unit 25. Note that the handwritten input saved data with Type being Image is naturally not data caused by handwritten input. Therefore, the handwritten input display control unit 23 may determine whether it is data caused by handwritten input according to the Type.

[0138] In Fig. 16, it is determined whether it is data caused by handwritten input based on the value of ColorId, but the handwritten input display control unit 23 may record whether it is data caused by handwritten input with a dedicated flag or the like. Also, although the Type is limited to handwritten data of strokes, it is also possible to determine whether it is data caused by handwritten input from the shape by machine learning. In this case, the learning device learns the correspondence between the teacher data of whether it is data caused by handwritten input or not and the shape of the characters by deep learning or the like, and outputs whether it is data caused by handwritten input or not based on the shape of the input characters.

[0139] Also, the data caused by handwritten input may include not only the text data obtained by character recognition and conversion of the handwritten data, but also stamps displayed as fixed characters or marks such as "completed" and "confidential", figures such as circles and stars, straight lines, etc., and data converted based on the user's operations.

[0140] The handwritten input saved data is used in steps S7 (pen coordinates and reception time) in Fig. 30, step S88 (acquisition of handwritten input saved data) in Fig. 37, step S96 (file saving) in Fig. 37, etc.

[0141] <Pen ID control data stored by the Pen ID control data storage unit> Next, with reference to FIG. 17, the pen ID control data will be described. FIG. 17 is a diagram for explaining the pen ID control data stored by the pen ID control data storage unit 36. The pen ID control data controls the color and the like of the data resulting from handwritten input. FIG. 17 shows that there are four pens 2500. One row in FIG. 17(a) shows the pen ID control data of one pen. Further, FIG. 17(b) is a diagram for explaining the angle information when the user writes by hand on the display device 2. The angle information can be said to be the angle in the direction where the user exists, the angle in the direction in which the pen is used, or the angle related to the rotation of the characters written by the user. With a predetermined direction (for example, the vertical and horizontal directions) of the display device 2 as 0 degrees (reference), the angle information of each user is 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees, 315 degrees in the counterclockwise direction.

[0142] Note that the angle information is not automatically determined by the user's position, and each user inputs (specifies) the angle information. The resolution of the specifiable angle information (45 degrees in FIG. 17) is just an example, and it may be smaller, such as 5 degrees to 30 degrees. However, it is considered that the user can read characters rotated by about 45 degrees.

[0143] The pen ID control data has PenId, ColorId, Angle, and AccountId. AccountId is associated with the pen used by the user for signing in. PenId is the identification information stored inside the pen 2500. ColorId indicates what color is selected by the user who uses this pen 2500 (which can be arbitrarily changed by the user). The color ID of the color definition data described later is set in ColorId. The details, thickness, etc. of each color are defined in the color definition data.

[0144] Angle is the angle information of the stroke set for this pen 2500 (which can be arbitrarily changed by the user). In the example of Fig. 17(a), the angle information of each pen is 0 degrees, 90 degrees, 180 degrees, and 270 degrees counterclockwise. AccountId is the identification information of the user. By associating the AccountId with the pen ID, the AccountId associated with the pen ID of the pen 2500 used by the user can be identified, and operation commands can be executed using user-defined data.

[0145] The pen ID control data 901 is the pen ID control data with PenId = 1. The color is black, the angle information is 0 degrees, and AccountId = 1. The user with AccountId = 1 is the user of the user-defined data 717 in Fig. 14. This indicates that this user signed in by handwriting a username, etc. with the pen with PenId = 1 and selected black. The pen ID control data without an AccountId indicates a signed-out state (not associated with a user).

[0146] The pen ID control data is used in step S5 (pen ID control data acquisition) in Fig. 30, step S20 (saving the angle information of the pen ID control data) in Fig. 32, step S21 (acquiring the angle information of the pen ID control data) in Fig. 32, and step S60 (pen ID control data acquisition) in Fig. 34.

[0147] <Color definition data> Fig. 18 shows an example of color definition data. One line of the color definition data defines one color. The color definition data defines the black-and-white highlighting (display method of the black-and-white pattern, width, and edged) on the black-and-white corresponding device for ColorId, and the color conversion method (display method of the color information and width on the color corresponding device). The color information is represented by # and eight-digit hexadecimal. The color information represents #R (red), G (green), B (blue), and A (transparency) every two digits, and px represents the pixel width. Note that the color definition data is only applicable to data caused by handwriting input.

[0148] The color definition data 1001 is an example of a definition where the ColorId is "Black". Pattern represents the content inside the stroke or the edge of the text. Edged represents the presence or absence of an edge. Color represents the RGBA color information. Width represents the line width. The color definition data 1001 indicates that in the black-and-white emphasis-compatible device, it displays the handwritten input saved data as solid black with a width of 5 pixels and no edge, and in the color-compatible device, it displays it with a black transparency of 0% and a width of 5 pixels. Similarly, the color definition data 1002 indicates that in the black-and-white emphasis-compatible device, it displays with a solid black width of 3 pixels + a 1-pixel-wide black edge from the outside + a 2-pixel-wide white edge, and in the color-compatible device, it displays with a red transparency of 0% on the color chart and a width of 5 pixels. The color definition data 1003 indicates that in the black-and-white emphasis-compatible device, it displays with a solid white width of 4 pixels + a 1-pixel-wide black edge, and in the color-compatible device, it displays with a blue transparency of 0% on the color chart and a width of 5 pixels. The color definition data 1004 indicates that in the black-and-white emphasis-compatible device, it displays with a black dot pattern width of 4 pixels + a 1-pixel-wide black edge, and in the color-compatible device, it displays with a green transparency of 0% on the color chart and a width of 5 pixels. The color definition data 1005 indicates that in the black-and-white emphasis-compatible device, it displays with a 5-pixel-wide black upper-right diagonal pattern where white is transparent and no border, and in the color-compatible device, it displays with a magenta transparency of 60% on the color chart. The color definition data 1006 indicates that in the black-and-white emphasis-compatible device, it displays with a 5-pixel-wide black horizontal line pattern where white is transparent, and in the color-compatible device, it displays with a cyan transparency of 60% on the color chart.

[0149] In this way, the color definition data has black-and-white emphasis and color-convertible data. The color definition data is held by the handwritten input storage unit 25 and is used for the conversion of the handwritten input saved data.

[0150] If the color definition data is defined as system definition data and user-defined data, and a form for changing the color definition data in the user-defined data is added, after signing in, the user can change it to their preferred color definition.

[0151] Figure 19 shows the data with black-and-white highlighting and the data displayed in a color display method using the color definition data of Figure 18. Figure 19(a) is what the black-and-white highlighting device highlighted in black and white, and Figure 19(b) is what the color-compatible device converted to color and displayed. For drawing convenience, Figure 19(b) is also in black and white (hatching).

[0152] Reference symbol C1 is the text of "black", "red", "blue", "green", "magenta", and "cyan", which is data caused by handwritten input. Reference symbol C2 is the stroke (handwriting) of "black", "red", "green", "magenta", and "cyan". Reference symbol C3 is a spring-shaped single-stroke (handwriting) stroke. Reference symbol C4 is a donut-shaped color chart (image). Reference symbol C5 is RGBCMY (text not caused by handwritten input) inside the color chart. Below reference symbol C3, there is the text "Transparent" in black, which is data caused by handwritten input.

[0153] The color chart of reference symbol C4 and the text RGBCMY of reference symbol C5 are data read from a PDF file or the like and are not data caused by handwritten input. Therefore, when described as the handwritten input storage data stored in the handwritten input storage unit 25 shown in Figure 16, the stroke (handwriting) of reference symbol C2 is stored as the handwritten input storage data 801 and 802 in Figure 16. The text that is the data caused by handwritten input of reference symbol C1 is stored as the handwritten input storage data 804 and 805 in Figure 16. The text that is not the data caused by handwritten input of reference symbol C5 is stored as the handwritten input storage data 806 in Figure 16. The image of reference symbol C4 is stored as handwritten input storage data such as the handwritten input storage data 807 in Figure 16. The color information of each handwritten input storage data is defined by ColorId.

[0154] As can be seen by comparing FIGS. 19(a) and (b), the red color of the text "R" inside the color chart of reference numeral C10 in FIG. 19(b) is not data resulting from handwritten input. For this reason, the text "R" is displayed in grayscale in FIG. 19(a). In contrast, since the text "red" of reference numeral C6 is data resulting from handwritten input, it is displayed in black-and-white highlighting in FIG. 19(a).

[0155] Also, since magenta and cyan are transparent colors, "Transparent" appears transparent in FIG. 19(b). Even in FIG. 19(a), since white is used as the transparent pattern for magenta and cyan, it is displayed transparently. In this way, the black-and-white highlighting device can black-and-white highlight the colors for the purpose of making them stand out, and the color device can convert them to colors for display.

[0156] The color definition data is used in step S6 (coordinate complement display data) of FIG. 30, step S82 (string object display data) of FIG. 37, and step S100 (object display data) of FIG. 37.

[0157] <Pen Color Selection Button Definition Data> Next, a method for a user to select a pen color will be described with reference to FIGS. 20 and 21. FIG. 20 is an example of pen color selection button definition data. The pen color selection button definition data is data that associates pen buttons displayed on the display 220 with ColorIds. The pen color selection button definition data has a pen button ID (PenButtonId), an icon file (Icon), a color ID (ColorId), and a font name (FontName).

[0158] FIG. 21 is an example of pen buttons 81 to 86 displayed on the display. The pen buttons 81 to 86 are displayed using the above icon file. It is assumed that the pen buttons 81 to 86 are arranged from left to right in ascending order of the pen button ID.

[0159] The pen color selection button definition data 1101 is the definition data of the pen button 81 displayed at the far left. When the user presses the pen button 81 with the pen 2500, the stroke data written by the pen 2500 will have a ColorId of "Black", and the font of the text after handwriting recognition will be "Mincho". The pen color selection button definition data 1101 is for the pen button 82 displayed second from the left, where the ColorId of the stroke data is "Red", and the font of the text after handwriting recognition is "Pop font". Similarly, the pen color selection button definition data 1103 - 1106 define the display position of the buttons, ColorId, and text font for pen buttons 83 to 86.

[0160] Figure 21(a) shows an example of the display of text fonts highlighted in black and white based on the pen color selection button definition data. Although the example of the text font display is for explanation purposes, it may be displayed when the user hovers the pen 2500. Figure 21(b) shows an example of the display of text fonts color - displayed by a color - corresponding device based on the pen color selection button definition data (it is actually in color but shown in black and white for drawing convenience).

[0161] As shown in Figure 21(a), not only by highlighting in black and white but also by changing the font, it becomes possible to highlight in black and white more effectively.

[0162] The pen color selection button definition data is also defined in system - defined data and user - defined data. If a form for changing the pen color selection button definition data is added to the user - defined data, it can be changed to a pen color selection button according to personal preferences after handwriting signature authentication. Therefore, the correspondence between ColorId and font is just an example.

[0163] The pen color selection button is used in step S6 (coordinate complement display data) of Figure 30, step S82 (string object display data) of Figure 37, and step S100 (object display data) of Figure 37.

[0164] <Example of display of selectable candidates> Figure 22 shows an example of an operation guide and selectable candidates 530 displayed by the operation guide. When the user writes a handwritten object 504 (due to the timeout of the selectable candidate display timer), the operation guide 500 is displayed. The operation guide 500 has an operation header 520, candidates 510 for operation commands, candidates 506 for handwritten recognition strings, candidates 507 for conversion strings, candidates 508 for string / prediction conversion, and a display 503 of the rectangular area of the handwritten object. The selectable candidates 530 are candidates 510 for operation commands, candidates 506 for handwritten recognition strings, candidates 507 for conversion strings, and candidates 508 for string / prediction conversion. Although there is no language conversion string in this example, it may be displayed. Also, the selectable candidates 530 excluding the candidates 510 for operation commands are called string candidates 539.

[0165] The operation header 520 has buttons 501, 509, 502, and 505. Button 501 accepts the switching operation between prediction conversion and kana conversion. In the example of Figure 22, when the user presses button 509 displayed as "Prediction", the handwriting input unit 21 accepts it and notifies the handwriting input display control unit 23 to that effect, and the display unit 22 changes the display to button 509 with "Kana". After the change, the string candidates 539 are arranged in descending order of the probability of "kana conversion".

[0166] Button 502 performs page operations for candidate display. In the example of Figure 22, there are 3 pages for candidate display, and currently the first page is being displayed. Button 505 accepts the deletion of the operation guide 500. When the user presses button 505, the handwriting input unit 21 accepts it and notifies the handwriting input display control unit 23 to that effect, and the display unit 22 deletes the display other than the handwritten object. Button 509 accepts batch display deletion. When the user presses button 509, the handwriting input unit 21 accepts it. The handwriting input unit 21 notifies the handwriting input display control unit 23 to that effect. The display unit 22 deletes all the displays shown in Figure 22 including the handwritten object, enabling the user to rewrite the handwriting from the beginning.

[0167] The handwritten object 504 is the character "gi" handwritten by the user. A handwritten object rectangular area display 503 surrounding the handwritten object 504 is displayed. The display procedure will be described with the sequence diagrams in FIGS. 30 to 37. In the example of FIG. 22, the handwritten object rectangular area display 503 is displayed with a dotted line frame.

[0168] The handwritten recognition string candidates 506, the conversion string candidates 507, and the string / predicted conversion candidates 508 each have their respective string candidates arranged in descending order of probability. The "gi" in the handwritten recognition string candidates 506 is a candidate for the recognition result. In this example, the display device 2 correctly recognizes "gi".

[0169] The conversion string candidates 507 are conversion string candidates converted from language string candidates. In this example, "giryou shisaku" is an abbreviation for "gijutsu ryousan shisaku". The string / predicted conversion candidates 508 are predicted string candidates converted from language string candidates or conversion string candidates. In this example, "Approve giryou shisaku" and "Destination of the minutes" are displayed.

[0170] The operation command candidates 510 are operation command candidates selected based on the operation command definition data 701 to 703, 709 to 716 in FIG. 11(a). In the example of FIG. 22, the leading character "》" 511 indicates that it is an operation command candidate. In FIG. 22, there is no selection object to be selected for the handwritten object 504 which is "gi", and the "minutes" which is a string candidate of "gi" partially matches the operation command definition data 701 and 702 in FIG. 11(a), so it is displayed as the operation command candidate 510.

[0171] When the user selects "Load minutes template", the operation command defined in the operation command definition data 701 is executed, and when the user selects "Save to the minutes folder", the operation command defined in the operation command definition data 702 is executed. Thus, the operation command candidates are displayed when operation command definition data including the converted string is found, so they are not always displayed.

[0172] As shown in FIG. 22, since the character string candidates and the operation command candidates are displayed simultaneously (together), the user can arbitrarily select either the character string candidate or the operation command that the user intends to input.

[0173] <Example of Designating a Selection Object> In the display device 2 of the present embodiment, the user can designate a selection object by selecting a determined object by handwriting. The selection object is an object to be edited or decorated.

[0174] FIG. 23 is an example of a diagram for explaining an example of designating a selection object. In FIG. 23, the handwritten object 11 is shown by a solid black line, the rectangular area 12 of the handwritten object is shaded in gray, the determined object 13 is shown by a black line, and the rectangular area 14 of the selection object is shown by a dotted line. Note that the lowercase alphabets added to the reference numerals are used to distinguish each of them. Further, as a determination condition (whether or not they are in a predetermined relationship) for determining the determined object as a selection object, the crossing line determination condition 406 or the surrounding line determination condition 407 of the defined control data shown in FIG. 7 is used.

[0175] FIG. 23(a) is an example in which the user designates two horizontally written determined objects 13a and 13b with a crossing line (handwritten object 11a). In this example, the length H1 of the short side and the length W1 of the long side of the rectangular area 12a of the handwritten object satisfy the conditions of the crossing line determination condition 406, and the overlapping rate with the determined objects 13a and 13b satisfies the conditions of the crossing line determination condition 406. For this reason, both of the determined objects 13a and 13b of "minutes" and "giji" are designated as selection objects.

[0176] FIG. 23(b) is an example in which the user designates a horizontally written determined object 13c with a surrounding line (handwritten object 11b). In this example, only the determined object 13c of "minutes", whose overlapping rate with the rectangular area 12c of the handwritten object satisfies the conditions of the surrounding line determination condition 407, is designated as a selection object.

[0177] Figure 23(c) shows an example in which a plurality of vertical written determined objects 13d and 13e are specified by a crossing line (handwritten object 11c). In this example, similar to Figure 23(a), the length H1 of the short side and the length W1 of the long side of the handwritten object rectangular area 12d satisfy the conditions of the crossing line determination condition 406, and the overlapping rate with each of the two determined objects 13d and 13e of "minutes" and "proceedings" satisfies the conditions of the crossing line determination condition 406. Therefore, both the determined objects 13d and 13e of "minutes" and "proceedings" are specified as selected objects.

[0178] Figure 23(d) shows an example in which a user specifies a vertical written determined object 13f with a surrounding line (handwritten object 11d). In this example, similar to Figure 23(b), only the determined object 13f of "minutes" is specified as a selected object.

[0179] <Display Example of Operation Command Candidates> Figure 24 shows a display example of operation command candidates based on operation command definition data when there is a handwritten object shown in Figure 13. Figure 24(a) shows candidates for editing system operation commands, and Figure 24(b) shows candidates for formatting system operation commands. Also, Figure 24(a) shows an example in which a selected object is specified by the handwritten object 11a in Figure 23(a).

[0180] As shown in Figures 24(a) and 24(b), the operation command candidates displayed following the line start character "》" 511 are the main menu 550. The main menu 550 displays the name of the last executed operation command or the name of the first operation command in the operation command definition data. The line start character "》" 511a in the first line is a candidate for an editing system operation command, and the line start character "》" 511b in the second line is a candidate for a formatting system operation command.

[0181] The ">" 512 at the end of the line indicates the presence of a sub-menu (an example of a sub-menu button). The ">" 512a on the first line displays the sub-menu (the last selected) that is a candidate for the editing operation command. The ">" 512b on the second line displays the remaining sub-menus that are candidates for the formatting operation commands. When the user presses the ">" 512, the sub-menu 560 is displayed on its right side. All the operation commands defined in the operation command definition data are displayed in the sub-menu 560. In the display example of FIG. 24(a), the sub-menu 560 corresponding to the ">" 512a on the first line is also displayed since the main menu is displayed. The sub-menu 560 may be displayed by pressing the ">" 512a on the first line.

[0182] When the user presses any operation command name with the pen, the Command of the operation command definition data associated with the operation command name is executed by the handwritten input display control unit 23 for the selected object. That is, when "Delete" 521 is selected, "Delete" is executed, when "Move" 522 is selected, "Move" is executed, when "Rotate" 523 is selected, "Rotate" is executed, and when "Select" 524 is selected, "Select" is executed respectively.

[0183] For example, when the user presses "Delete" 521 with the pen, "Proceedings" and "Minutes" can be deleted, and when "Move" 522, "Rotate" 523, or "Select" 524 is pressed, a bounding box (the circumscribed rectangle of the selected object) is displayed. When the user presses "Move" 522 and "Rotate" 523, the bounding box can be moved or rotated by the drag operation of the pen. When the user presses "Select" 524, other operations on the bounding box can be executed.

[0184] The strings "—" 541, "—," 542, "~" 543, "→" 544, "⇒" 545, which are string candidates other than the operation command candidates, are the recognition results of the strikethrough (handwritten object 11a). When the user intends to input a string instead of an operation command, the string candidate can be selected.

[0185] In Fig. 24(b), when the ">" 512b in the second row is pressed, the sub-menu in Fig. 24(b) is displayed. Similar to Fig. 24(a), the main menu 550 and the sub-menu 560 are displayed in the display example of Fig. 24(b). Based on the operation command definition data in Fig. 13, when "Thick" 531 is selected, the handwriting input display control unit 23 executes "Thick" on the selected object. When "Thin" 532 is selected, the handwriting input display control unit 23 executes "Thin" on the selected object. When "Large" 533 is selected, the handwriting input display control unit 23 executes "Large" on the selected object. When "Small" 534 is selected, the handwriting input display control unit 23 executes "Small" on the selected object. When the underline 535 is selected, the handwriting input display control unit 23 executes "Underline" on the selected object.

[0186] Note that when "Thick" 531 is selected, how thick to make it, when "Thin" 532 is selected, how thin to make it, when "Large" 533 is selected, how large to make it, when "Small" 534 is selected, how small to make it, and the line type, etc. when the underline 535 is selected are defined separately as fixed values. Alternatively, it would be even better if when the sub-menu in Fig. 24(b) is selected, a separate selection menu is opened so that the user can adjust it.

[0187] When the user presses "Thick" 531 with the pen, the handwriting input display control unit 23 thickens the lines that make up the determined objects 13a and 13b of "Minutes" and "Draft". When the user presses "Thin" 532 with the pen, the handwriting input display control unit 23 can thin the lines that make up "Minutes" and "Draft". When the user presses "Large" 533 with the pen, the handwriting input display control unit 23 can enlarge it. When the user presses "Small" 534 with the pen, the handwriting input display control unit 23 can reduce it. When the user presses "Underline" 535 with the pen, the handwriting input display control unit 23 can add an underline.

[0188] FIG. 25 shows an example of displaying candidates for operation commands based on operation command definition data when there is a handwritten object shown in FIG. 13. The difference from FIGS. 24(a) and (b) is that FIGS. 25(a) and (b) show an example in which the selected object is specified by the handwritten object 11b (enclosing line) in FIG. 23(b). As can be seen by comparing FIGS. 24 and 25, there is no difference in the candidates for operation commands displayed depending on whether the handwritten object is a line or an enclosing line. This is because when a selected object is specified, the handwritten input display control unit 23 causes the display unit 22 to display candidates for operation commands. However, the handwritten input display control unit 23 may recognize the handwritten object and change the candidates for operation commands to be displayed according to the handwritten object. In this case, operation command definition data like that in FIG. 13 is associated with the recognized handwritten object (such as a single stroke or a circle).

[0189] In FIG. 25, the "○" 551, "∞" 552, "0" 553, "00" 554, and "ロ" 555, which are string candidates other than candidates for operation commands, are recognition results of the enclosing line (handwritten object 11b). When the user intends to input a string rather than an operation command, the string candidates can be selected.

[0190] <Example of Inputting Angle Information> Next, the method of inputting angle information will be described with reference to FIG. 26. FIG. 26 is an example of a diagram for explaining the method of inputting angle information. FIG. 26 explains the case where a user existing in the 3 o'clock direction of the display device 2 inputs angle information. Since characters or the like handwritten from the 3 o'clock direction are correctly recognized when rotated 90 degrees clockwise, it is preferable that 90-degree angle information be input.

[0191] FIG. 26(a) shows a state in which, with the angle information of the pen ID control data being 0 degrees (initial value), a user existing in the 3 o'clock direction of the display device 2 handwrites "ぎ", and thus the operation guide 500 is displayed. Since the display device 2 performs character recognition on the "ぎ" handwritten from the 3 o'clock direction with the angle information remaining 0 degrees, selectable candidates 530 different from what is expected are displayed.

[0192] When inputting angle information, the user draws a straight line in the operation guide 500 from top to bottom as seen by the user. FIG. 26(b) shows an example of this straight line 521. The angle α formed counterclockwise between the 6 o'clock direction where the angle information is 0 degrees and this straight line 521 is the angle information. That is, the angle α formed counterclockwise between the straight line 522 drawn downward from the starting point S in the 6 o'clock direction and the straight line 521 input by the user is the angle information. Briefly, the end point direction of the straight line 521 is the angle information. Therefore, the angle information input by the user in FIG. 26(b) is 90 degrees.

[0193] Note that for the detection of a straight line, for example, there is a method of converting the coordinates from the starting point S to the end point E into a straight line by the least squares method, and comparing the obtained correlation coefficient with a threshold value to determine whether it is a straight line.

[0194] Immediately after the user starts writing the straight line 521 (immediately after the pen 2500 touches the starting point S of the straight line 521), the display device 2 erases the operation guide 500. Also, immediately after finishing writing the straight line 521 (immediately after the pen 2500 leaves the end point E of the straight line 521), the display device 2 searches for the value closest to the above-mentioned formed angle α from 45 degrees, 90 degrees, 135 degrees, 180 degrees, 215 degrees, 270 degrees, 315 degrees, 360 degrees and determines it as the angle information. The formed angle α itself may also be the angle information. The determined angle information is set in the Angle of the pen ID control data. When the pen tip of the pen 2500 is pressed for handwriting or the like, the pen event transmission unit 41 of the pen 2500 transmits the pen ID to the display device 2, so the display device 2 can associate the angle information with the pen ID control data.

[0195] Note that the user can input angle information only by handwriting a straight line in the operation guide 500. As a result, when the user handwrites a straight line outside the operation guide 500, it is recognized as "1" or "—", etc., and when handwriting a straight line in the operation guide 500, angle information can be input. That is, the handwriting recognition control unit 26 detects a straight line from a predetermined range and converts the stroke data handwritten outside the predetermined range into text data.

[0196] FIG. 26(c) shows the operation guide 500 immediately after the operation of FIG. 26(b). Since 90 degrees is set as the angle information (Angle) in the pen ID control data, the handwritten object (stroke data) is rotated 90 degrees clockwise internally for handwriting recognition, and the operation guide 500 is rotated 90 degrees counterclockwise and displayed. Note that the angle information may be manually input by the user from the menu.

[0197] <Example of registration of handwritten signature data> Next, an example of registering handwritten signature data will be described with reference to FIG. 27. FIG. 27 is a diagram for explaining the method of registering handwritten signature data. First, FIG. 27(a) is an example of selectable candidates 530 displayed when the user handwrites "Signature". Two operation commands 513 and 514, namely "Register handwritten signature" and "Log out of handwritten signature", based on the operation command definition data 713 and 715, and string candidates such as "Signature", "Signature meeting", and "With signature" that partially match the string "Signature" are displayed. The two operation commands 513 and 514 are displayed because the String in the operation command definition data 713 and 715 in FIG. 11 has "Signature".

[0198] When the user presses "Register handwritten signature" with the pen 2500, the handwritten signature registration form 561 in FIG. 27(b) is added to the handwritten input storage unit 25 and displayed on the operation screen 101. For example, the operation guide 500 in FIG. 27(a) is erased, and the handwritten signature registration form 561 is displayed at the same location as the operation guide 500. The handwritten signature registration form 561 has, from the upper part, a name input field 561a, signature input fields 561b to 561d, and a registration confirmation field 561e. The user inputs the text of the name in the name input field 561a, the first handwritten signature, the second handwritten signature, and the third handwritten signature in the signature input fields 561b to 561d, and a check mark or a cancellation mark in the registration confirmation field 561e. The text of the name is the display name of this user and is converted into text data. The reason for inputting the handwritten signature three times is to register the feature amounts on the premise that the handwritten signature is different each time the user writes and does not exactly match.

[0199] Note that a handwritten signature is generally characters related to the user, such as a user name, etc. In addition to the user name, it may also be numbers such as an employee number, a nickname, a portrait, etc. Further, the handwritten signature is not limited to characters related to the user, and any handwritten object may be sufficient. For example, the handwritten signature may be a circle, a triangle, a square, a symbol, or a combination thereof. Since the handwritten signature is not only characterized by coordinates, even if users with the same surname (e.g., Mr. Suzuki) both register a handwritten signature of "Suzuki", it can be correctly authenticated.

[0200] When the user writes by hand as instructed on the handwritten signature registration form 561, it becomes the handwritten signature registration form 561 shown in FIG. 27(c). When the user handwrites a "check mark" in the registration confirmation field 561e, the handwritten signature data is registered in the handwritten signature data storage unit 39 and the handwritten signature registration form 561 is deleted. A SignatureId is assigned by the registration, and the text of the name in the name input field 561a corresponding to the similarly assigned AccountId is registered in the user-defined data in association with the SignatureId. When the user handwrites a user name or the like to sign in, the SignatureId associated with the AccountId in the user-defined data is acquired and registered in the pen ID control data in association with the pen ID of the pen 2500 used for the handwritten signature. Thereafter, if the user uses the pen 2500, the pen ID is transmitted to the display device 2, the AccountId associated with the pen ID in the pen ID control data can be specified, and the execution of the operation command using the user-defined data becomes possible without the user being aware of it.

[0201] When the user handwrites an "X" in the registration confirmation field 561e, the handwritten signature registration is canceled and the handwritten signature registration form 561 is deleted. Note that when any error occurs regarding the registration, the fact is displayed in the system reservation area of the operation screen 101 or the like.

[0202] In this way, the handwritten input display control unit 23 can accept handwritten input without distinguishing between handwritten input to a form and handwritten input outside the form.

[0203] <An example of sign-in by handwriting> Next, with reference to FIG. 28, a method for a user to sign in after registering handwritten signature data will be described. FIG. 28 is an example of an operation guide 500 displayed when a user writes "Suzuki", which is the handwritten signature data registered by the user. Since "Suzuki" is registered in the handwritten signature data storage unit 39 as handwritten signature data, it matches the handwritten signature data. For this reason, an operation command 512 of "sign in by handwriting" is displayed.

[0204] Also, since the handwritten signature data matches, a SignatureId representing "Suzuki" is specified, and user-defined data having an AccountId associated with this SignatureId is specified.

[0205] When the user selects the operation command 512 of "sign in by handwriting", the AccountId of "Suzuki" is associated with the pen ID of the pen 2500 being used and added to the pen ID control data, and the user-defined data of "Suzuki" is used when the operation command is used.

[0206] The registration of handwritten signature data using the handwritten signature registration form 561 in FIG. 27 is controlled in the same way as normal handwritten input such as characters. For this reason, the handwritten signature registration form 561 is displayed on the same operation screen as the operation screen where characters and the like are handwritten. There is no difference in the handwritten operation inside and outside the handwritten signature registration form 561, and the user can complete the input to the handwritten signature registration form 561 by simply performing handwritten input in the area separated by the grid lines of the handwritten signature registration form 561.

[0207] <An example of an operation for changing user-defined data> Next, using FIG. 29, the method for changing user-defined data will be described. FIG. 29 is a diagram for explaining the method for changing user-defined data. FIG. 29(a) is an example of the operation guide 500 displayed when the user writes "se" by hand. In the operation command definition data 716 shown in FIG. 11, "Settings" is defined in the String, and since "Settings" is in the predicted string of "se", the operation command "Change settings" is displayed.

[0208] When the user selects "Change settings" with the pen 2500 that has signed in by hand, the AccountId of the user associated with the pen ID of this pen 2500 in the pen ID control data is identified. As a result, the user-defined data of the signed-in user is identified, and the user-defined data change form 562 in FIG. 29(b) is added to the handwritten input storage unit 25. The user-defined data change form 562 is displayed on the operation screen 101. In the example of FIG. 29, the user-defined data change form 562 is created according to the user-defined data 718 in FIG. 14. The user-defined data change form 562 has a name field 562a, a password field 562b, a folder username field 562c, a folder password field 562d, a folder file name field 562e, and a register or cancel field 562f.

[0209] Note that if the user has not signed in by hand in advance, the display device 2 cannot identify the user's AccountId, resulting in an error, and an error message is displayed in the system reservation area of the operation screen 101 or the like.

[0210] The user handwrites the password in the password field 562b of the user-defined data change form 562 in FIG. 29(b). The user handwrites the folder user name in the folder user name field 562c. The user handwrites the folder password in the folder password field 562d. The user handwrites the folder file name in the folder file name field 562e. The user handwrites a "check mark" or "x" in the registration / cancellation field 562f. By doing so, the change of user-defined data is executed and the user-defined data change form 562 is deleted.

[0211] In this way, the user can display the user-defined data change form 562 and arbitrarily change the user-defined data by handwriting the stroke data for calling the user-defined data change form 562. The handwritten input display control unit 23 accepts handwritten input without distinguishing between handwritten input to the form and handwritten input outside the form.

[0212] Note that the AccountUsername included in the user-defined data is automatically displayed in the name field 562a. Also, the user-defined data change form 562 can be used not only for changes but also for registration.

[0213] The change of user-defined data using the user-defined data change form 562 in FIG. 29 is controlled in the same way as normal handwritten input such as characters. For this reason, the user-defined data change form 562 is displayed on the same operation screen as the operation screen where characters etc. are handwritten. There is no difference in the handwritten operation inside and outside the user-defined data change form 562, and the user can complete the input to the user-defined data change form 562 simply by handwriting in the area separated by the ruled lines of the user-defined data change form 562.

[0214] <Operation Procedure> The operation of the display device 2 will be described using the above configuration and FIGS. 30 to 37. FIGS. 30 to 37 are sequence diagrams showing an example of the process in which the display device 2 displays candidates for character strings and operation commands. The process in FIG. 30 starts when the display device 2 is activated (when the application is activated). Note that in FIGS. 30 to 37, for the sake of space, the functions in FIG. 6 are indicated by reference numerals.

[0215] Before starting the input of handwritten data, the user has already selected the pen buttons 81 to 86 (the PenId has been specified). Therefore, a. The pen button ID, ColorId, and font are specified in the pen color selection button definition data, b. The PenId and ColorId are registered in the pen ID control data.

[0216] S1: First, the handwritten input display control unit 23 sends a handwritten object start to the handwritten input storage unit 25. The handwritten input storage unit 25 secures a handwritten object area (a memory area for storing handwritten objects). The user may secure the handwritten object area after contacting the pen with the handwritten input unit 21.

[0217] S2: Next, the user contacts the pen with the handwritten input unit 21. The handwritten input unit 21 detects the pen down and sends it to the handwritten input display control unit 23.

[0218] S3: The handwritten input display control unit 23 sends a stroke start to the handwritten input storage unit 25, and the handwritten input storage unit 25 secures a stroke area.

[0219] S4: When the user moves the pen while keeping it in contact with the handwritten input unit 21, the handwritten input unit 21 sends the pen coordinates to the handwritten input display control unit 23.

[0220] S5: The handwritten input display control unit 23 specifies the pen ID received from the pen 2500 at the same time as the input of the coordinates, and acquires the current pen ID control data stored in the pen ID control data storage unit 36. Since the pen ID is transmitted when the coordinates are input, the stroke and the pen ID are associated with each other. The pen ID control data storage unit 36 transmits the pen ID control data (ColorId (the angle information is still zero and there is no AccountId yet)) to the handwritten input display control unit 23. Note that the angle information is still the initial value of zero. Also, since the user is not signed in, there is no AccountId.

[0221] S6: The handwritten input display control unit 23 transmits pen coordinate interpolation display data (data for interpolating discrete pen coordinates) to the display unit 22. The display unit 22 interpolates the pen coordinates with the pen coordinate interpolation display data, specifies the line type and thickness from the color definition data based on the ColorId, and displays the stroke.

[0222] S7: The handwritten input display control unit 23 transmits the pen coordinates, the reception time thereof, the ColorId, and the angle information to the handwritten input storage unit 25. The handwritten input storage unit 25 adds the pen coordinates to the stroke. While the user is moving the pen, the handwritten input unit 21 periodically repeats the transmission of the pen coordinates to the handwritten input display control unit 23, so the processes of steps S4 to S7 are repeated until the pen is lifted up.

[0223] S8: When the user releases the pen from the handwritten input unit 21, the handwritten input unit 21 transmits a pen up to the handwritten input display control unit 23.

[0224] S9: The handwritten input display control unit 23 transmits a stroke end to the handwritten input storage unit 25, and the handwritten input storage unit 25 determines the pen coordinates of the stroke. With the determination of the pen coordinates of the stroke, pen coordinates cannot be added to the stroke hereafter.

[0225] S10: Next, based on the handwritten object vicinity rectangular area 403, the handwritten input display control unit 23 transmits to the handwritten input storage unit 25 a request to obtain the overlapping situation between the handwritten object vicinity rectangular area and the stroke rectangular area. The handwritten input storage unit 25 calculates the overlapping situation and transmits it to the handwritten input display control unit 23.

[0226] Subsequent steps S11 - S17 are executed when the handwritten object vicinity rectangular area and the stroke rectangular area do not overlap.

[0227] S11: When the handwritten object vicinity rectangular area and the stroke rectangular area do not overlap, since one handwritten object is determined, the handwritten input display control unit 23 transmits a hold data clear to the handwritten recognition control unit 26.

[0228] S12 - S14: The handwritten recognition control unit 26 transmits the hold data clear to the character string conversion control unit 28, the prediction conversion control unit 30, and the operation command recognition control unit 32 respectively. The handwritten recognition control unit 26, the character string conversion control unit 28, the prediction conversion control unit 30, and the operation command recognition control unit 32 clear the data related to the character string candidates and operation command candidates that they have held so far. Note that at the time of clearing, the last handwritten stroke has not been added to the handwritten object.

[0229] S15: The handwritten input display control unit 23 transmits a handwritten object end to the handwritten input storage unit 25. The handwritten input storage unit 25 finalizes the handwritten object. Finalizing the handwritten object means that one handwritten object is completed (no more strokes are added).

[0230] S16: The handwritten input display control unit 23 transmits a handwritten object start to the handwritten input storage unit 25. In preparation for the start of handwriting (pen down) of the next handwritten object, the handwritten input storage unit 25 secures a new handwritten object area.

[0231] S17: Next, the handwritten input display control unit 23 transmits a stroke addition to the handwritten input storage unit 25 regarding the stroke that ended in step S9. When steps S11 to S17 are executed, the added stroke is the first stroke of the handwritten object, and the handwritten input storage unit 25 adds the stroke data to the ongoing handwritten object. When steps S11 to S17 are not executed, the added stroke is added to the handwritten object that is already being handwritten.

[0232] S18: Subsequently, the handwritten input display control unit 23 transmits a stroke addition to the handwritten recognition control unit 26. The handwritten recognition control unit 26 adds the stroke data to the stroke data holding area (the area where the stroke data is temporarily stored) where the character string candidates are stored.

[0233] S19: The handwritten recognition control unit 26 performs gesture handwritten recognition on the stroke data holding area. Gesture handwritten recognition means recognizing angle information from a straight line. Since the gesture handwritten recognition is performed inside the operation guide 500, the handwritten recognition control unit 26 detects the straight line inside the operation guide 500. The position information of the operation guide 500 is transmitted to the handwritten recognition control unit 26 in step S67 described later.

[0234] S20: When a straight line inside the operation guide 500 is detected, the angle α formed counterclockwise between the straight line 522 drawn in the 6 o'clock direction from the starting point of the straight line and the straight line 521 input by the user is determined in 45-degree units. Then, the handwritten recognition control unit 26 stores the determined angle information in the pen ID control data storage unit 36 in association with the pen ID that the stroke data of the straight line 521 has. Note that step S20 is executed when a straight line is detected inside the operation guide 500.

[0235] S20-2: The handwritten recognition control unit 26 clears the selectable candidate display rectangle.

[0236] S21: Next, the handwriting recognition control unit 26 designates the pen ID received from the handwriting input unit 21 and acquires the angle information of the current pen ID control data from the pen ID control data storage unit 36.

[0237] S22: The handwriting recognition control unit 26 rotates the stroke data in the stroke data holding area clockwise by the acquired angle information.

[0238] S23: The handwriting recognition control unit 26 transmits the rotated stroke data to the handwriting signature authentication control unit 38. In this way, the stroke data is always transmitted to the handwriting signature authentication control unit 38 in a state where it is unknown whether it is a handwriting signature or not.

[0239] S24: The handwriting signature authentication control unit 38 receives the stroke data and receives the registered handwriting signature data from the handwriting signature data storage unit 39. Then, the handwriting signature authentication control unit 38 performs a comparison (matching) between the stroke data and the handwriting signature data, and holds the authentication result of the handwriting signature so that the authentication result of the handwriting signature can be obtained in the subsequent step S61. If the authentication is successful, the AccountId is registered in the pen ID control data.

[0240] S25: Next, the handwriting recognition control unit 26 performs handwriting recognition on the stroke data. If it is a "check mark" or "x" in the registration or cancellation field of the form, the form is processed; otherwise, normal handwriting recognition processing is performed.

[0241] S26: When the registration or cancellation field in the handwriting signature data registration form is a "check mark", the handwriting recognition control unit 26 transmits the handwriting signature data (stroke data) input by the user for the handwriting signature registration form to the handwriting signature authentication control unit 38. The handwriting signature registration form is generated by the handwriting input display control unit 23 in the handwriting input storage unit 25 in step S86 described later.

[0242] S27: The handwritten signature authentication control unit 38 registers the received handwritten signature data (stroke data) in the handwritten signature data storage unit 39. As a result, a SignatureId is assigned. The SignatureId is returned to the handwriting recognition control unit 26. When the name entered in the name input field 561a of the handwritten signature registration form 561 and the SignatureId are not in the user-defined data, the handwriting recognition control unit 26 newly adds user-defined data. Also, the handwriting recognition control unit 26 assigns an AccountId and stores the SignatureId in the user-defined data. When the name entered in the name input field 561a is in the user-defined data, the SignatureId is stored in the user-defined data. By this process, the AccountId and the SignatureId are associated. Note that when new user-defined data is added, other values are not set, but the user can register and change from the user-defined data change form 562.

[0243] S28: By registering the handwritten signature data, the handwriting recognition control unit 26 deletes the handwritten signature registration form 561 from the handwriting input storage unit 25.

[0244] S29: When the registration or cancellation field in the user-defined data change form is a "check mark", the handwriting recognition control unit 26 sends the change value entered in the user-defined data change form 562 to the operation command definition unit 33. The user-defined data change form 562 is generated by the handwriting input display control unit 23 in the handwriting input storage unit 25 in step S86 described later.

[0245] S30: By executing the change of the user-defined data, the handwriting recognition control unit 26 deletes the user-defined data change form 562 from the handwriting input storage unit 25.

[0246] S31: When the registration or cancellation field in the form added in step S86 described later is "×", the handwriting recognition control unit 26 deletes the form added in step S86 from the handwriting input storage unit 25.

[0247] S33: If it is not form processing, the handwritten recognition control unit 26 transmits the handwritten recognition character string candidate, which is the execution result, to the handwritten recognition dictionary unit 27. The handwritten recognition dictionary unit 27 transmits a linguistically plausible language character string candidate to the handwritten recognition control unit 26.

[0248] S34: The handwritten recognition control unit 26 transmits the handwritten recognition character string candidate and the received language character string candidate to the character string conversion control unit 28.

[0249] S35: The character string conversion control unit 28 transmits the handwritten recognition character string candidate and the language character string candidate to the character string conversion dictionary unit 29. The character string conversion dictionary unit 29 transmits a converted character string candidate to the character string conversion control unit 28.

[0250] S36: The character string conversion control unit 28 transmits the received converted character string candidate to the prediction conversion control unit 30.

[0251] S37: The prediction conversion control unit 30 transmits the received converted character string candidate to the prediction conversion dictionary unit 31. The prediction conversion dictionary unit 31 transmits a predicted character string candidate to the prediction conversion control unit 30.

[0252] S38: The prediction conversion control unit 30 transmits the received predicted character string candidate to the operation command recognition control unit 32.

[0253] S39: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits candidates for operation commands to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can obtain candidates for operation commands corresponding to the operation command definition data having a character string (String) that matches the predicted character string candidate.

[0254] The display device 2 performs the same processing until the candidates for the operation commands described in the subsequent steps S40 to S47 are transmitted.

[0255] S40: The character string conversion control unit 28 transmits the received candidate conversion character string to the operation command recognition control unit 32.

[0256] S41: The operation command recognition control unit 32 transmits the received candidate conversion character string to the operation command definition unit 33. The operation command definition unit 33 transmits candidates for operation commands to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can obtain candidates for operation commands corresponding to operation command definition data having a character string (String) that matches the candidate conversion character string.

[0257] S42: The handwriting recognition control unit 26 transmits the candidate handwriting recognition character string and the candidate language character string to the prediction conversion control unit 30.

[0258] S43: The prediction conversion control unit 30 transmits the candidate handwriting recognition character string and the received candidate language character string to the prediction conversion dictionary unit 31. The prediction conversion dictionary unit 31 transmits candidate prediction character strings to the prediction conversion control unit 30.

[0259] S44: The prediction conversion control unit 30 transmits the received candidate prediction character string to the operation command recognition control unit 32.

[0260] S45: The operation command recognition control unit 32 transmits the received candidate prediction character string to the operation command definition unit 33. The operation command definition unit 33 transmits candidates for operation commands to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can obtain candidates for operation commands corresponding to operation command definition data having a character string (String) that matches the candidate prediction character string.

[0261] S46: The handwriting recognition control unit 26 transmits the candidate handwriting recognition character string and the received candidate language character string to the operation command recognition control unit 32.

[0262] S47: The operation command recognition control unit 32 transmits the handwritten recognition character string candidates and the received language character string candidates to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidates to the operation command recognition control unit 32. Thereby, the operation command recognition control unit 32 can obtain the operation command candidates corresponding to the operation command definition data having a character string (String) that matches the language character string candidates.

[0263] S48: Next, the handwriting recognition control unit 26 transmits the stroke addition to the operation command recognition control unit 32.

[0264] S49: The operation command recognition control unit 32 transmits the acquisition of the position information of the determined object to the handwritten input storage unit 25. The handwritten input storage unit 25 transmits the position information of the determined object to the operation command recognition control unit 32.

[0265] S50: The operation command recognition control unit 32 determines whether the position information of the stroke received from the handwriting recognition control unit 26 in the stroke addition of step S48 is in a predetermined relationship with the position information of the determined object received from the handwritten input storage unit 25 based on the crossing line determination condition 406 and the surrounding line determination condition 407. Thereby, the operation command recognition control unit 32 determines the selection object. The operation command recognition control unit 32 stores the determined object that can be determined to be selected as the selection object. Also, in this case, since the selection object is specified, the operation command candidates of the input / output system are obtained from the operation command definition unit 33.

[0266] Also, the handwriting recognition control unit 26, the character string conversion control unit 28, the prediction conversion control unit 30, and the operation command recognition control unit 32 each hold the handwritten recognition character string candidates, language character string candidates, conversion character string candidates, prediction character string candidates, operation command candidates, and data related to the selection object so that they can be obtained in the subsequent steps S55 to S58.

[0267] After the handwriting input display control unit 23 transmits the stroke addition to the handwriting recognition control unit 26 in step S18, it immediately transmits the start of the selectable candidate display timer to the candidate display timer control unit 24. The candidate display timer control unit 24 starts this timer.

[0268] Subsequently, steps S51 to S53 are executed when a pen down occurs before a certain period of time has elapsed (before the timer times out).

[0269] S51: If the user touches the pen to the handwriting input unit 21 before the timer times out, the handwriting input unit 21 transmits a pen down (the same event as in step S2) to the handwriting input display control unit 23.

[0270] S52: The handwriting input display control unit 23 transmits a stroke start (the same as in step S3) to the handwriting input storage unit 25. The subsequent sequence is the same as after step S3.

[0271] S53: Further, the handwriting input display control unit 23 transmits the stop of the selectable candidate display timer to the candidate display timer control unit 24. The candidate display timer control unit 24 stops the timer. This is because the timer is no longer needed since a pen down has been detected.

[0272] Steps S54 to S103 are executed when a pen down does not occur before a certain period of time has elapsed (before the timer times out). Therefore, the operation guide 500 shown in FIG. 22 is displayed.

[0273] S54: If the user does not touch the pen to the handwriting input unit 21 while the selectable candidate display timer is running, the candidate display timer control unit 24 transmits a timeout to the handwriting input display control unit 23.

[0274] S55: The handwritten input display control unit 23 transmits a request for obtaining a handwritten recognition string / language string candidate to the handwritten recognition control unit 26. The handwritten recognition control unit 26 transmits the currently held handwritten recognition string / language string candidate to the handwritten input display control unit 23.

[0275] S56: The handwritten input display control unit 23 transmits a request for obtaining a conversion string candidate to the character conversion control unit 28. The character conversion control unit 28 transmits the currently held conversion string candidate to the handwritten input display control unit 23.

[0276] S57: The handwritten input display control unit 23 transmits a request for obtaining a predicted string candidate to the prediction conversion control unit 30. The prediction conversion control unit 30 transmits the currently held predicted string candidate to the handwritten input display control unit 23.

[0277] S58: The handwritten input display control unit 23 transmits a request for obtaining a candidate for an operation command to the operation command recognition control unit 32. The operation command recognition control unit 32 transmits the currently held candidate for the operation command and the selected object to the handwritten input display control unit 23.

[0278] S59: Further, the handwritten input display control unit 23 transmits a request for obtaining an estimated writing direction to the handwritten input storage unit 25. The handwritten input storage unit 25 determines the estimated writing direction based on the stroke addition time, horizontal distance, and vertical distance of the handwritten object rectangular area, and transmits the estimated writing direction to the handwritten input display control unit 23.

[0279] S60: Next, the handwritten input display control unit 23 specifies the pen ID received from the handwritten input unit 21, and obtains the ColorId, AccountId, and angle information of the current pen ID control data from the pen ID control data storage unit 36.

[0280] S61: The handwriting input display control unit 23 acquires the authentication result of the handwritten signature from the handwritten signature authentication control unit 38. As a result, the user's SignatureId can be obtained. Therefore, when executing the operation commands described later, the AccountId is registered in the pen ID control data in association with the pen ID. When the user signs in, the color definition data associated with the user-defined data specified by the AccountId is specified. The display device 2 can display the handwritten data in which the color information of the user-defined data is emphasized in black and white, or the text data converted from the handwritten data.

[0281] S62: The handwriting input display control unit 23 creates selectable candidate display data as shown in FIG. 22 from these handwriting recognition character string candidates (in FIG. 22, "gi"), language character string candidates (although not shown in FIG. 22, for example, "yi"), conversion character string candidates (in FIG. 22, "minutes of meeting", "skill test"), prediction character string candidates (in FIG. 22, "settle the skill test", "destination of the minutes of meeting"), operation command candidates (in FIG. 22, "load the minutes of meeting template", "save to the minutes of meeting folder"), each selection probability, and the estimated writing direction. Also, the handwriting input display control unit 23 rotates the selectable candidate display data (operation guide 500) counterclockwise by the angle information acquired in step S60, and transmits the rotated selectable candidate display data (operation guide 500) to the display unit 22 for display.

[0282] S63: Further, the handwriting input display control unit 23 rotates the rectangular area display data (rectangular frame) of the handwriting object and the selection object (in FIG. 22, the handwriting object rectangular area display 503) counterclockwise by the angle information acquired in step S60, and transmits it to the display unit 22 for display.

[0283] S64: The handwriting input display control unit 23 transmits a command to start the selectable candidate display erasure timer to the candidate display timer control unit 24 in order to erase the selectable candidate display data after a certain time. The candidate display timer control unit 24 starts this timer.

[0284] Steps S65 to S70 are executed when, during the selection candidate elimination timer start, the user has erased the selectable candidate display shown on the display unit 22, or when a change has occurred in the handwritten object (i.e., when the stroke of the handwritten object has been added, deleted, moved, deformed, or divided), or when no candidate has been selected by the time of timeout.

[0285] Furthermore, steps S65 to S67 are executed when the candidate display has been erased or when a change has occurred in the handwritten object.

[0286] S65: The handwriting input unit 21 transmits to the handwriting input display control unit 23 the elimination of the selectable candidate display or the occurrence of a change in the handwritten object.

[0287] S66: The handwriting input display control unit 23 transmits a stop of the selectable candidate elimination timer. The candidate display timer control unit 24 stops the timer. This is because an operation has been performed on the handwritten object within a certain period of time, making the timer unnecessary.

[0288] S67: The handwriting input display control unit 23 stores the position information of the operation guide 500 in the handwriting recognition control unit 26 so that it can be used in the gesture determination of the gesture handwriting recognition in step S19. The position information is, for example, the coordinates of the upper left corner and the lower right corner or coordinates equivalent thereto. Thereby, the handwriting recognition control unit 26 can determine whether the straight line used for the input of the angle information is within the operation guide 500.

[0289] S69: The handwriting input display control unit 23 causes the display to be erased by transmitting the elimination of the selectable candidate display data to the display unit 22.

[0290] S70: The handwriting input display control unit 23 causes the display to be erased by transmitting the elimination of the rectangular area display data of the handwritten object and the selected object to the display unit 22. Therefore, when the display of the operation command candidates is erased under conditions other than when the operation command candidates are selected, the handwritten object remains displayed as it is.

[0291] S68: If, while the selectable candidate elimination timer is running, there is no occurrence of the elimination of the selectable candidate display or a change in the handwritten object (when the user does not perform a pen operation), the candidate display timer control unit 24 sends a timeout to the handwritten input display control unit 23.

[0292] Similarly, after the timeout of the selectable candidate display elimination timer, the handwritten input display control unit 23 executes steps S69 and S70. This is because it may be possible to eliminate the selectable candidate display data, the handwritten object, and the rectangular area display data of the selected object after a certain period of time has elapsed.

[0293] If the user selects a selectable candidate while the selectable candidate elimination timer is running, steps S71 to S103 are executed.

[0294] S71: If the user selects a selectable candidate while the selectable candidate elimination timer is running, the handwritten input unit 21 sends the selection of the character string candidate or the operation command candidate to the handwritten input display control unit 23.

[0295] S71-2: The handwritten input display control unit 23 sends a stop of the selectable candidate display elimination timer to the candidate display timer control unit 24. The candidate display timer control unit 24 stops this timer.

[0296] S72: Next, the handwritten input display control unit 23 sends a clear of the held data to the handwritten recognition control unit 26.

[0297] S73: The handwritten recognition control unit 26 sends a clear of the held data to the character string conversion control unit 28.

[0298] S74: The handwritten recognition control unit 26 sends a clear of the held data to the prediction conversion control unit 30.

[0299] S75: The handwritten recognition control unit 26 sends a hold data clear to the operation command recognition control unit 32. The handwritten recognition control unit 26, the character string conversion control unit 28, the prediction conversion control unit 30, and the operation command recognition control unit 32 clear the data related to the character string candidates and operation command candidates that they have held so far.

[0300] S76: Next, the handwritten input display control unit 23 sends a selectable candidate display data deletion to the display unit 22 to erase the display.

[0301] S77: The handwritten input display control unit 23 sends a rectangular area display data deletion of the handwritten object and the selection object to the display unit 22 to erase the display.

[0302] S78: The handwritten input display control unit 23 sends a handwritten object display data deletion and a pen coordinate completion display data deletion sent in step S6 to the display unit 22 to erase the display. This is because the handwritten object etc. becomes unnecessary since a character string candidate or an operation command candidate has been selected.

[0303] S79: The handwritten input display control unit 23 sends a handwritten object deletion to the handwritten input storage unit 25.

[0304] When a character string candidate is selected, steps S80 to S82 are executed.

[0305] S80: When a character string candidate is selected, the handwritten input display control unit 23 sends a character string object addition to the handwritten input storage unit 25.

[0306] S81: Further, the handwritten input display control unit 23 sends a character string object font acquisition to the handwritten input storage unit 25. The handwritten input storage unit 25 specifies the font associated with the PenId of the pen ID control data from the pen color selection button definition data, and also sends the estimated character size of the handwritten object to the handwritten input display control unit 23.

[0307] S82: Next, the handwritten input display control unit 23 causes the display unit 22 to display the string object display data to be displayed at the same position as the handwritten object by using the defined font received from the handwritten input storage unit 25. The handwritten input display control unit 23 identifies the line type and thickness from the color definition data based on the ColorId of the pen ID control data, and displays the text (string object) rotated by the angle information.

[0308] When a candidate for the operation command is selected, steps S83 to S101 are executed. If there is a selected object, steps S83 to S85 are executed.

[0309] S83: When a candidate for the operation command to the selected object is selected (when there is a selected object), the handwritten input display control unit 23 causes the display unit 22 to erase the display by transmitting the selected object display data erasure. This is to erase the original selected object once.

[0310] S84: Next, the handwritten input display control unit 23 transmits the execution of the operation command to the selected object to the handwritten input storage unit 25. The handwritten input storage unit 25 transmits the display data of the new selected object (the display data after editing or modification) to the handwritten input display control unit 23.

[0311] S85: Next, the handwritten input display control unit 23 transmits the selected object display data to the display unit 22 to redisplay the selected object after the execution of the operation command.

[0312] S86: When "register handwritten signature" in the operation command definition data 713 or "change settings" in the operation command definition data 716 is specified, the handwritten input display control unit 23 adds the handwritten signature registration form 561 or the user-defined data change form to the handwritten input storage unit 25.

[0313] S87: When a candidate for an operation command of "file saving" or "printing" is selected, the handwritten input display control unit 23 transmits a file transmission request to the file transmission / reception control unit 37.

[0314] S88: The file transmission / reception control unit 37 transmits a request for acquiring handwritten input storage data to be transmitted to the handwritten input storage unit 25.

[0315] S89: The handwritten input storage unit 25 determines whether the transmission destination is a color-compatible device using an MIB or the like. · In the case of a color-compatible device, the handwritten input storage unit 25 transmits the handwritten input storage data converted to color to the file transmission / reception control unit 37. That is, the handwritten input storage unit 25 determines whether it is handwritten input-derived data based on the ColorId of the handwritten input storage data, acquires color information associated with the ColorId from the color definition data, and converts the handwritten input-derived data into this color information. Since data that is not handwritten input-derived data already contains color information, the handwritten input storage unit 25 transmits the color information. It suffices for the handwritten input storage unit 25 to be able to determine whether there is handwritten input-derived data (handwritten data drawn on this display device 2) even if it is not the ColorId. Also, the information indicating whether it is handwritten input-derived data may be a value indicating a device indicating whether it is handwritten data drawn on this display device 2. · In the case where the transmission destination is a black-and-white compatible device, the handwritten input storage unit 25 may transmit merely a coordinate point sequence or text while ignoring the color information associated with the ColorId of the handwritten input storage data. Also, the handwritten input storage unit 25 may acquire the color information associated with the ColorId from the color definition data and convert the handwritten input-derived data into this color information. However, the handwritten input storage unit 25 may also acquire the color information associated with the ColorId from the color definition data, convert it to black-and-white highlighting (line type and thickness), and transmit it. Since data that is not handwritten input-derived data already contains color information, the handwritten input storage unit 25 transmits the color information. · When the destination is a black-and-white enhancement compatible device, the handwritten input storage unit 25 transmits only the handwritten input storage data, or both the handwritten input storage data and the color definition data to the file transmission / reception control unit 37. When the handwritten input storage unit 25 transmits only the handwritten input storage data, the line type and thickness are determined according to the color definition data held by the black-and-white enhancement compatible device at the destination. When the handwritten input storage unit 25 transmits both the handwritten input storage data and the color definition data, the line type and thickness are determined according to the color definition data held by the black-and-white enhancement compatible device at the source. Note that the handwritten input storage unit 25 may transmit after the black-and-white enhancement compatible device at the source converts to black-and-white enhancement display. · Also, when the file transmission / reception control unit 37 writes to a file, for example, when writing to a PDF file, etc., it acquires the color information associated with ColorId from the color definition data, and sets the handwritten input origin data to the PDF file according to the format of the PDF file. The same applies to the font, character size, etc. The file transmission / reception control unit 37 further stores all or part of the handwritten input storage data (for example, ColorId, FontNmae, etc.) in the metadata of the PDF file. Since the PDF file reader ignores the metadata and has no effect on the display, when the display device 2 reads this PDF file, the handwritten input storage data can be reproduced from the metadata. In the case of data other than the handwritten input origin data, the file transmission / reception control unit 37 sets the color information, font, character size, etc. in the handwritten input storage data to the PDF file according to the format of the PDF file.

[0316] S90: The file transmission / reception control unit 37 transmits the handwritten input data received from the handwritten input storage unit to the destination or writes it to a file.

[0317] S91: When a candidate for the operation command of "file reading" is selected, the handwritten input display control unit 23 transmits a file list information acquisition request to the file transmission / reception control unit 37.

[0318] S92: The file transmission / reception control unit 37 receives file list information from a storage medium such as a USB memory, a network storage, a web server, or an external device.

[0319] S93: The file transmission / reception control unit 37 transmits the file list information to the handwritten input display control unit 23.

[0320] S94: The handwritten input display control unit 23 transmits the file list display data to the display unit 22. As a result, the display unit 22 displays the file list on the display as shown in FIG. 12.

[0321] S95: When the user selects a file and the handwritten input unit 21 receives it, the handwritten input unit 21 transmits the file selection to the handwritten input display control unit 23.

[0322] S96: The handwritten input display control unit 23 transmits a file reception request for the selected file to the file transmission / reception control unit 37.

[0323] S97: The file transmission / reception control unit 37 acquires the file from the external device.

[0324] S98: The file transmission / reception control unit 37 saves the file in the handwritten input storage unit 25.

[0325] S99: The handwritten input storage unit 25 analyzes the file received from the file transmission / reception control unit 37 and converts the handwritten input-derived data into handwritten input storage data (black-and-white emphasized / color-convertible data). That is, the handwritten input storage unit 25 determines the presence or absence of metadata. If there is metadata, the handwritten input storage unit 25 determines whether conversion to handwritten input storage data is possible (whether there is a ColorId, etc.), and stores it as handwritten input storage data. Then, the handwritten input storage unit 25 reads the ColorId for the handwritten input-derived data, refers to the color definition data, and converts it to the black-and-white emphasized display associated with the ColorId. The handwritten input storage unit 25 transmits the display data of the handwritten object to the handwritten input display control unit 23. In the case of data that is not handwritten input-derived data, the handwritten input storage unit 25 reads color information, font, character size, etc. according to the file format and stores it as handwritten input storage data.

[0326] S100: The handwritten input display control unit 23 displays the display data of the handwritten input data on the display unit 22. As a result, the display unit 22 displays the handwritten input-derived data as handwritten data with black-and-white emphasis, and displays other data in black-and-white using conventional luminance conversion.

[0327] In addition, when the operation command 512 for signing in is executed, the handwritten input display control unit 23 acquires the pen ID received by the display device 2 when the operation command 512 is executed. The handwritten input display control unit 23 specifies the user-defined data having the SignatureId acquired in step S61 and acquires the AccountId from the user-defined data. Then, the handwritten input display control unit 23 registers the AccountId in the pen ID control data in association with the pen ID. As a result, the pen 2500 is associated with the user, and the display device 2 can perform processing using the user-defined data.

[0328] When the user writes by hand or reads a file after signing in, the handwritten input display control unit 23 obtains the AccountId associated with the pen ID received by the display device 2 at the time of executing the operation command from the pen ID control data. The handwritten input display control unit 23 specifies the user-defined data with this AccountId, and sets color definition data and the like in the %~% of the operation command and executes it.

[0329] S101: When an operation command of the input / output system is selected, the handwritten input display control unit 23 executes the operation command character string (Command) of the operation command definition data corresponding to the operation command selected by the user.

[0330] S102: When the user manually inputs angle information, the handwritten input display control unit 23 associates the received angle information with the pen ID received from the pen 2500 when the rotation operation button 511 is pressed and stores it in the pen ID control data storage unit 36.

[0331] S103: For the next handwritten object, the handwritten input display control unit 23 sends a handwritten object start to the handwritten input storage unit 25. The handwritten input storage unit 25 secures a handwritten object area. Thereafter, the processes of steps S2 to S103 are repeated.

[0332] <Summary> As described above, the display device 2 of the present embodiment associates the data caused by the handwritten input with that fact so that it can detect that it is the data caused by the handwritten input. Thereby, even when the display device 2 acquires a file from the outside, when displaying the data caused by the handwritten input, black-and-white highlighting can be performed based on the color information associated with the handwritten data. Further, when the display device 2 transmits the data caused by the handwritten input to an external device, it can be displayed without determining whether it is the data caused by the handwritten input.

Example

[0333] In this embodiment, a display device 2 for converting handwritten data into English will be described. Note that the configuration of the display device 2 is the same as that of the first embodiment except that the dictionary for conversion and the operation command definition data support English. Therefore, the differences from the first embodiment when the display device 2 converts handwritten data into English (hereinafter referred to as English conversion) will be described.

[0334] FIG. 38 is a diagram for explaining an outline of a method for displaying data caused by handwritten input by the display device 2 of this embodiment in the case of English conversion. Note that in the description of FIG. 38, the differences from FIG. 2 will mainly be described.

[0335] (1) First, the user can select in what color to input handwritten data from the buttons. In FIG. 2, the user can select each color of Bk, R, B, G, M, and C.

[0336] (2) When the user writes, for example, "hand-written", the display device 2 displays the handwritten data in a manner corresponding to the color. The manner corresponding to the color means displaying with different line types depending on the color. By displaying with line types corresponding to the color in this way, black-and-white highlighting can be enabled. The display device 2 can convert the handwritten data into text (character code). The handwritten data 4 and 5 in FIG. 2 have been converted into text, and the handwritten data 6 is handwritten data that has not been converted.

[0337] (3) The display device 2 of this embodiment has information on whether the data is caused by handwritten input and color information for each handwritten data. The handwritten data 4 is data caused by handwritten input and the color information is Bk, the handwritten data 5 is data caused by handwritten input and the color information is R, and the handwritten data 6 is data caused by handwritten input and the color information is B. Since the text (character code) converted from the handwritten data like the handwritten data 4 and 5 also holds information on whether it is data caused by handwritten input, the display device 2 can display the data caused by handwritten input with a line type corresponding to the specified color.

[0338] (4) When the display device 2 converts handwritten data into a PDF file or the like, the display device 2 sets color information of the handwritten data or text data according to the PDF format, and also attaches a message indicating that the data is handwritten input-derived data and the color information as metadata. When the display device 2 reads this PDF file from a storage medium (USB memory, etc.), it can display the handwritten input-derived data with a line type corresponding to the color information based on the metadata.

[0339] In this way, even if the user handwrites English, when displaying handwritten input derived data, the display device 2 of this embodiment can display the data in black and white highlighting based on the color information associated with the handwritten data.

[0340] FIG. 39 is a diagram for explaining an outline of a method for displaying handwriting input-induced data by the display device 2 of this embodiment in the case of Chinese conversion. "手掛け的" means "handwritten". In this way, Chinese can be converted in the same way. Moreover, the display device 2 of this embodiment is language independent and can handle other languages ​​in the same way.

[0341] <An example of operation command definition data> Figure 40 is an example of operation command definition data when there is no selected object. The explanation of Figure 40 will mainly focus on the differences from Figure 11. The content of each operation command is the same as that of Figure 11, but the name and string are associated with English expressions. Therefore, the user can handwrite operation commands in English and select English operation commands.

[0342] Figure 41 is an example of operation command definition data when there is a selected object. The explanation of Figure 41 will mainly focus on the differences from Figure 13. The content of each operation command is the same as in Figure 13, but the Name is associated with an English expression. Therefore, the user can select an English operation command.

[0343] Figure 42 is an example of user-defined data. In the description of Figure 42, the differences from Figure 14 will be mainly explained. The content of the user-defined data is the same as that of Figure 14, but AccountUsername is set to "Bob". Also, English expressions are associated with Name. Therefore, the user can select user-defined data in English.

[0344] Note that the handwritten signature data held by the handwritten signature data storage unit 39 may be the same as that in Figure 15, and the handwritten input storage data may be the same as that in Figure 16. The pen ID control data may be the same as that in Figure 17, and the color definition data may be the same as that in Figure 18.

[0345] <An example of dictionary data> The dictionary data for English conversion will be described with reference to Figures 43 to 45. In the description of Figures 43 to 45, the differences from Figures 8 to 10 will be mainly explained. Figure 43 is an example of the dictionary data of the handwritten recognition dictionary unit 27. The dictionary data of the handwritten recognition dictionary unit 27 in Figure 43 indicates that the handwritten "a (in the state of stroke data)" is converted to "a" with a probability of 0.90 and to "o" with a probability of 0.10.

[0346] Figure 44 is an example of the dictionary data of the character string conversion dictionary unit 29. The dictionary data of the character string conversion dictionary unit 29 in Figure 44 indicates that the character string "a" is converted to "ab" with a probability of 0.55 and to "AI" with a probability of 0.45. The same applies to other "before conversion" character strings.

[0347] Figure 45 is an example of the dictionary data of the prediction conversion dictionary unit 31. The dictionary data of the prediction conversion dictionary unit 31 in Figure 45 indicates that the character string "agenda" is converted to "agenda list" with a probability of 0.55 and to "agenda template" with a probability of 0.30.

[0348] Note that the dictionary data is not language-dependent, and any character strings may be registered before and after conversion.

[0349] <Data highlighted in black and white and data displayed in color display method> Figure 46 shows the data highlighted in black and white and the data displayed in a color display method using the color definition data of Figure 18. In the description of Figure 46, the differences from Figure 19 will mainly be explained.

[0350] Reference numeral C21 is the text of "Bk", "R", "B", "G", "magenta", "cyan", which is data caused by handwritten input. Reference numeral C22 is the stroke (handwriting) of "Bk", "R", "B", "G", "magenta", "cyan". Reference numeral C23 is a spring-shaped single-stroke (handwriting). Reference numeral C24 is a donut-shaped color chart (image). Reference numeral C25 is RGBCMY (text not caused by handwritten input) inside the color chart. Below reference numeral C23 is the text "Transparent" in black, which is data caused by handwritten input.

[0351] Therefore, black, red, blue, green, magenta, and cyan in Japanese are the same as in Figure 19, except that "Bk", "R", "B", "G", "magenta", "cyan" are changed. The same applies to Figure 46(b).

[0352] As can be seen by comparing Figure 46(a) and (b), the red of the text "R" inside the color chart of reference numeral C30 in Figure 46(b) is not data caused by handwritten input. For this reason, the text "R" is displayed in grayscale in Figure 46(a). On the other hand, the text "R" of reference numeral C26 is data caused by handwritten input, so it is highlighted in black and white in Figure 46(a).

[0353] Also, since magenta and cyan are transparent colors, "Transparent" appears to be transparent in Figure 46(b). In Figure 46(a) as well, since white is used as a transparent pattern for magenta and cyan, it is displayed transparently. In this way, the black-and-white highlighting device can highlight the colors for the purpose of making them stand out, and the color device can convert them to colors for display.

[0354] <Example of Display of Selectable Candidates> FIG. 47 is an example of an operation guide 500 and selectable candidates 530 displayed by the operation guide 500 in the case of English conversion. In the description of FIG. 47, the differences from FIG. 22 will mainly be described. In FIG. 47, the user handwrote "a" as a handwritten object 504. Based on "a", candidates for operation commands 510, candidates for handwritten recognition character strings 506, candidates for conversion character strings 507, and candidates for string / predicted conversion 508 are displayed. Therefore, it may be the same as FIG. 22 except that Japanese in FIG. 22 is changed to English.

[0355] The candidate for the operation command 510 is, for example, a candidate for the operation command having "agenda" as a string in the operation command definition data of FIG. 40(a) (operation command definition data 701, 702).

[0356] In this way, the user can also display the operation guide 500 in the case of English conversion.

[0357] <Example of Designating a Selection Object> FIG. 48 is an example of a diagram for explaining an example of designating a selection object in the case of English conversion. Note that in the description of FIG. 48, the differences from FIG. 23 will mainly be described.

[0358] FIG. 48(a) is an example in which the user designated two horizontal fixed objects 13a2 and 13b2 with a straddle line (handwritten object 11a2). In this example, the length H1 of the short side and the length W1 of the long side of the handwritten object rectangular area 12a2 satisfy the conditions of the straddle line determination condition 406, and the overlapping rate with the fixed objects 13a2 and 13b2 satisfies the conditions of the straddle line determination condition 406. For this reason, both fixed objects 13a2 and 13b2 of "agenda" and "ag" are designated as selection objects.

[0359] Figure 48(b) shows an example where the user designates a horizontally-written finalized object 13c2 with an enclosing line (handwritten object 11b2). In this example, only the finalized object 13c2 with the word "agenda", whose overlapping rate with the rectangular area of the handwritten object 12c2 satisfies the condition of the enclosing line determination condition 407, is designated as the selected object.

[0360] Even in the case of English conversion like this, the user can similarly select a selected object.

[0361] <Example of Display of Candidate Operation Commands> Figure 49 shows an example of the display of candidate operation commands based on the operation command definition data when there is the handwritten object shown in Figure 48. In the description of Figure 49, the differences from Figure 24 will mainly be explained.

[0362] Figure 49(a) shows candidates for editing system operation commands, and Figure 49(b) shows candidates for formatting system operation commands. Also, Figure 49(a) shows an example where a selected object is designated with the handwritten object 11a2 in Figure 48(a). As shown in Figures 49(a) and (b), the candidates for operation commands displayed following the leading character "》" 511 are the main menu 550.

[0363] In Figure 49(a), when the "〉" 512a in the first line is pressed, the sub-menu 560 in Figure 49(a) is displayed. When the user presses any operation command name with the pen, the Command of the operation command definition data associated with the operation command name is executed by the handwritten input display control unit 23 for the selected object. That is, when "Delete" 521b is selected, the handwritten input display control unit 23 executes "Delete". When "Move" 522b is selected, the handwritten input display control unit 23 executes "Move". When "Rotate" 523b is selected, the handwritten input display control unit 23 executes "Rotate". When "Select" 524b is selected, the handwritten input display control unit 23 executes "Select".

[0364] When the user presses "Delete" 511b with a pen, the handwriting input display control unit 23 deletes the confirmed objects 13a2 and 13b2 of "agenda" and "ag". When the user presses "Move" 522b with a pen, the handwriting input display control unit 23 accepts the movement of the confirmed objects 13a2 and 13b2 of "agenda" and "ag". When the user presses "Rotate" 523b with a pen, the handwriting input display control unit 23 rotates the confirmed objects 13a2 and 13b2 of "agenda" and "ag" by a certain angle. When the user presses "Select" 524b with a pen, the handwriting input display control unit 23 accepts the selection of the confirmed objects 13a2 and 13b2 of "agenda" and "ag".

[0365] The strings "―" 541b, "-" 542b, "~" 543b, "→" 544b, and "⇒" 545b, which are string candidates other than operation command candidates, are recognition results of strikethroughs (handwritten object 11a2). When the user intends to input a string rather than an operation command, the string candidate can be selected.

[0366] In Fig. 49(b), when the "〉" 512b in the second row is pressed, the sub-menu in Fig. 49(b) is displayed. Similar to Fig. 49(a), the main menu 550 and the sub-menu 560 are also displayed in the display example of Fig. 49(b). Based on the operation command definition data in Fig. 41, when "Thick" 531b is selected, the handwriting input display control unit 23 executes "Thick" on the selected object. When "Thin" 532b is selected, the handwriting input display control unit 23 executes "Thin" on the selected object. When "Large" 533b is selected, the handwriting input display control unit 23 executes "Large" on the selected object. When "Small" 534b is selected, the handwriting input display control unit 23 executes "Small" on the selected object. When "Underline" 535b is selected, the handwriting input display control unit 23 executes "Underline" on the selected object.

[0367] When the user presses "Thick" 531b with the pen, the handwriting input display control unit 23 thickens the lines forming the determined objects 13a2 and 13b2 of "agenda" and "ag". When the user presses "Thin" 532b with the pen, the handwriting input display control unit 23 thins the lines forming "agenda" and "ag". When the user presses "Large" 533b with the pen, the handwriting input display control unit 23 can be enlarged. When the user presses "Small" 534b with the pen, the handwriting input display control unit 23 can be reduced. When the user presses "Underline" 535b with the pen, the handwriting input display control unit 23 can add an underline.

[0368] In this way, even in the case of English conversion, the user can display the operation commands when there are handwritten objects.

[0369] <Example of input of angle information> Next, FIG. 50 is an example of a diagram for explaining a method of inputting angle information of 90 degrees. The method of inputting angle information is the same as that for Japanese conversion even in the case of English conversion. In the description of FIG. 50, the differences from FIG. 26 will mainly be explained.

[0370] FIG. 50(a) shows a state in which, since the user existing in the 3 o'clock direction of the display device 2 has handwritten "a" with the angle information of the pen ID control data being 0 degrees (initial value), the operation guide 500 is displayed. Since the display device 2 recognizes the handwritten "a" from the 3 o'clock direction with the angle information remaining 0 degrees, selectable candidates 530 different from the expectation are displayed.

[0371] When inputting angle information, the user handwrites a straight line in the operation guide 500 from top to bottom as viewed by the user. FIG. 50(b) shows an example of this straight line 521b. The counterclockwise angle α formed by the 6 o'clock direction where the angle information is 0 degrees and this straight line 521b is the angle information.

[0372] Figure 50(c) shows the operation guide 500 immediately after the operation in Figure 50(b). Since 90 degrees is set as the angle information (Angle) in the pen ID control data, the handwritten object (stroke data) is rotated 90 degrees clockwise internally for handwriting recognition, and the operation guide 500 is rotated 90 degrees counterclockwise and displayed. Note that the angle information may be manually input by the user from the menu.

[0373] In this way, the user can input the angle information even in the case of English conversion.

[0374] <Example of registration of handwritten signature data> Next, Figure 51 is a diagram for explaining the method of registering handwritten signature data. In the explanation of Figure 51, the differences from Figure 27 will mainly be explained. First, Figure 51(a) is an example of selectable candidates 530 displayed when the user handwrites "Sign". Two operation commands 513b and 514b, "Register handwritten signature" and "Hand sign out", based on the operation command definition data 713 and 715 that partially match the string "Sign", and the string candidates "Sign", "Signature session", and "Signed" are displayed.

[0375] When the user presses "Register handwritten signature" with the pen 2500, the handwritten signature registration form 561 in Figure 51(b) is added to the handwritten input storage unit 25 and displayed on the operation screen 101. For example, the operation guide 500 in Figure 51(a) is erased, and the handwritten signature registration form 561 is displayed at the same location as the operation guide 500.

[0376] The user inputs the text of the name in the name input field 561a, the first handwritten signature, the second handwritten signature, and the third handwritten signature in the signature input fields 561b to 561d, and a check mark or a cancellation mark in the registration confirmation field 561e.

[0377] When the user writes by hand as instructed on the handwritten signature registration form 561, the handwritten signature registration form 561 shown in Fig. 51(c) is obtained.

[0378] In this way, the handwritten input display control unit 23 can accept handwritten input without distinguishing between handwritten input to the form and handwritten input outside the form. The user can register handwritten signature data in English.

[0379] <An example of signing in by hand> Fig. 52 shows an example of the operation guide 500 displayed when the user writes "Bob", which is the handwritten signature data registered in the English conversion. In the description of Fig. 52, the differences from Fig. 28 will be mainly described.

[0380] Since "Bob" is registered in the handwritten signature data storage unit 39 as handwritten signature data, it matches the handwritten signature data. Therefore, the operation command 512 "Hand Sign in" is displayed.

[0381] Also, since the handwritten signature data matches, the SignatureId representing "Bob" is specified, and the user-defined data having the AccountId associated with this SignatureId is specified.

[0382] When the user selects the operation command 512 "Hand Sign in", the AccountId of "Bob" is associated with the pen ID of the pen 2500 being used and added to the pen ID control data, and the user-defined data of "Bob" is used when the operation command is used.

[0383] In this way, the user can sign in in English.

[0384] <An example of an operation for changing user-defined data> FIG. 53 is a diagram for explaining a method of changing user-defined data in English conversion. In the explanation of FIG. 53, the differences from FIG. 29 will mainly be explained. FIG. 53(a) is an example of the operation guide 500 displayed when the user handwrites "set". In the operation command definition data 716, "set" is defined for String, and the operation command "Change settings" is displayed.

[0385] When the user selects "Change settings" with the pen 2500 that has signed in by handwriting, the AccountId of the user associated with the pen ID of this pen 2500 in the pen ID control data is identified. Thereby, the user-defined data of the signed-in user is identified, and the user-defined data change form 562 in FIG. 53(b) is added to the handwriting input storage unit 25 and displayed on the operation screen 101. Each item of the user-defined data change form 562 is the same as that in FIG. 29(b).

[0386] In this way, the user can change the user definition in the case of English conversion in the same manner as in Japanese conversion.

[0387] Note that the operation procedure may be the same as that in FIGS. 30 to 37 of the first embodiment.

Example

[0388] In the first embodiment, the display device 2 is described as having a large touch panel, but the display device is not limited to having a touch panel. In this embodiment, a projector-type display device will be described.

[0389] <<Another Configuration Example 1 of the Display Device>> FIG. 54 is a diagram showing another configuration example of the display device. In FIG. 54, a projector 411 is installed on the upper side of a normal whiteboard 413. This projector 411 corresponds to the display device. The normal whiteboard 413 is not a flat panel display integrated with a touch panel, but a whiteboard on which a user can directly write by hand with a marker. Note that the whiteboard may be a blackboard, as long as it is a flat surface large enough to project an image.

[0390] The projector 411 has an ultra-short focus optical system and can project an image with little distortion onto the whiteboard 413 from about 10 cm away. This image may be transmitted from a PC 400-1 connected wirelessly or by wire, or may be stored in the projector 411.

[0391] The user writes on the whiteboard 413 using a dedicated electronic pen 2501. The electronic pen 2501 has, for example, a light emitting part at its tip that emits light when the user presses it against the whiteboard 413 for handwriting, and the switch turns on. Since the wavelength of the light is near infrared or infrared, it is not visible to the user's eyes. The projector 411 has a camera, captures the light emitting part, analyzes the image, and identifies the direction of the electronic pen 2501. In addition, the electronic pen 2501 emits sound waves along with the light emission, and the projector 411 calculates the distance based on the arrival time of the sound waves. The projector 411 can identify the position of the electronic pen 2501 based on the direction and the distance. A stroke is drawn (projected) at the position of the electronic pen 2501.

[0392] Since the projector 411 projects the menu 430, when the user presses a button with the electronic pen 2501, the projector 411 identifies the position of the electronic pen 2501 and the button pressed by the ON signal of the switch. For example, when the save button 431 is pressed, the strokes (set of coordinates) handwritten by the user are saved by the projector 411. The projector 411 saves the handwritten information to a predetermined server 412 or USB memory 2600, etc. The handwritten information is saved page by page. Since it is saved as coordinates instead of image data, the user can re-edit it. However, in this embodiment, since the operation command can be called by handwriting, the menu 430 may not be displayed.

[0393] <<Another Configuration Example 2 of the Display Device>> FIG. 55 is a diagram showing another configuration example of the display device 2. In the example of FIG. 55, the display device 2 includes a terminal device 600, an image projection device 700A, and a pen motion detection device 810.

[0394] The terminal device 600 is connected to the image projection device 700A and the pen motion detection device 810 by wire. The image projection device 700A projects the image data input by the terminal device 600 onto the screen 800.

[0395] The pen motion detection device 810 communicates with the electronic pen 820 and detects the motion of the electronic pen 820 in the vicinity of the screen 800. Specifically, the electronic pen 820 detects the coordinate information indicating the point indicated by the electronic pen 820 on the screen 800 and transmits it to the terminal device 600.

[0396] Based on the coordinate information received from the pen motion detection device 810, the terminal device 600 generates the image data of the stroke image input by the electronic pen 820. The terminal device 600 causes the image projection device 700A to draw the stroke image on the screen 800.

[0397] Further, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by synthesizing the background image projected by the image projection device 700A and the stroke image input by the electronic pen 820.

[0398] <<Another Configuration Example 3 of the Display Device>> FIG. 56 is a diagram showing a configuration example of the display device. In the example of FIG. 56, the display device 2 includes a terminal device 600, a display 800A, and a pen motion detection device 810.

[0399] The pen motion detection device 810 is disposed near the display 800A. The pen motion detection device 810 detects coordinate information indicating the point indicated by the electronic pen 820A on the display 800A and transmits it to the terminal device 600. In the example of FIG. 56, the electronic pen 820A may be charged by the terminal device 600 via a USB connector.

[0400] Based on the coordinate information received from the pen motion detection device 810, the terminal device 600 generates image data of the stroke image input by the electronic pen 820A. The terminal device 600 causes the display 800A to display the stroke image.

[0401] <<Another Configuration Example 4 of the Display Device>> FIG. 57 is a diagram showing a configuration example of the display device. In the example of FIG. 57, the display device 2 includes a terminal device 600 and an image projection device 700A.

[0402] The terminal device 600 performs wireless communication (such as Bluetooth (registered trademark)) with the electronic pen 820B and receives coordinate information of the point indicated by the electronic pen 820B on the screen 800. Then, based on the received coordinate information, the terminal device 600 generates image data of the stroke image input by the electronic pen 820B. The terminal device 600 causes the image projection device 700A to project the stroke image.

[0403] Further, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by synthesizing a background image projected by the image projection device 700A and a stroke image input by the electronic pen 820.

[0404] As described above, each of the above-described embodiments can be applied in various system configurations.

[0405] <Other application examples> As described above, the best mode for carrying out the present invention has been described using examples. However, the present invention is not limited to such examples, and various modifications and substitutions can be made without departing from the gist of the present invention.

[0406] For example, the display method of the present embodiment can be suitably applied to any information processing device having a touch panel. Also, a device having the same function as the display device is also referred to as an electronic blackboard, an electronic whiteboard, an electronic information board, an interactive board, or the like. Examples of the information processing device equipped with a touch panel include, for example, an output device such as a PJ (Projector), a digital signage, a HUD (Head Up Display) device, an industrial machine, an imaging device, a sound collecting device, a medical device, a network home appliance, a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game machine, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, or a desktop PC.

[0407] Also, in the present embodiment, a part of the processing performed by the display device 2 may be performed by the server. For example, the display device transmits stroke information to the server, and acquires and displays information to be displayed on the operation guide 500 from the server.

[0408] In addition, in this embodiment, the display device 2 detects the pen coordinates by detecting the coordinates of the pen tip with a touch panel, but the coordinates of the pen tip may also be detected by ultrasonic waves. The pen emits ultrasonic waves while emitting light, and the display device 2 calculates the distance based on the arrival time of the ultrasonic waves. The display device 2 can specify the position of the pen based on the direction and the distance. The display device 2 draws (projects) the trajectory of the pen as a stroke.

[0409] In addition, in this embodiment, when there is a selection object, candidates for editing and modification operation commands are displayed, and when there is no selection object, candidates for input / output operation commands are displayed. However, the display device 2 may simultaneously display candidates for editing and modification operation commands and candidates for input / output operation commands.

[0410] Also, the display device 2 may not have the user's handwritten signature data. It may be held on the cloud or an information processing device within the company.

[0411] In addition, the configuration example in FIG. 6 and the like are divided according to the main functions in order to facilitate the understanding of the processing by the display device 2. The present invention is not limited by the way of dividing the processing units and the names. The processing of the display device 2 can be further divided into more processing units according to the processing content. Also, it can be divided so that one processing unit includes more processing.

[0412] In addition, each function of the embodiment described above can be realized by one or a plurality of processing circuits. Here, the "processing circuit" in this specification includes a processor programmed to execute each function by software like a processor implemented by an electronic circuit, an ASIC (Application Specific Integrated Circuit) designed to execute each function described above, a DSP (digital signal processor), an FPGA (field programmable gate array), and devices such as conventional circuit modules.

Description of Reference Numerals

[0413] 2 represents a device

Prior Art Documents

Patent Documents

[0414]

Non-Patent Document 1

Claims

1. A display device having a display unit for displaying handwritten data on the display unit, a display control unit that, in a manner corresponding to color information indicating a color selected from a plurality of colors, and in a different manner for each of the plurality of colors, displays the handwritten data in black and white on the display unit, and that displays on the display unit a button capable of accepting a selection of one color from the plurality of colors; a handwriting input unit that accepts a handwriting input and converts the accepted handwriting input into the handwritten data, wherein the display control unit displays the handwritten data in black and white on the display unit in a manner corresponding to the color information of the color selected by the button, the display device further comprising a storage unit that stores the color information of the color selected by the button in association with the handwritten data; a transmission unit that transmits, based on a user request, the stored handwritten data and data based on the color information to an external device; and characterized by having the above.

2. The display device according to claim 1, wherein the transmission unit transmits, based on a user request, a PDF file written out from the stored handwritten data and having the color information stored in association with the handwritten data set therein, to the external device.

3. The display device according to claim 1, further comprising a handwriting recognition control unit that performs handwriting recognition on the handwritten data to convert it into text data, and displays, in a black and white line type pre-associated with the color information stored in association with the handwritten data, the handwritten data that is the stroke data corresponding to the handwriting input handwritten on the display device, or the text data obtained by converting the stroke data by the handwriting recognition control unit by handwriting recognition. the stroke data corresponding to the handwriting input handwritten on the display device, or the text data obtained by converting the stroke data by the handwriting recognition control unit by handwriting recognition, which is the handwritten data, and characterized by displaying it in a black and white line type pre-associated with the color information stored in association with the handwritten data.

4. The display unit is a touch panel, The handwriting recognition control unit performs handwriting recognition on the handwriting data corresponding to the handwriting input written based on the position of the input means in contact with the touch panel and converts it into text data. The handwriting input unit receives a color selection by pressing a button capable of receiving a selection of one color from the plurality of colors. When displaying the text data converted by the handwriting recognition control unit, the display device according to claim 3 is characterized in that the text data is displayed in a black-and-white line type pre-associated with the color information of the color received by the handwriting input unit.

5. The handwriting input unit receives a font specification. When displaying the text data converted by the handwriting recognition control unit, the display device according to claim 4 is characterized in that the text data is displayed in a black-and-white line type pre-associated with the font received by the handwriting input unit and the color information.

6. Holds definition data in which color information is associated with the user's identification information. When the user signs in, the display device according to claim 4 or 5 is characterized in that the handwriting data or the text data converted from the handwriting data is displayed in a black-and-white line type pre-associated with the color information registered in the definition data.

7. When saving the handwriting data corresponding to the handwriting input written on the touch panel to a file. The display device according to any one of claims 4 to 6 is characterized in that the color information received by the handwriting input unit is converted into color information according to the file format and saved.

8. When saving the handwriting data corresponding to the handwriting input written on the touch panel to a file. The display device according to claim 7, characterized in that the pornographic information received by the handwriting input unit is converted into pornographic information in accordance with a file format and saved, and the pornographic information received by the handwriting input unit is attached to the metadata of the file.

9. The display device according to any one of claims 4 to 8, characterized in that the handwriting data is transmitted to a device that outputs the handwriting data in a black and white line type that is pre-associated with the pornographic information.

10. The display device according to any one of claims 4 to 9, characterized in that when the handwriting data is transmitted to a color correspondence device, the color of the handwriting data is converted into the pornographic information received by the handwriting input unit and then transmitted.

11. The display device according to any one of claims 4 to 9, characterized in that when the handwriting data is transmitted to a black and white correspondence device, the color of the handwriting data is converted into the pornographic information received by the handwriting input unit, or the black and white handwriting data is transmitted.

12. The display device according to claim 7 or 8, characterized in that when the handwriting data is included in the read file, the handwriting data is displayed in a black and white line type that is pre-associated with the pornographic information included in the metadata attached to the file.

13. The display device according to claim 12, characterized in that the handwriting data is text data.

14. The display device according to claim 12, characterized in that when the data included in the read file is not the handwriting data, the pornographic information of the data is converted into grayscale and the handwriting data is displayed.

15. The display device according to any one of claims 1 to 14, characterized in that the display device is an electronic paper that displays an image in white or black.

16. A color correspondence device for displaying a file saved by the display device according to any one of claims 1 to 15, or a file transmitted from the display device, stroke data corresponding to the handwriting input handwritten into the display device, or text data obtained by converting the stroke data by handwriting recognition by the display device, characterized in that the color correspondence device displays the text data in color. **Claim 17**: A display method performed by a display device having a display unit for displaying handwriting data on the display unit, wherein a display control unit, in a manner corresponding to color information indicating a color selected from a plurality of colors, and in a different manner for each of the plurality of colors, displays the handwriting data in black and white, and to this end, displays on the display unit a button capable of receiving a selection of one color from the plurality of colors, a handwriting input unit receives a handwriting input and converts the received handwriting input into the handwriting data, the display control unit displays the handwriting data in black and white on the display unit in a manner corresponding to the color information of the color selected by the button, a storage unit further stores the color information of the color selected by the button in association with the handwriting data, based on a user's request, a transmission unit transmits the stored handwriting data and data based on the color information to an external device. A display method characterized by the above. **Claim 18**: A display device having a display unit for displaying handwriting data on the display unit, a display control unit that, in a manner corresponding to color information indicating a color selected from a plurality of colors, and in a different manner for each of the plurality of colors, displays the handwriting data in black and white, and to this end, displays on the display unit a button capable of receiving a selection of one color from the plurality of colors, and a handwriting input unit that receives a handwriting input and converts the received handwriting input into the handwriting data, and functions as such, the display control unit The handwritten data is displayed in black and white on the display unit in a manner corresponding to the color information of the color selected by the button. Further, a storage unit that stores the color information of the color selected by the button in association with the handwritten data. A transmission unit that transmits the stored handwritten data and the data based on the color information to an external device based on a user's request. A program characterized by functioning as such.

Images