Information processing device, information processing method, and program

Abstract

To provide a display device that can perform data processing depending on a display position of an operation menu.SOLUTION: In a display device 2 that displays an operation menu which is related to data processing performed for handwriting data, the operation menu is characterized to have information associated with the data processing depending on a display position of the operation menu.SELECTED DRAWING: Figure 30

Description

[Technical field]

[0001] The present invention relates to a display device, a display method, and a program. [Background technology]

[0002] 2. Description of the Related Art Display devices that display data handwritten on a touch panel with a pen or a finger are known. Display devices equipped with a relatively large touch panel are placed in places such as conference rooms and used by multiple users as electronic whiteboards.

[0003] The display device may display a menu for the user to select the pen color, etc. (For example, see Patent Document 1.) Patent Document 1 discloses a handwriting input device that displays a menu for color setting, transparency setting, thickness setting, line type setting, stamp setting, and operation setting by pressing a pen button. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional display devices have a problem in that the information related to data processing is the same even if the display position of the operation menu is different. For example, when users write by hand on a display device, each user operates the operation menu near him / her. However, regardless of the display position of the operation menu operated by the user, the data processing performed by the display device on the handwritten data is the same.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide a display device capable of performing data processing according to the display position of an operation menu. [Means for solving the problem]

[0006] In view of the above problem, the present invention is a display device that displays an operation menu associated with data processing performed on handwritten data, wherein the operation menu has information regarding the data processing corresponding to the display position of the operation menu. Effect of the Invention

[0007] It is possible to provide a display device capable of performing data processing according to the display position of an operation menu. [Brief description of the drawings]

[0008] [Figure 1] FIG. 4 is a diagram showing an example of an operation menu displayed on the display device. [Diagram 2] 1 is an example of a display device placed flat. [Diagram 3] FIG. 2 is a diagram showing an example of a perspective view of a pen. [Figure 4] FIG. 1 is a diagram showing an example of an overall configuration diagram of a display device. [Diagram 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device. [Figure 6] FIG. 2 is a diagram illustrating functions of a display device and a pen. [Figure 7] FIG. 11 is a diagram illustrating an example of pre-defined control data. [Figure 8] 5 is a diagram showing an example of dictionary data in a handwriting recognition dictionary section; FIG. [Figure 9] FIG. 13 is a diagram illustrating an example of dictionary data of a character string conversion dictionary unit. [Figure 10] FIG. 4 is a diagram showing an example of dictionary data in a predictive conversion dictionary section; [Figure 11] 4A and 4B are diagrams illustrating an example of operation command definition data and system definition data held in an operation command definition unit. [Figure 12] 13 is a diagram showing an example of user-defined data held in an operation command definition unit; FIG. [Figure 13] 5 is a diagram showing an example of handwritten signature data stored in a handwritten signature data storage unit; FIG. [Figure 14] 11 is a diagram showing an example of handwritten input saved data saved in a handwritten input saving unit; FIG. [Figure 15] 11 is a diagram illustrating pen ID control data stored in a pen ID control data storage unit. FIG. [Figure 16] FIG. 11 is a diagram illustrating an example of color definition data. [Figure 17] FIG. 11 is a diagram illustrating an example of pen color selection button definition data. [Figure 18] FIG. 11 is a diagram showing an example of a pen color selection button displayed on a display. [Figure 19] 11A and 11B are diagrams illustrating an example of a manipulation guide and selectable candidates displayed by the manipulation guide. [Figure 20] 13A and 13B are diagrams illustrating an example of display of operation command candidates based on operation command definition data when handwritten data is present; [Figure 21] FIG. 13 is a diagram for explaining a method for inputting angle information of 90 degrees. [Figure 22] FIG. 13 is a diagram showing an example of an operation guide that is displayed when a user handwrites "SUZUKI" which is registered handwritten signature data. [Figure 23A] FIG. 2 is an example of a top view of a display device laid flat. [Figure 23B] FIG. 2 is a top view of the display device laid flat when the display device converts handwritten data into English. [Figure 24] FIG. 4 illustrates an example of a pen operation panel. [Diagram 25] FIG. 13 illustrates an example of a page operation panel. [Figure 26] FIG. 13 is a diagram illustrating an example of a page navigation operation panel. [Figure 27] FIG. 11 is a diagram showing an example of operation button definition data of the operation menu operated by the user on the lower side (part 1). [Figure 28] FIG. 2 is a diagram showing an example of operation button definition data of the operation menu operated by the user on the lower side (part 2). [Figure 29] FIG. 11 is a diagram showing an example of operation button definition data of the operation menu operated by the user on the upper side (part 1). [Diagram 30] FIG. 2 is a diagram showing an example of operation button definition data of the operation menu operated by the user on the upper side (part 2). [Diagram 31] FIG. 11 is a diagram showing an example of page navigation data. [Diagram 32] FIG. 11 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 1). [Diagram 33] FIG. 2 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 2). [Diagram 34] FIG. 11 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 3). [Diagram 35] FIG. 4 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 4). [Diagram 36] FIG. 5 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 5). [Figure 37] FIG. 6 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 6). [Figure 38] FIG. 7 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 7). [Figure 39] FIG. 8 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 8). [Diagram 40] 11 is a flowchart illustrating an example of a procedure for the display device to display an operation menu. [Diagram 41] 11 is a flowchart illustrating an example of a process performed when a button on a pen operation panel is pressed. [Diagram 42] 13 is a flowchart illustrating an example of a process performed when a user executes an operation command "set to page name." [Diagram 43] 11 is a flowchart illustrating an example of a method for displaying a page navigation. [Diagram 44] 10A to 10C are diagrams illustrating examples of arrangement of pen holders. [Diagram 45] 13 is an example of an arrangement determination screen. [Figure 46] FIG. 13 is a diagram showing another configuration example of the display device. [Figure 47] FIG. 13 is a diagram showing another configuration example of the display device. [Figure 48] FIG. 13 is a diagram showing another configuration example of the display device. [Figure 49] FIG. 13 is a diagram showing another configuration example of the display device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A display device and a display method performed by the display device will be described below as an example of an embodiment of the present invention with reference to the drawings. EXAMPLES

[0010] The input means may be any means that allows handwriting by specifying coordinates on a touch panel. Examples include a pen, a human finger or hand, or a rod-shaped member. Line-of-sight input may also be possible. Handwritten data is data that displays, as a trajectory, a sequence of coordinate points that a user moves continuously with the input means on a touch panel. A series of operations in which a user presses the input means against the touch panel, moves it continuously, and then releases it from the touch panel is called a stroke. Data handwritten using strokes is called stroke data. Handwritten data has one or more strokes. Handwritten input refers to handwritten data being input by a user.

[0011] <Comparative Example of Display Device> In describing the display device of this embodiment, first, a display example of an operation menu of the display device will be briefly described.

[0012] 1 shows an example of an operation menu displayed by a display device. Since this display device 2 is propped up, an operation menu 102 is displayed on the right side of the display. Although the user can move the operation menu 102 to the left side of the display, conventionally, only one operation menu 102 is displayed at a time. For this reason, for example, even if a user is present on the left side facing the display, the user has difficulty pressing the operation menu 102.

[0013] Next, Fig. 2 shows an example of a flat-laying display device 2. In Fig. 2, the users face each other, but in this case too, it is difficult to operate if there is only one operation menu. Therefore, it is being considered to have the display device 2 display an operation menu for each user. The operation menu does not have information regarding data processing according to the display position.

[0014] In this case, the user can press an operation menu near him / her, but the operation menu does not include information regarding data processing according to the display position, and it is difficult for the user to process data according to the display position even if the user presses the operation menu. For example, when the display device 2 recognizes handwritten characters, the handwritten data cannot be correctly recognized unless it is rotated according to the orientation of the characters, but conventionally, the content of data processing has not been changed according to the pressed operation menu.

[0015] In this embodiment, a display device that displays an operation menu having information regarding data processing according to the display position will be described.

[0016] 1, the display device stores page data including handwritten data on a page-by-page basis, and displays thumbnails 106 of each page at the bottom of the display. However, the thumbnails 106 are small, which is inconvenient because it is difficult to distinguish the contents. Also, there is an upper limit to the number of pages that can be displayed with the thumbnails 106, so it is not possible to display all of the contents contained on each page.

[0017] Therefore, in this embodiment, character strings selected by the user or automatically selected by the display device are extracted for each page, and a page navigation displaying these character strings is displayed for each page.

[0018] <Terminology> Handwritten data is data that displays a sequence of coordinate points on a display where a user moves an input tool continuously, as a trajectory. A series of operations in which a user presses an input tool against a display, moves it continuously, and then removes it from the display is called a stroke, and data handwritten using strokes is called stroke data. Handwritten data has one or more stroke data.

[0019] The information regarding data processing is the content of processing of handwritten data or data resulting from handwritten input displayed by the display device 2. For example, rotation of handwritten data for handwriting recognition and rotation after character recognition can be included. The display device 2 can rotate handwritten data according to the display position of the operation menu. Furthermore, if the display device is a color display, it can also control whether to display in color or monochrome, and in which color. The display device 2 can also control which font to display.

[0020] Black and white compatible device: A black and white compatible device is a device that can only output black or white, or grayscale. Examples include electronic paper and monochrome printers. It can also be said to be a device that does not support color.

[0021] Electronic paper is also called E-paper. Electronic paper is a general term for displays that are about 1 / 10th of a millimeter thick and can display and erase data electrically. The representative technology for realizing electronic paper is the microcapsule electrophoresis method, known as E-ink (registered trademark) technology. Each microcapsule contains white and black particles, with the white particles being positively charged and the black particles being negatively charged.

[0022] Color-compatible devices: Color-compatible devices are devices that can output color as well as black and white. These include devices with LCD or OLED displays, and color printers. Color-compatible devices are also black and white compatible devices.

[0023] A black and white emphasis device is a device that can only output black or white, or grayscale, but performs black and white emphasis display based on color information associated with handwritten data with respect to handwritten input-derived data. An example is the display device of this embodiment. Note that the display device is capable of handwritten input, and may also be called an input device or handwriting input device.

[0024] Data resulting from handwritten input refers to data that is input by handwriting on a touch panel. It does not matter whether the data is left handwritten or converted to text data after input. In addition, data obtained from an external device is also considered to be caused by handwritten input. Data resulting from handwritten input may include text data converted by character recognition, as well as data converted based on user operations, such as set characters such as "Done" and "Confidential," stamps displayed as marks, shapes such as circles and stars, and straight lines.

[0025] Data that is not derived from handwritten input refers to data other than data input by handwriting on a touch panel, such as image data, handwritten data input on a device that does not support black and white emphasis, or text data.

[0026] <Example of pen appearance> FIG. 3 shows an example of a perspective view of the pen 2500. FIG. 3 shows an example of a multi-function pen 2500. The pen 2500 that has a built-in power source and can send commands to the display device 2 is called an active pen (a pen without a built-in power source is called a passive pen). The pen 2500 in FIG. 3 has one physical switch at the tip, one at the end, and two on the sides of the pen, the tip of the pen is for writing, the end of the pen is for erasing, and the sides of the pen are for allocating user functions. In this embodiment, the pen 2500 further has a non-volatile memory and stores a pen ID that is unique to other pens.

[0027] A pen with a switch can reduce the number of steps a user must take to operate the display device 2. Although the term "pen with a switch" primarily refers to an active pen, in the case of the electromagnetic induction method, even passive pens that do not have a built-in power source can generate power using only an LC circuit, so it includes not only active pens but also passive pens that use the electromagnetic induction method. Pens with a switch that use optical, infrared, and capacitive methods other than the electromagnetic induction method are all active pens.

[0028] The hardware configuration of the pen 2500 is assumed to be similar to that of a general control system equipped with a communication function and a microcomputer. The pen 2500 may be of an electromagnetic induction type or an active electrostatic coupling type. The pen 2500 may also have functions such as pressure detection, tilt detection, and a hover function (displaying the cursor before the pen touches the screen).

[0029] <Overall configuration of the device> The overall configuration of the display device 2 according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing the overall configuration of the display device 2. Fig. 4(a) shows, as an example of the display device 2, a display device 2 used as a horizontally long electronic whiteboard hung on a wall.

[0030] 4(a), a display 220 as an example of a display device is provided on the upper part of the display device 2. A user U can handwrite (also referred to as input or drawing) characters or the like on the display 220 using a pen 2500.

[0031] FIG. 4(b) shows a display device 2 used as a vertically long electronic whiteboard suspended on a wall.

[0032] 4(c) shows the display device 2 placed flat on a desk 230. Since the display device 2 is about 1 cm thick, there is no need to adjust the height of the desk even if it is placed flat on a normal desk. In addition, the user can easily move the display device 2.

[0033] The tilt sensor automatically detects how the display device 2 is placed when it is being used.

[0034] <Hardware configuration of the device> Next, the hardware configuration of the display device 2 will be described with reference to Fig. 5. As shown in the figure, the display device 2 has the configuration of an information processing device or a computer. Fig. 5 is an example of a hardware configuration diagram of the display device 2. As shown in Fig. 5, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an SSD (Solid State Drive) 204.

[0035] Of these, the CPU 201 controls the overall operation of the display device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201. The SSD 204 stores various data, such as programs for the display device 2.

[0036] The display device 2 also includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, a power switch 227, a tilt sensor 217, a serial interface 218, a speaker 219, a microphone 221, a wireless communication device 222, an infrared I / F 223, a power control circuit 224, an AC adapter 225, and a battery 226.

[0037] The display controller 213 controls and manages the screen display in order to output an output image to the display 220, etc. The touch sensor 216 detects that the pen 2500, the user's hand, etc. (the pen and the user's hand serve as input means) have touched the display 220. The touch sensor 216 also receives a pen ID.

[0038] The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 inputs and detects coordinates. An example of a method for inputting and detecting coordinates will be described. For example, in the case of an optical type, two light receiving and emitting devices installed at both ends of the upper side of the display 220 emit a plurality of infrared rays in parallel to the display 220. A reflecting member installed around the display 220 reflects the plurality of infrared rays, and receives the light returning on the same optical path as the optical path of the light emitted by the light receiving element. The touch sensor 216 outputs position information of the infrared rays emitted by the two light receiving and emitting devices that are blocked by an object to the touch sensor controller 215, and the touch sensor controller 215 specifies the coordinate position that is the contact position of the object. In addition, the touch sensor controller 215 has a communication unit 215a and can communicate wirelessly with the pen 2500. For example, when communication is performed according to a standard such as Bluetooth (registered trademark), a commercially available pen can be used. If one or more pens 2500 are registered in advance in the communication unit 215a, the user can communicate with the display device 2 without having to perform connection settings to allow the pen 2500 to communicate with the display device 2.

[0039] The power switch 227 is a switch for switching ON / OFF the power of the display device 2. The tilt sensor 217 is a sensor for detecting the tilt angle of the display device 2. The tilt sensor 217 is mainly used for detecting whether the display device 2 is being used in the installation state of Fig. 4(a), Fig. 4(b), or Fig. 4(c), and can automatically change the thickness of characters, etc. according to the installation state.

[0040] The serial interface 218 is an external communication interface such as a USB or LAN interface. The serial interface 218 is used for inputting information from the outside. The speaker 219 is used for outputting audio, and the microphone 221 is used for inputting audio. The wireless communication device 222 communicates with a terminal carried by the user, and relays a connection to the Internet, for example. The wireless communication device 222 communicates using Wi-Fi, Bluetooth (registered trademark), or the like, but any communication standard can be used. The wireless communication device 222 forms an access point, and when the user sets the SSID (Service Set Identifier) ​​and password obtained by the user in the terminal carried by the user, the user can connect to the access point.

[0041] It is preferable that the wireless communication device 222 is provided with two access points. a. Access point → Internet b. Access point → Internal network → Internet Access point a is for users outside the company, and users cannot access the company network but can use the Internet. Access point b is for users inside the company, and users can use the company network and the Internet.

[0042] The infrared I / F 223 detects the display device 2 arranged adjacently. The infrared I / F 223 can detect only the display device 2 arranged adjacently by utilizing the rectilinearity of infrared rays. It is preferable that one infrared I / F 223 is provided on each side, and it can detect in which direction of the display device 2 another display device 2 is arranged. The adjacent display device 2 can display handwritten information that was handwritten in the past (handwritten information of another page, with the area of ​​one display 220 being one page).

[0043] A power supply control circuit 224 controls an AC adapter 225 and a battery 226 which are power sources for the display device 2. The AC adapter 225 converts AC common to commercial power sources into DC.

[0044] If the display 220 is so-called electronic paper, it consumes little or no power to maintain an image after it has been drawn, and therefore can be driven by a battery 226. This makes it possible to use the display device 2 for purposes such as digital signage even in places where it is difficult to connect to a power source, such as outdoors.

[0045] The display device 2 further includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201 shown in FIG.

[0046] The touch sensor 216 is not limited to an optical type, and may be a capacitive touch panel that identifies the contact position by detecting a change in capacitance. The touch sensor 216 may be a resistive film type touch panel that identifies the contact position by a voltage change in two opposing resistive films. The touch sensor 216 may use various detection means, such as an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by a contact object touching the display unit. The touch sensor 216 may be of a type that does not require an electronic pen to detect the presence or absence of a touch with the tip of the pen. In this case, a fingertip or a pen-shaped stick can be used to perform a touch operation. The pen 2500 does not need to be a long, thin pen type.

[0047] <About the device functions> Next, the functions of the display device 2 and the pen 2500 will be described with reference to FIG. 6. FIG. 6(a) is an example of a functional block diagram showing the functions of the display device 2 in a block form. The display device 2 includes a handwriting input unit 21, a display unit 22, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a character string conversion control unit 28, a character string conversion dictionary unit 29, a predictive conversion control unit 30, a predictive conversion dictionary unit 31, an operation command recognition control unit 32, an operation command definition unit 33, a pen ID control data storage unit 36, a file transmission / reception control unit 37, a handwritten signature authentication control unit 38, and a handwritten signature data storage unit 39. Each function of the display device 2 is a function or means realized by any of the components shown in FIG. 5 operating in response to an instruction from the CPU 201 in accordance with a program expanded from the SSD 204 onto the RAM 203.

[0048] The handwriting input unit 21 is realized by a touch sensor 216 etc., accepts handwriting input by a user, and receives a pen ID from the pen 2500. The handwriting input unit 21 converts the user's pen input d1 into pen operation data d2 (pen up, pen down, or pen coordinate data) with the pen ID, and transmits it to the handwriting input display control unit 23. The pen coordinate data is periodically transmitted as discrete values, and the coordinates between the discrete values ​​are interpolated.

[0049] The display unit 22 is realized by a display 220 etc., and displays handwritten data, operation menus, etc. The display unit 22 converts the drawing data d3 written to the video memory by the handwritten input display control unit 23 into data according to the characteristics of the display 220 and transmits it to the display 220.

[0050] The handwriting input display control unit 23 performs overall control over handwriting input and display. The handwriting input display control unit 23 processes pen operation data d2 from the handwriting input unit 21 and transmits it to the display unit 22 for display. Details of the processing of the pen operation data d2 and the display of strokes will be described later with reference to Figs. 32 to 39. The handwriting input display control unit 23 holds operation button definition data, which will be described later, detects an operation menu (button) operated based on the pen operation data d2 from the handwriting input unit 21, and performs control according to the operation menu.

[0051] The candidate display timer control unit 24 is a display control timer for selectable candidates. The candidate display timer control unit 24 starts or stops the timer to generate the timing for starting the display of selectable candidates and the timing for erasing the display. The selectable candidates are handwriting recognition string / language string candidates, conversion string candidates, string / predictive conversion candidates, and operation command candidates that are selectably displayed in an operation guide (see FIG. 19 ) described later. The candidate display timer control unit 24 receives a timer start request d4 (or a timer stop request in some cases) from the handwriting input display control unit 23, and transmits a timeout event d5 to the handwriting input display control unit 23.

[0052] The handwriting input saving unit 25 has a storage function for saving user data (handwritten data / character string data). The handwriting input saving unit 25 receives and holds user data d6-1 from the handwriting input display control unit 23. The handwriting input saving unit 25 receives an acquisition request d6-2 from the handwriting input display control unit 23 and transmits user data d7 saved in the handwriting input saving unit 25. The handwriting input saving unit 25 transmits position information d36 of confirmed data (character string data that has already been recognized or handwritten data that has been confirmed not to be converted) to the operation command recognition control unit 32.

[0053] The handwriting recognition control unit 26 is a recognition engine that performs online handwriting recognition. Unlike a typical OCR (Optical Character Reader), it recognizes characters (not only Japanese but also multiple languages ​​such as English), numbers, symbols (%, $, &, etc.), figures (lines, circles, triangles, etc.), etc. in parallel with the user's pen operation. Various algorithms have been devised for the recognition method, but details will be omitted since known techniques can be used in this embodiment.

[0054] The handwriting recognition control unit 26 receives the pen operation data d8-1 from the handwriting input display control unit 23, performs handwriting recognition, and holds the handwriting-recognized character string candidates. The handwriting recognition control unit 26 also holds the language character string candidates converted from the handwriting-recognized character string candidates d12 using the handwriting recognition dictionary unit 27. When the handwriting recognition control unit 26 separately receives an acquisition request d8-2 from the handwriting input display control unit 23, it transmits the held handwriting-recognized character string candidates and language character string candidates d9 to the handwriting input display control unit 23.

[0055] The handwriting recognition dictionary unit 27 stores dictionary data for language conversion in handwriting recognition. The handwriting recognition dictionary unit 27 receives handwriting recognition character string candidates d12 from the handwriting recognition control unit 26, converts them into linguistically probable language character string candidates d13, and transmits them to the handwriting recognition control unit 26. For example, in the case of Japanese, hiragana is converted into kanji or katakana.

[0056] The character string conversion control unit 28 controls the conversion of the conversion character string candidates into character strings. A conversion character string is a character string that is likely to be generated by including a handwriting recognition character string or a language character string. The character string conversion control unit 28 receives the handwriting recognition character string and the language character string candidates d11 from the handwriting recognition control unit 26, converts them into conversion character string candidates using the character string conversion dictionary unit 29, and stores them. Separately, when the character string conversion control unit 28 receives an acquisition request d14 from the handwriting input display control unit 23, it transmits the stored conversion character string candidates d15 to the handwriting input display control unit 23.

[0057] The character string conversion dictionary unit 29 stores dictionary data for character string conversion. The character string conversion dictionary unit 29 receives handwritten recognition character strings and language character string candidates d17 from the character string conversion control unit 28, and transmits converted character string candidates d18 to the character string conversion control unit 28.

[0058] The predictive conversion control unit 30 receives the handwriting-recognized character string and the language string candidate d10 from the handwriting recognition control unit 26, and receives the converted character string candidate d16 from the character string conversion control unit 28. The predictive conversion control unit 30 converts each of the handwriting-recognized character string, the language string candidate, and the converted character string candidate into a predicted character string candidate using the predictive conversion dictionary unit 31. The predicted converted character string is a character string that is likely to be generated by including the handwriting-recognized character string, the language string, or the converted character string. When the predictive conversion control unit 30 separately receives an acquisition request d19 from the handwriting input display control unit 23, it transmits the predicted character string candidate d20 to the handwriting input display control unit 23.

[0059] The predictive conversion dictionary unit 31 stores dictionary data for predictive conversion. The predictive conversion dictionary unit 31 receives handwritten recognition character strings, language character string candidates, and converted character string candidates d21 from the predictive conversion control unit 30, and transmits predicted character string candidates d22 to the predictive conversion control unit 30.

[0060] The operation command recognition control unit 32 receives the handwriting-recognized character string and the language string candidate d30 from the handwriting recognition control unit 26, and receives the converted character string candidate d28 from the character string conversion control unit 28. The operation command recognition control unit 32 receives the predicted character string candidate d29 from the predictive conversion control unit 30. Then, the operation command recognition control unit 32 transmits an operation command conversion request d26 for each of the handwriting-recognized character string, the language string candidate, the converted character string candidate, and the predicted character string candidate d29 to the operation command definition unit 33, and receives the operation command candidate d27 from the operation command definition unit 33. The operation command recognition control unit 32 holds the operation command candidate d27.

[0061] If the operation command conversion request d26 partially matches the operation command definition, the operation command definition unit 33 transmits operation command candidates d27 to the operation command recognition control unit 32.

[0062] Furthermore, the operation command recognition control unit 32 receives pen operation data d24-1 from the handwriting input display control unit 23. The operation command recognition control unit 32 transmits a position information acquisition request d23 of previously input and confirmed confirmed data to the handwriting input storage unit 25, and stores the confirmed data designated by the pen operation data as selected data (including position information). The operation command recognition control unit 32 identifies the position of the pen operation data d24-1 and selected data that satisfies a predetermined criterion. Separately, when the operation command recognition control unit 32 receives an acquisition request d24-2 from the handwriting input display control unit 23, it transmits the selected data d25 identified as a candidate for the stored operation command to the handwriting input display control unit 23.

[0063] The pen ID control data storage unit 36 ​​holds pen ID control data (which may be called a storage unit). Before the handwriting input display control unit 23 transmits display data to the display unit 22, the pen ID control data storage unit 36 ​​transmits pen ID control data d41 to the handwriting input display control unit 23. The handwriting input display control unit 23 draws the display data under the operating conditions stored in association with the pen ID. In addition, before the handwriting recognition control unit 26 executes handwriting recognition, the pen ID control data storage unit 36 ​​transmits angle information d44 of the pen ID control data to the handwriting recognition control unit 26, and the handwriting recognition control unit 26 rotates the stroke using the angle information stored in association with the pen ID to execute handwriting recognition.

[0064] Furthermore, after recognizing a straight line for setting angle information when the user handwrites characters, etc., the handwriting recognition control unit 26 transmits angle information d43 of the pen ID control data to the pen ID control data storage unit 36. The handwriting recognition control unit 26 stores the angle information d43 in association with the pen ID. Furthermore, after the handwriting input display control unit 23 executes an operation command for setting angle information, the handwriting input display control unit 23 transmits pen ID control data d42 to the pen ID control data storage unit 36. The handwriting input display control unit 23 stores the execution result of the operation command (angle information set by the user) in association with the pen ID. Thereafter, the stroke of that pen ID is rotated by the set angle information before handwriting recognition is performed.

[0065] Furthermore, the handwriting recognition control unit 26 transmits stroke data d49 rotated clockwise using the angle information of the pen ID control data to the handwritten signature authentication control unit 38. This makes it possible to authenticate a handwritten signature regardless of the user's operation position (the direction from which the handwriting is made with respect to the display device 2).

[0066] The handwritten signature data storage unit 39 holds handwritten signature data. When the handwritten signature data storage unit 39 receives a handwritten signature data acquisition request d45 from the handwritten signature authentication control unit 38, it transmits handwritten signature data d46 to the handwritten signature authentication control unit 38. The format of the handwritten signature data depends on the handwritten signature authentication algorithm of the handwritten signature authentication control unit 38. The data in the handwritten signature data storage unit 39 will be described with reference to FIG. 13.

[0067] When the handwritten signature authentication control unit 38 receives the clockwise rotated stroke data d49 from the handwriting recognition control unit 26, it transmits a handwritten signature data acquisition request d45 to the handwritten signature data storage unit 39. The handwritten signature data storage unit 39 transmits handwritten signature data d46 to the handwritten signature authentication control unit 38.

[0068] The handwritten signature authentication control unit 38 performs user authentication based on the handwritten signature data. Various algorithms have been devised for authenticating users based on handwritten signature data, but in this embodiment, a technology that can recognize with a recognition rate that does not impair practical use is used. For example, the handwritten signature authentication control unit 38 generates a feature vector whose elements are the coordinates, writing pressure, and time to write the stroke that constitute the handwritten signature data. The handwritten signature authentication control unit 38 appropriately weights the elements and compares the feature vector of the registered handwritten signature data with the feature vector of the user name handwritten by the user when signing in. If the degree of match is equal to or greater than a threshold, the handwritten signature authentication control unit 38 determines that the authentication is successful, and if it is less than the threshold, determines that the authentication is unsuccessful.

[0069] The handwritten signature authentication control unit 38 holds the authentication result of the handwritten signature, which is the comparison result between the stroke data d49 and the handwritten signature data d46. When an acquisition request d48 is separately received from the handwritten input display control unit 23, the handwritten signature authentication control unit 38 transmits the held authentication result d47 of the handwritten signature to the handwritten input display control unit 23. The authentication result of the handwritten signature indicates whether or not the stroke data d49 and the handwritten signature data d46 can be regarded as matching, and if they are considered to match, has a SignatureId (described later) associated with the matching handwritten signature data d46.

[0070] If the handwriting recognition result of handwriting recognition control unit 26 matches an operation command instructing the execution of handwritten signature registration, handwriting recognition control unit 26 obtains data d52 inputted into a handwritten signature registration form (a frame into which handwritten signature data is inputted, as described below) from handwriting input storage unit 25. Handwriting recognition control unit 26 transmits handwritten signature data d50 out of data d52 to handwritten signature authentication control unit 38. Handwritten signature authentication control unit 38 transmits the received handwritten signature data d50 to handwritten signature data storage unit 39 for registration.

[0071] If the handwriting recognition result of the handwriting recognition control unit 26 is an instruction to cancel the handwritten signature or to execute registration, the handwriting recognition control unit 26 sends a request d51 to delete the handwritten signature registration form to the handwriting input storage unit 25 and deletes the handwritten signature registration form from the handwriting input storage unit 25.

[0072] When the handwriting recognition result of handwriting recognition control unit 26 is an instruction to change the user-defined data, handwriting recognition control unit 26 acquires data d53 input to the user-defined data change form from handwriting input storage unit 25. Handwriting recognition control unit 26 transmits change value d54 of data d53 to operation command definition unit 33 to change the user-defined data. The user-defined data will be described with reference to FIG. 12.

[0073] If the handwriting recognition result of the handwriting recognition control unit 26 is an instruction to cancel the user-defined data change form or to execute registration, the handwriting recognition control unit 26 sends a request d55 to delete the user-defined data change form to the handwriting input storage unit 25 and deletes the user-defined data change form from the handwriting input storage unit 25.

[0074] The file transmission / reception control unit 37 stores and acquires handwriting-input-derived data or data files not originating from handwriting input in a storage medium, communicates with external devices (printing requests, display requests, etc.), etc. The file transmission / reception control unit 37 receives a request d64 to send or receive a file from the handwriting input display control unit 23. When saving or printing a file, the handwriting input display control unit 23 transmits a file transmission request to the file transmission / reception control unit 37, and the file transmission / reception control unit 37 transmits a request to acquire handwriting input saved data d61 to the handwriting input saving unit 25. In the case of data that is not handwritten input-derived data, the handwritten input storage unit 25 transmits the color information held in the data as is. In the case of handwritten input origin data, the handwritten input saving unit 25 converts the data into color and sends the handwritten input saved data d62 to the file transmission / reception control unit 37 if the destination is a color compatible device (for example, a color printer). In the case of data resulting from handwriting input, if the destination is a black-and-white compatible device (e.g., a monochrome printer), the handwriting input storage unit 25 transmits the handwriting input saved data d62 converted to black-and-white to the file transmission / reception control unit 37. Since the black-and-white compatible device may be capable of converting to grayscale, the data may be converted to color before transmission. When the destination is a device that supports black and white emphasis, the handwritten input storage data, which will be described later, is transmitted to the file transmission / reception control unit 37. Furthermore, when the handwriting input saving unit 25 saves the handwriting input originated data in a file, it converts the handwriting input saved data d62 into color in accordance with the file format and attaches the handwriting input saved data as metadata for the file. When saving data that is not handwriting input originated data in a file, it converts the handwriting input saved data d62 into color in accordance with the file format.

[0075] Whether a device is color-compatible or monochrome-compatible is stored, for example, in an MIB (Management Information Base) held by the network device, and the file transmission / reception control unit 37 can determine this by acquiring the MIB. Similarly, whether a device is monochrome emphasis-compatible can also be determined from the model name and the like disclosed by the MIB.

[0076] When reading a file, the handwritten input display control unit 23 transmits a file list acquisition request d65 to the file transmission / reception control unit 37. The file transmission / reception control unit 37 transmits a file list acquisition request to the external device, acquires a file list d63, and transmits it to the handwritten input display control unit 23. The handwritten input display control unit 23 displays the file list on the display unit 22, the handwritten input unit 21 transmits the display position of the selected file to the handwritten input display control unit 23, and the handwritten input display control unit 23 transmits a file reception request d66 to the file transmission / reception control unit 37. The file transmission / reception control unit 37 acquires a file from the external device and transmits this file d67 to the handwritten input storage unit 25. The handwritten input storage unit 25 analyzes the metadata of the file to determine whether it is handwritten input-derived data, and if it is handwritten input-derived data, extracts handwritten input storage data (black and white highlight / color convertible data, which will be described later). The file transmission / reception control unit 37 converts the handwritten input-derived data to black and white highlight display, and if it is not handwritten input-derived data, transmits it to the handwritten input display control unit 23 without conversion (displayed in grayscale). The handwritten input display control unit 23 transmits display data of the handwritten data to the display unit 22 .

[0077] 6(b) is a functional block diagram showing the functions of the pen 2500. The pen 2500 has a pen event transmission unit 41. The pen event transmission unit 41 attaches a pen ID to event data of pen-up, pen-down, and pen coordinates and transmits them to the display device 2.

[0078] <About predefined control data> Next, the predefined control data used by the display device 2 for various processes will be described with reference to Fig. 7. Fig. 7 shows an example of the predefined control data. The predefined control data in Fig. 7 shows control data for each control item.

[0079] The selectable candidate display timer 401 defines the time until the selectable candidates are displayed. This is because the display device 2 does not display the selectable candidates while handwriting. In Fig. 7, this means that the selectable candidates are displayed if the pen is not down within TimerValue=500 [ms] after the pen is up. The selectable candidate display timer 401 is held by the candidate display timer control unit 24. The selectable candidate display timer 401 is used when the selectable candidate display timer is started in step S18-2 in Fig. 34, which will be described later.

[0080] The selectable candidate deletion timer 402 defines the time until the displayed selectable candidates are deleted. This is to delete the selectable candidates if the user does not select a selectable candidate. In Fig. 7, this means that if a selectable candidate is not selected within TimerValue = 5000 [ms] from the display of the selectable candidates, the selectable candidate display data is deleted. The selectable candidate deletion timer 402 is held by the candidate display timer control unit 24. The selectable candidate deletion timer 402 is used when the selectable candidate display deletion timer is started in step S64 in Fig. 36.

[0081] The handwritten data rectangular area 403 defines a rectangular area that is considered to be in the vicinity of the handwritten data. In the example of FIG. 7, the handwritten data rectangular area 403 is a rectangular area obtained by enlarging the handwritten data rectangular area by 50% (Horizontal) of the estimated character size in the horizontal direction and by 80% (Vertical) of the estimated character size in the vertical direction. In the example of FIG. 7, the rectangular area is a percentage of the estimated character size (specified as a percentage), but it is also possible to set the length to a fixed length by using units such as "mm". The handwritten data rectangular area 403 is held by the handwritten input storage unit 25. The font data 405 is used in step S10 of FIG. 33 to determine whether the handwritten data rectangular area and the stroke rectangular area overlap.

[0082] The estimated writing direction / character size judgment condition 404 defines constants for judging the writing direction and the measurement direction of the character size. In the example of FIG. The difference between the time when the first stroke was added to the rectangular area of ​​the handwritten data and the time when the last stroke was added is MinTime = 1000 [ms] or more, The difference between the horizontal distance (width) and vertical distance (height) of the rectangular area of ​​the handwritten data is MinDiff=10 [mm] or more, If the horizontal distance is longer than the vertical distance, This means that the estimated writing direction is determined to be "horizontal writing" and the estimated character size is determined to be the vertical distance. If the horizontal distance is shorter than the vertical distance, this means that the estimated writing direction is determined to be "vertical writing" and the estimated character size is determined to be the horizontal distance. If the above conditions are not met, the estimated writing direction is determined to be "horizontal writing" (DefaultDir="Horizontal") and the estimated character size is determined to be the longer of the horizontal distance and the vertical distance. The estimated writing direction / character size determination condition 404 is held by the handwriting input storage unit 25. The estimated writing direction / character size determination condition 404 is used in obtaining the estimated writing direction in step S59 of FIG. 36 and in obtaining the character string data font in step S81 of FIG. 38.

[0083] Font data 405 defines data for estimating the size of characters, etc. In the example of FIG. 7, this means that the estimated character size determined by estimated writing direction / character size determination condition 404 is compared with small characters 405a (hereinafter referred to as minimum font size) and large characters 405c (hereinafter referred to as maximum font size) of font data 405. If the estimated character size is smaller than the minimum font size, the estimated character size is determined to be the minimum font size. If the estimated character size is larger than the maximum font size, the estimated character size is determined to be the maximum font size. Otherwise, the character size is determined to be that of medium characters 405b. Font data 405 is held by handwritten input storage unit 25. Font data 405 is used in acquiring the character string data font in step S81 of FIG. 38.

[0084] Specifically, the handwriting input saving unit 25 compares the estimated character size determined by the estimated writing direction / character size determination condition 404 with the FontSize of the font data 405, and uses the font with the closest size. For example, the handwriting input saving unit 25 determines the estimated character size as "small characters" when it is 25 mm (FontSize of small characters) or less. The handwriting input saving unit 25 determines the estimated character size as "medium characters" when it is more than 25 mm and less than 50 mm (FontSize of medium characters). The handwriting input saving unit 25 determines the estimated character size as "large characters" when it is more than 100 mm (FontSize of large characters). "Small characters" 405a uses a 25mm Ming font (FontStyle="Ming font" FontSize="25mm"), "medium characters" 405b uses a 50mm Ming font (FontStyle="Ming font" FontSize="50mm"), and "large characters" 405c uses a 100mm Gothic font (FontStyle="Gothic font" FontSize="100mm"). If there is a need to increase the variety of font sizes or styles, the display device provider can simply increase the variety of font data 405.

[0085] The crossing line determination condition 406 defines the data used to determine whether multiple display data are selected. This data is a single stroke. In the example of FIG. The length of the long side of the handwritten data is 100 mm or more (MinLenLongSide="100mm"), and - The length of the short side is 50 mm or less (MaxLenShortSide="50mm"), and - The overlap rate with the handwritten data in the long and short sides is 80% or more (MinOverLapRate="80%") If there is such display data, it is determined that a plurality of display data have been selected (selected data). The crossing line determination condition 406 is held by the operation command recognition control unit 32. The crossing line determination condition 406 is used in the crossing line determination for determining the selected data in step S50 of FIG.

[0086] The encircling line determination condition 407 defines data used to determine whether the handwritten data is an encircling line. In the example of Fig. 7, the operation command recognition control unit 32 determines that the finalized data, in which the overlap rate of the long side direction and the short side direction of the handwritten data is 100% or more (MinOverLapRate="100%"), is selected data. The encircling line determination condition 407 is held by the operation command recognition control unit 32. The encircling line determination condition 407 is used in the encircling line determination for determining the selected data in step S50 of Fig. 35.

[0087] Either of the crossing line determination condition 406 and the encircling line determination condition 407 may be given priority in the determination. For example, if the crossing line determination condition 406 is loosened (to make it easier to select a crossing line) and the encircling line determination condition 407 is set to a strict value (to make it possible to select only an encircling line), the operation command recognition control unit 32 may give priority to the encircling line determination condition 407 in the determination.

[0088] <An example of dictionary data> The dictionary data will be described with reference to Fig. 8 to Fig. 10. Fig. 8 is an example of dictionary data in handwriting recognition dictionary unit 27. Fig. 9 is an example of dictionary data in character string conversion dictionary unit 29. Fig. 10 is an example of dictionary data in predictive conversion dictionary unit 31. These dictionary data are used in steps S33 to S41 in Fig. 35, respectively.

[0089] In this embodiment, the conversion result by the dictionary data of the handwriting recognition dictionary unit 27 in FIG. 8 is called a language string candidate, the conversion result by the dictionary data of the string conversion dictionary unit 29 in FIG. 9 is called a converted string candidate, and the conversion result by the dictionary data of the predictive conversion dictionary unit 31 in FIG. 10 is called a predicted string candidate. "Before conversion" of each dictionary data is a string to search the dictionary data, "after conversion" is a converted string corresponding to the string to be searched, and "probability" represents the probability of selection by the user. The probability is calculated from the result of the user selecting each string in the past. Therefore, the probability may be calculated for each user. Various algorithms have been devised as a method of calculating the probability, but it is sufficient to calculate it by an appropriate method as appropriate, and details will be omitted. This embodiment is characterized in that the string candidates are displayed in descending order of selection probability from the estimated writing direction.

[0090] The dictionary data of the handwriting recognition dictionary unit 27 in Fig. 8 indicates that handwritten "gi" is converted to "gi" with a probability of 0.55 and to "gi" with a probability of 0.45, and handwritten "gishi" is converted to "gishi" with a probability of 0.55 and to "gishi" with a probability of 0.45. The same is true for other "before conversion" character strings. In Fig. 8, the "before conversion" character string is handwritten hiragana, but characters other than hiragana may be registered as "before conversion".

[0091] 9 indicates that the character string "" is converted to "" with a probability of 0.95, and the character string "" is converted to "" with a probability of 0.85. The same applies to the other "pre-conversion" character strings.

[0092] The dictionary data of the predictive conversion dictionary unit 31 in Fig. 10 indicates that the character string "Minutes" is converted to "Destination of minutes" with a probability of 0.65, and the character string "Skill test" is converted to "Approve skill test" with a probability of 0.75. In the example of Fig. 10, all the character strings before conversion are kanji, but characters other than kanji may be registered.

[0093] The dictionary data is not language-dependent, and any character string may be registered before and after conversion.

[0094] <Operation command definition data held by the operation command definition section> Next, the operation command definition data used by the operation command recognition control unit 32 will be described with reference to Fig. 11. Fig. 11 shows an example of the operation command definition data and system definition data held by the operation command definition unit 33.

[0095] Fig. 11(a) shows an example of operation command definition data. The operation command definition data in Fig. 11(a) is an example of operation command definition data in the case where there is no selection data selected by handwritten data, and covers all operation commands that can be executed by the display device 2. The operation command in Fig. 11(a) has an operation command name (Name), a string (String) that partially matches a string candidate, and an operation command string (Command) to be executed. "%~%" in the operation command string is a variable, and is associated with system definition data as shown in Fig. 11(b). In other words, "%~%" is replaced with the system definition data shown in Fig. 11(b).

[0096] First, the operation command definition data 701 indicates that the operation command name is "Read minutes template", the string that partially matches the string candidate is "minutes" or "template", and the operation command string to be executed is "ReadFile https: / / %username%:%password%@server.com / template / minutes.pdf". In this example, the operation command string to be executed contains the system definition data "%~%", and "%username%" and "%password%" are replaced by the system definition data 704 and 705, respectively. Therefore, the operation command string to be finally executed is the string "ReadFile https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf", which indicates that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf" is to be read (ReadFile).

[0097] Operation command definition data 702 indicates that the operation command name is "Save in minutes folder", the string that partially matches the string candidate is "Minutes" or "Save", and the operation command string to be executed is "WriteFile https: / / %username%:%password%@server.com / minutes / %machinename%_%yyyy-mm-dd%.pdf". As with operation command definition data 701, "%username%", "%password%", and "%machinename%" in the operation command string are replaced with system definition data 704 to 706, respectively. Note that "%yyyy-mm-dd%" indicates that it should be replaced with the current date. For example, if the current date is September 26, 2018, it should be replaced with "2018-09-26". The final operation command to be executed is "WriteFile https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf", which indicates that the minutes will be saved (WriteFile) to the file "https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf".

[0098] The operation command definition data 703 indicates that the operation command name is "Print", the character string that partially matches the character string candidate is "Print" or "Print", and the operation command character string to be executed is "PrintFile https: / / %username%:%password%@server.com / print / %machinename%-"%yyyy-mm-dd%.pdf". When the operation command character string is replaced in the same manner as the operation command definition data 702, the operation command to be finally executed becomes "PrintFile https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf", which indicates that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf" is to be printed (PrintFile). In other words, the file is sent to the server. When the user makes the printer communicate with the server and specifies a file, the printer prints the contents of the file on paper.

[0099] In this way, the operation command definition data 701-703 can be identified from the character string candidates, so that the user can display the operation command by handwriting. Also, if the user authentication is successful, "%username%", "%password%", etc. of the operation command definition data are replaced with the user information, making it possible to input and output files in association with the user.

[0100] If user authentication is not performed (including the case where authentication fails but the user can use the display device 2), the display device 2 replaces it with "%username%", "%password%", etc., that are preset for the display device 2. Therefore, file input / output becomes possible in association with the display device 2 even without user authentication.

[0101] The operation command definition data 709, 710, 711, 720, 721, 722, 723, 724, 725, and 726 are operation commands for changing the pen color. The pen color is the color of handwritten data input with the pen used by the user. The operation command names of the operation command definition data 709, 710, 711, 720, 721, 722, 723, 724, 725, and 726 are "black pen", "red pen", "blue pen", "green pen", "magenta pen", "cyan pen", "thin black pen", "thin red pen", "thin blue pen", and "thin green pen". For example, in the case of "black pen" and "thin black pen", the character strings that partially match the character string candidates are "kuro" and "pen", and if the user writes "kuro", only "black pen" and "thin black pen" are displayed as operation command candidates. On the other hand, "pen" is also a string that partially matches the string candidates such as "red pen", so if the user writes "pen", the operation command candidates displayed are "black pen" through "cyan pen" and "thin black pen" through "thin green pen". When these operation commands are executed, as shown in Fig. 15, the control data associated with the pen ID of the pen 2500 used by the user is updated, and the pen color of this pen ID is set to ColorId of the pen ID control data (for example, Black in "Command="ChangePen Black"" becomes ColorId).

[0102] In the operation command definition data 719, the operation command name is "Read file", the character strings (String) that partially match the character string candidates are "File", "Read", and "Read", and the operation command to be executed is "ReadFile https: / / %username%:%password%@server.com / files / ". When the handwriting input display control unit 23 replaces this with the system definition data in the same way as the operation command definition data 702, the operation command becomes "ReadFile https: / / taro.tokkyo:x2PDHTyS@server.com / files / ", which indicates that the file at this address (folder) is to be read (ReadFile). When this operation command is executed, the display device 2 displays a file selection window.

[0103] The operation command definition data 712 is an operation command for aligning the direction of text data in a certain direction. The operation command name of the operation command definition data 712 is "Align text direction", the character string that partially matches the character string candidate is "Text", "Orientation" or "Direction", and the operation command character string is "AlignTextDirection". Text data written by the user in a direction other than up-down is in various orientations, making it difficult to read the entire data from a single direction. When the user executes the operation command definition data 712, the display device 2 aligns the handwritten and recognized character strings in the same direction (for example, up-down). In this case, aligning refers to rotating the text data by the angle information.

[0104] The operation command definition data 713 indicates that the operation command name is “Register handwritten signature”, the character strings that partially match the character string candidate are “Sign” and “Register”, and the operation command character string is “RegistSignature”. When the RegistSignature command is executed, a handwritten signature registration form is added to the handwritten input storage unit 25, and the handwritten signature registration form that allows the user to register a handwritten signature is displayed on the operation screen 101.

[0105] The operation command definition data 714 indicates that the operation command name is "Sign in by handwriting", the character string located in the character string candidate portion is "%signature%", and the operation command is "Signin". Here, "%signature%" is a reserved word in the system definition data, and indicates the fact that the registered handwritten signature data matches the stroke data such as the user name. In other words, when there is a match, the operation command 512 based on the operation command definition data 714 is displayed in the operation guide 500 (see FIG. 22).

[0106] When the Signin command is executed, the AccountId of the user having the matching SignatureId of the handwritten signature data is stored in the pen ID control data of the pen 2500 that handwritten the stroke data such as the user name. This associates the pen ID with the AccountId, and the display device 2 can use the user-defined data specified by this AccountId (see FIG. 15(a)).

[0107] The operation command definition data 715 indicates that the operation command name is "handwritten sign out", the character strings that partially match the character string candidates are "sign" or "out", and the operation command is "Signout". When the Signout command is executed, the AccountId is deleted from the pen ID control data of the pen 2500 that operated the handwritten sign out. This removes the association between the pen ID and the AccountId, and any user can use the pen 2500.

[0108] Operation command definition data 716 indicates that the operation command name is “Change settings”, the character string that partially matches the character string candidate is “Settings” or “Change”, and the operation command is “ConfigSettings”. When the ConfigSettings command is executed, a user-defined data change form is added to handwritten input storage unit 25, and the user-defined data change form is displayed on operation screen 101.

[0109] Although the operation command definition data for when handwritten data is present is omitted, when handwritten data is selected, the operation command for when handwritten data is present is displayed (see Fig. 20). In this case, the operation command for when handwritten data is present is displayed by overlapping the circumscribed rectangle of the handwritten data already displayed and the stroke handwritten by the user.

[0110] <User defined data> Next, the user-defined data will be described with reference to Fig. 12. Fig. 12 shows an example of user-defined data held by operation command definition unit 33. The user-defined data in Fig. 12 is an example of definition data for one user. In user-defined data 717, AccountId is user identification information that is automatically numbered for each user, AccountUsername and AccountPassword are a user name and password, SignatureId is handwritten signature data identification information that is automatically numbered when handwritten signature data is registered, and username, password, and machinename are character strings that are set in operation command definition data 701 to 703 instead of system definition data 704 to 706, respectively. This makes it possible to execute operation commands using the user-defined data.

[0111] When a user signs in by handwriting a user name or the like, the pen ID and AccountId are associated with the pen ID control data (see FIG. 15(a)), and by utilizing this, a character string in the user-defined data having an AccountId associated with the pen ID of the pen 2500 used by the user is used when an operation command is executed. After the user signs out, the character string in the system-defined data is used when an operation command is executed, even if the pen 2500 used for signing in is used.

[0112] User-defined data 718 is data used in the user-defined data change form. Name is the item name of AccountUsername, AccountPassword, username, password, or machinename in user-defined data 717, and Data is the changed value of AccountUsername, AccountPassword, username, password, or machinename. In this example, the data for "name" is "%AccountName%, "the data for "password" is "%AccountPassword%, "the data for "folder user name" is "%username%, "the data for "folder password" is "%password%, "the data for "folder file name" is "%machinename%", which correspond to each item of user-defined data 717. These items entered in the user-defined data change form are reflected in user-defined data 717.

[0113] It is also possible for the user to register color definition data, which will be described later, in user-defined data 717, and the user can input color definition data that he or she has defined.

[0114] <Handwritten signature data> Next, the handwritten signature data will be described with reference to Fig. 13. Fig. 13 shows an example of handwritten signature data stored in the handwritten signature data storage unit 39. The handwritten signature data has Data representing a handwritten signature associated with SignatureId. SignatureId is identification information that is automatically assigned when the handwritten signature data is registered, and Data is data calculated by the handwritten signature authentication algorithm of the handwritten signature authentication control unit 38 from the stroke data received from the handwritten signature authentication control unit 38.

[0115] <Handwritten input data stored by the handwritten input storage unit> Next, the handwritten input saved data will be described with reference to Fig. 14. Fig. 14 shows an example of handwritten input saved data saved in the handwritten input saving unit 25. One line in Fig. 14 represents one stroke, one sentence (text), or one image (image).

[0116] When Type is Stroke, one handwritten input saved data has the following items: DataId, Type, PenId, ColorId, Angle, StartPoint, StartTime, EndPoint, EndTime, Point, and Pressure. DataId is stroke identification information. Type is the type of handwritten input saved data. Types include Stroke, Group, Text, and Image. The type of handwritten input saved data 801 and 802 is Stroke. The type of handwritten input saved data 803 is Group. The type of handwritten input saved data 804, 805, and 806 is Text. The type of handwritten input saved data 807 is Image.

[0117] Group means to group other strokes, and handwriting input saved data of type Group specifies the stroke to be grouped by DataId. PenId, ColorId, and Angle are transcriptions of the pen ID control data described below. StartPoint is the start coordinate of the stroke, and StartTime is the start time of the stroke. EndPoint is the end coordinate of the stroke, and EndTime is the end time of the stroke. Point is the coordinate string from the start point to the end point, and Pressure is the pen pressure from the start point to the end point. As shown in Angle, handwriting input saved data 804 and 805 were rotated clockwise by 180 degrees and 270 degrees, respectively, before being recognized as handwriting.

[0118] Handwritten input saved data 804, 805, and 806, whose Type is Text, have FontName, FontSize, and Text, where FontName is the font name of the text, FontSize is the character size, and Text is the content of the text (character code).

[0119] If there is no FontName in the pen ID control data, the handwriting input saving unit 25 copies the FontStyle of the font data 405 of the defined control data shown in Fig. 7 to the handwriting input saved data. The FontSize is copied from the font data 405 of the defined control data in Fig. 7.

[0120] The handwritten input save data 801-805 may be attached as metadata as it is when the file is saved as handwritten input origin data. Therefore, the handwritten input save data 801-805 can be acquired by the display device 2 when the file is read. When the display device 2 transmits the handwritten input save data to an external device, the handwritten input save data is converted to color for a color compatible device, and may be transmitted without converting to color or may be converted to color for a black and white compatible device. When transmitting to a black and white emphasis compatible device, the handwritten input save data is transmitted. In this way, the handwritten input save data 801-805 supports both black and white emphasis and color conversion.

[0121] While handwritten input saved data 801 to 805 are handwritten input origin data, handwritten input saved data 806 and 807 are not. These are files loaded by a file load command. This is determined by whether or not a color value defined by color definition data (described later) is registered in ColorId. For example, handwritten input saved data 801 has a ColorId of "Black," while handwritten input saved data 806 has a ColorId of "#e6001200." This is expressed as # and eight hexadecimal digits, with every two digits representing the values ​​of R (red), G (green), B (blue), and A (transparency).

[0122] In this way, in text data entered by handwriting, the ColorId contains a character string that indicates color information, but in text data not entered by handwriting, the ColorId is "#color value." In this way, whether or not the text data is entered by handwriting is determined by focusing on the ColorId.

[0123] The handwritten input storage unit 25 stores the color of the text data of the file read by the file transmission / reception control unit 37 as ColorId="#e6001200". Therefore, the handwritten input storage data 806 does not support black and white emphasis, but only supports color conversion.

[0124] The handwriting input display control unit 23 determines whether the data is handwriting input-derived data by ColorId, and if there is no handwriting input-derived data, calculates and displays a grayscale value from RGBA. If the display device 2 is a color-compatible device, it displays the RGBA as is.

[0125] The handwritten input saved data 807 is also an image file read from a storage medium by a file read command. If Type is Image, the handwritten input saved data 807 has a FileId and a FileName. FileId is a management number within the handwritten input saved data, and FileName is the original file name.

[0126] In Fig. 14, the value of ColorId indicates whether data is handwriting input-derived data or not, but a dedicated flag or the like may indicate whether data is handwriting input-derived data or not. Also, although Type is limited to stroke handwriting data, it is also possible to determine whether data is handwriting input-derived data from its shape using machine learning. In this case, the learning device learns the correspondence between teacher data for whether data is handwriting input-derived data or not and character shapes using deep learning or the like, and outputs whether the shape of the input character is handwriting input-derived data or not.

[0127] In addition, data derived from handwritten input may include not only text data converted from handwritten data through character recognition, but also data converted based on user operations, such as fixed characters such as "Done" and "Confidential," stamps displayed as marks, shapes such as circles and stars, and straight lines.

[0128] The handwriting input originated data is used in step S7 in FIG. 32 (pen coordinates and reception time), step S88 in FIG. 39 (obtaining handwriting input saved data), step S96 in FIG. 39 (saving a file), and so on.

[0129] <Pen ID control data stored in the pen ID control data storage unit> Next, the pen ID control data will be described with reference to Fig. 15. Fig. 15 is a diagram for explaining the pen ID control data 901 to 904 stored in the pen ID control data storage unit 36. The pen ID control data controls the color of the handwriting input originated data, etc. Fig. 15 shows that there are four pens 2500. Four is just an example, and one or more may be used.

[0130] One line in Fig. 15(a) shows the pen ID control data for one pen. Also, Fig. 15(b) is a diagram explaining angle information when a user writes handwriting on the display device 2. The angle information corresponds to the operation position, and can be said to be the angle of the direction in which the user is present, the angle of the direction in which the pen is used, or the angle related to the rotation of characters handwritten by the user. With a specific direction of the display device 2 (e.g., the vertical direction) as 0 degrees (reference), the angle information for each user is 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees, and 315 degrees counterclockwise.

[0131] Note that the angle information is not automatically determined by the user's operation position, but is set by each user. There are two methods for setting the angle: inputting a straight line, which will be described later, and selecting from an operation menu. The resolution of angle information that can be specified by inputting a straight line (45 degrees in FIG. 15) is merely an example, and a smaller resolution of 5 to 30 degrees, for example, may also be used. However, it is believed that users will be able to read characters rotated at about 45 degrees.

[0132] The pen ID control data 901 to 904 can have a PenId, ColorId, Angle, FontName (font designation), handwriting recognition candidate off state (RecommendMenuOff), and AccountId. Items other than PenId are optional. AccountId is associated with the pen used by the user to sign in. PenId is identification information stored inside the pen 2500. ColorId indicates which color has been selected by the user using this pen 2500 (can be changed by the user at will). ColorId is set with the color ID of the color definition data described later. The details of each color (thickness, etc.) are defined in the color definition data.

[0133] Angle is stroke angle information set for this pen 2500 (can be changed as desired by the user). In the example of FIG. 15(a), the angle information for each pen is 0 degrees, 90 degrees, 180 degrees, and 270 degrees counterclockwise. AccountId is user identification information. By associating AccountId with Pen ID, it is possible to identify the AccountId associated with the Pen ID of the pen 2500 used by the user, and to execute an operation command using user-defined data.

[0134] FontName is added when the user presses the pen color selection button in Fig. 18. When FontName is added, when obtaining the character string data font in step S81 in Fig. 38, FontName is obtained instead of FontStyle that is included in font data 405 of the defined control data shown in Fig. 7. The character string data is displayed in the font of this FontName.

[0135] Pen ID control data 901 is pen ID control data with a PenId of 1, with color black, angle information of 0 degrees, AccountId=1, no FontName, and no RecommendMenuOff. The user with AccountId=1 is the user of user-defined data 717 in FIG. 12. This indicates that this user signed in by handwriting a user name and the like with the pen with PenId=1, and selected black. Pen ID control data with no AccountId indicates a signed-out state (not associated with a user). Similarly, pen ID control data 902 is pen ID control data with a PenId of 2. The color is red. The angle is 90 degrees. There is no AccountId. FontName is pop. There is no RecommendMenuOff.

[0136] The handwriting recognition candidate off state (RecommendMenuOff) is set in the pen ID control data 903. The handwriting recognition candidate off state (RecommendMenuOff) is control data for controlling the operation guide 500 not to display the character string candidate 539 (displaying only the operation command) or not to display even the operation command candidate 510. In the latter case, the operation guide 500 itself is not displayed. The handwriting recognition candidate off state (RecommendMenuOff) is set by pressing a handwriting recognition candidate on / off button (see FIG. 24) described later. When a user inputs handwritten data that does not require character recognition such as a figure, some users may find the operation guide 500 to be a nuisance. In this embodiment, since the handwriting recognition candidate can be controlled to the off state, the operation guide 500 can be hidden and the processing load can be reduced when the display device 2 does not recognize characters. The pen ID control data 903 has a PenID of 3. The color is blue. The angle is 180 degrees. There is no AccountId. There is no FontName.

[0137] The pen ID control data 904 has a PenId of 4, a color of Green, an angle of 270 degrees, no AccountId, no FontName, and no RecommendMenuOff.

[0138] If RecommendMenuOff="True" is added to the pen ID control data, adopting control that does not display operation commands will result in the operation commands not being executed either, so in order to cancel RecommendMenuOff="True", a button operation such as that shown in Figure 24 is required.

[0139] Furthermore, the pen ID control data of the pen 2500 when the user presses the pen color selection button in FIG. 18 or the button in FIG. 24 is updated with the ColorId, Angle, FontName, and RecommendMenuOff defined for that button.

[0140] The pen ID control data is used in step S5 of Figure 32 (obtaining pen ID control data), step S20 of Figure 34 (saving angle information of pen ID control data), step S21 of Figure 34 (obtaining angle information of pen ID control data), and step S60 of Figure 36 (obtaining pen ID control data).

[0141] <Color definition data> Figure 16 shows an example of color definition data. One line of color definition data defines one color. The color definition data defines the black and white highlighting display on a black and white compatible device for ColorId (how to display the black and white pattern, width (Pattern), and edged (Edged)) and the color conversion method (how to display color information (Color) and width (Width) on a color compatible device). Color information is represented by # and eight hexadecimal digits. Every two digits of color information represent #R (red), G (green), B (blue), A (transparency), and px represents the pixel width. Note that color definition data is only applied to data resulting from handwritten input.

[0142] Color definition data 1001 is an example of a line type or color definition with ColorId "Black". A monochrome emphasis compatible device that cannot display color represents the color by the line type. Pattern represents the contents of the edge of the stroke or text. Edged represents the presence or absence of a border. Color represents RGBA color information. Width represents the line width. Color definition data 1001 indicates that handwritten input saved data is displayed as solid black 5 pixels wide + no border on a monochrome emphasis compatible device, and is displayed as black with 0% transparency + 5 pixels wide on a color compatible device. Similarly, color definition data 1002 is an example of a line type or color definition with ColorId "Red". Color definition data 1002 indicates that handwritten input saved data is displayed as solid black 3 pixels wide + black border 1 pixel wide from the outside + white border 2 pixels wide on a monochrome emphasis compatible device, and is displayed as red with 0% transparency of the color chart + 5 pixels wide on a color compatible device. Color definition data 1003 is a definition example of a line type or color with ColorId "Blue", which indicates that in a black and white emphasis compatible device, the data is displayed with a solid white 4 pixel width + black border 1 pixel width, and in a color compatible device, the data is displayed with 0% blue transparency of the color chart + 5 pixel width. Color definition data 1004 is a definition example of a line type or color with ColorId "Green". The color definition data 1004 indicates that in a black and white emphasis compatible device, the data is displayed with a black dot pattern 4 pixel width + black border 1 pixel width, and in a color compatible device, the data is displayed with 0% green transparency of the color chart + 5 pixel width. Color definition data 1005 is a definition example of a line type or color with ColorId "Magenta". The color definition data 1005 indicates that in a black and white emphasis compatible device, the data is displayed with a black upper right diagonal line pattern with a transparent white 5 pixel width + borderless, and in a color compatible device, the data is displayed with 60% magenta transparency of the color chart. Color definition data 1006 is an example of a line type or color definition for ColorId "Cyan." The color definition data 1006 indicates that a black horizontal line pattern is displayed with white transparent at 5 pixel width on a monochrome emphasis compatible device, and that a color compatible device displays it with 60% transparency of cyan on the color chart.

[0143] Color definition data 1007 defines a line type or color with ColorId "ThinBlack". The color definition data 1007 indicates that a black and white emphasis compatible device displays solid black 3 pixels wide + no border, and a color compatible device displays black with 0% transparency + 3 pixels wide. Color definition data 1008 defines a line type or color with ColorId "ThinRed". The color definition data 1008 indicates that a black and white emphasis compatible device displays solid black 1 pixel wide + black border 1 pixel wide from the outside + white border 1 pixel wide, and a black and white emphasis compatible device displays red with 0% transparency of the color chart + 3 pixels wide. Color definition data 1009 defines a line type or color with ColorId "ThinkBlue". The color definition data 1009 indicates that a black and white emphasis compatible device displays solid white 2 pixels wide + black border 1 pixel wide, and a color compatible device displays blue with 0% transparency of the color chart + 3 pixels wide. The color definition data 1010 defines a line type or color with ColorId "ThinGreen." The color definition data 1010 indicates that a black dot pattern with a width of 2 pixels plus a black border of 1 pixel is displayed on a monochrome emphasis compatible device, and that a color compatible device is displayed with a transparency of 0% for green on the color chart plus a width of 3 pixels.

[0144] Thus, the color definition data includes black and white emphasis and color conversion possible data. The color definition data is held by the handwritten input storage unit 25 and is used to convert the handwritten input storage data.

[0145] By defining color definition data as system-defined data and user-defined data and adding a form for changing color definition data to the user-defined data, users can change the color definition to their personal preference after signing in.

[0146] <Pen color selection button definition data> Next, a method for a user to select a pen color will be described with reference to Figures 17 and 18. Figure 17 is an example of pen color selection button definition data. The pen color selection button definition data is data that associates a pen button displayed on the display 220 with a ColorId. The pen color selection button definition data has a pen button ID (PenButtonId), an icon file (Icon), a color ID (ColorId), and a font name (FontName).

[0147] Fig. 18 is an example of the pen color selection buttons 81-86 displayed on the display. The pen color selection buttons 81-86 are displayed using the icon file described above. The pen color selection buttons 81-86 are displayed in order from left to right in ascending order of pen button ID. When the user presses a pen color selection button 81-86 with the pen 2500, ColorId and FontName are added to the pen ID control data, and handwritten data or character string data input with the pen 2500 thereafter is displayed using ColorId and FontName.

[0148] Pen color selection button definition data 1101 is definition data for the pen color selection button 81 displayed at the far left. When the user presses the pen color selection button 81 with the pen 2500, the stroke data handwritten with the pen 2500 will have a ColorId of "Black", and the font of the text after handwriting recognition will be Mincho. Pen color selection button definition data 1102 is for the pen color selection button 82 displayed second from the left, the ColorId of the stroke data will be "Red", and the font of the text after handwriting recognition will be Pop. Similarly, pen color selection button definition data 1103 to 1106 define the button display position, ColorId, and text font of the pen color selection buttons 83 to 86.

[0149] Fig. 18(a) shows an example of a text font display highlighted in black and white based on the pen color selection button definition data. The text font display example is for illustrative purposes only, but may be displayed when the user hovers the pen 2500. Fig. 18(b) shows an example of a text font display displayed in color on a color-capable device based on the pen color selection button definition data (actually it is in color, but is displayed in black and white for drawing purposes).

[0150] As shown in FIG. 18(a), by changing the font in addition to the black and white emphasis, it is possible to achieve more effective black and white emphasis display.

[0151] If the pen color selection button definition data is also defined in the system-defined data and the user-defined data, and a form for changing the pen color selection button definition data is added to the user-defined data, the pen color selection button can be changed to one of personal preference after handwritten signature authentication. Therefore, the correspondence between ColorId and font is merely one example.

[0152] The pen color selection button is held by the handwriting input display control unit 23, and is used in step S6 in FIG. 32 (coordinate complement display data), step S82 in FIG. 38 (character string data display data), and step S100 in FIG. 39 (display data).

[0153] <Example of selectable candidates> FIG. 19 shows an example of an operation guide and selectable candidates 530 displayed by the operation guide. When the user handwrites handwritten data 504 (due to a timeout of the selectable candidate display timer), the operation guide 500 is displayed. The operation guide 500 has an operation header 520, operation command candidates 510, handwriting recognition character string candidates 506, conversion character string candidates 507, character string / predictive conversion candidates 508, and handwritten data rectangular area display 503. The selectable candidates 530 are operation command candidates 510, handwriting recognition character string candidates 506, conversion character string candidates 507, and character string / predictive conversion candidates 508. In this example, "No language conversion character string" may be displayed. The selectable candidates 530 excluding the operation command candidates 510 are called character string candidates 539.

[0154] Operation header 520 has buttons 501, 509, 502, and 505. Button 501 accepts a switching operation between predictive conversion and "kana conversion." In the example of Fig. 19, when the user presses button 509 displaying "predictive," handwriting input unit 21 accepts it and notifies handwriting input display control unit 23, and display unit 22 changes the display to button 509 displaying "kana." After handwritten data 504 is recognized, character string candidates 539 are arranged in descending order of the probability of "kana conversion."

[0155] Button 502 operates the page of the candidate display. In the example of FIG. 19, there are three candidate display pages, and the first page is currently displayed. Button 505 accepts the deletion of operation guide 500. When the user presses button 505, handwriting input unit 21 accepts it and notifies handwriting input display control unit 23 of that fact, and display unit 22 erases the display except for the handwritten data. Button 509 accepts the deletion of all displays. When the user presses button 509, handwriting input unit 21 accepts it. Handwriting input unit 21 notifies handwriting input display control unit 23 of that fact. Display unit 22 erases all displays shown in FIG. 19, including the handwritten data, allowing the user to start over with handwriting.

[0156] Handwritten data 504 is the character "gi" handwritten by the user. Handwritten data rectangular area display 503 is displayed surrounding handwritten data 504. The display procedure will be described with reference to the sequence diagrams of Figures 32 to 39. In the example of Figure 19, handwritten data rectangular area display 503 is displayed in a dotted line frame.

[0157] The character string candidates are arranged in descending order of probability in handwriting recognition character string candidates 506, conversion character string candidates 507, and character string / predictive conversion candidates 508. "Gi" in the handwriting recognition character string candidate 506 is a candidate for the recognition result. In this example, the display device 2 correctly recognizes "Gi".

[0158] Conversion string candidate 507 is a conversion string candidate converted from a language string candidate. In this example, "Technology Test" is an abbreviation for "Technology Mass Production Trial." String / predictive conversion candidate 508 is a predicted string candidate converted from a language string candidate or a conversion string candidate. In this example, "Approve the Skill Test" and "Destination for sending the minutes" are displayed.

[0159] Operation command candidate 510 is a candidate for an operation command selected based on operation command definition data 701-703, 709-716 in Fig. 11(a). In the example of Fig. 19, the first character of the line "》" 511 is shown to be a candidate for an operation command. In Fig. 19, there is no selection data to be selected by "gi" which is handwritten data 504, and "meiji", which is a candidate character string for "gi", partially matches operation command definition data 701, 702 in Fig. 11(a), and is therefore displayed as operation command candidate 510.

[0160] When the user selects "Load minutes template", the operation command defined in operation command definition data 701 is executed, and when the user selects "Save in minutes folder", the operation command defined in operation command definition data 702 is executed. In this way, operation command candidates are displayed when operation command definition data containing the converted character string is found, so they are not always displayed.

[0161] As shown in FIG. 19, character string candidates and operation command candidates are displayed simultaneously (together), so the user can arbitrarily select either the character string candidate or the operation command that he or she intended to input.

[0162] <Example of selection data> In the display device 2 of this embodiment, the user can specify selected data by handwriting the finalized data. The selected data is the subject of editing or modification, or the page name.

[0163] <Example of display of operation command candidates> Fig. 20 shows an example of displaying operation command candidates based on operation command definition data when handwritten data is present. Fig. 20(a) shows candidates for edit-related operation commands, and Fig. 20(b) shows candidates for modification-related operation commands. Fig. 20(a) also shows an example in which selection data is specified by handwritten data 11a.

[0164] As shown in Figures 20(a) and (b), the operation command candidates displayed following the first character "》" 511 are a main menu 550. The name of the last executed operation command or the first operation command name in the operation command definition data is displayed in the main menu 550. The first character "》" 511a on the first line is a candidate for an edit-related operation command, and the first character "》" 511b on the second line is a candidate for a modification-related operation command. The first character "》" 511c on the third line is a candidate for the operation command "Set as page name."

[0165] The ">" 512 at the end of the line indicates that there is a submenu (an example of a submenu button). The ">" 512a on the first line displays a submenu (the last selected one) as a candidate for edit-related operation commands. The ">" 512b on the second line displays the remaining submenus as candidates for modifier-related operation commands. When the user presses the ">" 512, a submenu 560 is displayed to the right of it. All operation commands defined in the operation command definition data are displayed in the submenu 560. In the display example of FIG. 20(a), the submenu 560 corresponding to the ">" 512a on the first line is also displayed from the time the main menu is displayed. The display device 2 may display a submenu by pressing the ">" 512a on the first line.

[0166] Regarding the operation command for setting the page name on the third line, the operation command candidates are not grouped, so ">" is not displayed at the end of the line.

[0167] When the user presses any of the operation command names with the pen, the handwriting input display control unit 23 executes the Command of the operation command definition data associated with the operation command name on the selected data. That is, when "Erase" 521 is selected, "Delete" is executed, when "Move" 522 is selected, "Move" is executed, when "Rotate" 523 is selected, "Rotate" is executed, and when "Select" 524 is selected, "Select" is executed.

[0168] For example, if the user presses "Delete" 521 with the pen, the "Minutes" can be erased. If the user presses "Move" 522, "Rotate" 523, or "Select" 524, a bounding box (a rectangle circumscribing the selected data) is displayed. When "Move" 522 and "Rotate" 523 are pressed, the user can move or rotate "Minutes" by dragging the pen. When "Select" 524 is pressed, the user can perform other bounding box operations.

[0169] 20, character string candidates other than the operation command candidates, "○" 551, "∞" 552, "0" 553, and "00" 554, are the recognition results of the encircling line (handwritten data 11b). If the user intends to input a character string instead of an operation command, the character string candidate can be selected.

[0170] In FIG. 20(b), pressing ">" 512b on the second line displays the submenu of FIG. 20(b). The display example of FIG. 20(b) also displays a main menu 550 and a submenu 560, similar to FIG. 20(a). When "Thick" 531 is selected, the handwriting input display control unit 23 executes "Thick" on the selected data. When "Thin" 532 is selected, the handwriting input display control unit 23 executes "Thin" on the selected data. When "Large" 533 is selected, the handwriting input display control unit 23 executes "Large" on the selected data. When "Small" 534 is selected, the handwriting input display control unit 23 executes "Small" on the selected data. When underline 535 is selected, the handwriting input display control unit 23 executes "Underline" on the selected data.

[0171] <Example of angle information input> Next, a method for inputting angle information will be described with reference to Fig. 21. Fig. 21 is an example of a diagram for explaining a method for inputting angle information. Fig. 21 illustrates a case where a user positioned at the 3 o'clock position of the display device 2 inputs angle information. Characters handwritten from the 3 o'clock position are correctly recognized when rotated 90 degrees clockwise, so it is preferable to input angle information of 90 degrees.

[0172] 21(a) shows a state in which the angle information of the pen ID control data is 0 degrees (initial value) and the operation guide 500 is displayed because a user standing at the 3 o'clock position of the display device 2 has handwritten "ぎ". Because the display device 2 performs character recognition of "ぎ" handwritten from the 3 o'clock position with the angle information remaining at 0 degrees, an unexpected selectable candidate 530 is displayed.

[0173] When inputting angle information, the user hand-draws a straight line in the operation guide 500 from top to bottom as seen by the user. Figure 21(b) shows an example of this straight line 521. The angle information is the angle α formed by the straight line 521 in the counterclockwise direction and the 6 o'clock direction, which has angle information of 0 degrees. In other words, the angle α formed by the straight line 522 drawn from the starting point S in the 6 o'clock direction and the straight line 521 input by the user in the counterclockwise direction is the angle information. Simply put, the end point direction of the straight line 521 is the angle information. Therefore, the angle information input by the user in Figure 21(b) is 90 degrees.

[0174] For detecting a straight line, for example, the coordinates from the start point S to the end point E are converted into a straight line by the least squares method, and the obtained correlation coefficient is compared with a threshold value to determine whether or not it is a straight line.

[0175] Immediately after the user starts writing the straight line 521 (immediately after the pen 2500 touches the starting point S of the straight line 521), the display device 2 erases the operation guide 500. Also, immediately after finishing writing the straight line 521 (immediately after the pen 2500 leaves the end point E of the straight line 521), the display device 2 searches for the value closest to the included angle α from 45 degrees, 90 degrees, 135 degrees, 180 degrees, 215 degrees, 270 degrees, 315 degrees, 360 degrees and determines it as the angle information. The included angle α itself may also be the angle information. The determined angle information is set in the Angle of the pen ID control data. Since the pen event transmission unit 41 of the pen 2500 transmits the pen ID to the display device 2 when the pen tip is pressed for handwriting or the like, the display device 2 can associate the angle information with the pen ID control data.

[0176] Note that the user can input angle information by handwriting a straight line only within the operation guide 500. As a result, when the user handwrites a straight line outside the operation guide 500, it is recognized as "1" or "一", and when the user handwrites a straight line within the operation guide 500, angle information can be input. That is, the handwriting recognition control unit 26 detects a straight line from a predetermined range and converts the stroke data handwritten outside the predetermined range into text data.

[0177] FIG. 21(c) shows the operation guide 500 immediately after the operation in FIG. 21(b). Since 90 degrees is set as the angle information (Angle) in the pen ID control data, the handwriting data (stroke data) is rotated 90 degrees clockwise internally for handwriting recognition, and the operation guide 500 is rotated 90 degrees counterclockwise and displayed. Note that the angle information may be manually input by the user from the menu.

[0178] <An example of sign-in by handwriting> Next, a method for a user to sign in after registering handwritten signature data will be described with reference to Fig. 22. Fig. 22 shows an example of operation guide 500 that is displayed when the user handwrites "Suzuki", which is registered handwritten signature data. Since "Suzuki" is registered as handwritten signature data in operation command definition section 33, "Suzuki" matches the handwritten signature data. Therefore, operation command 512 "Sign in by handwriting" is displayed.

[0179] In addition, since the handwritten signature data matches, the SignatureId representing "Suzuki" is identified, and the user-defined data having the AccountId associated with this SignatureId is identified.

[0180] When the user selects the operation command 512 "Sign in by handwriting," the AccountId of "Suzuki" is associated with the pen ID of the pen 2500 being used and added to the pen ID control data, and the user-defined data of "Suzuki" becomes available for use when the operation command is used.

[0181] <Example of operation menu display> Next, an operation menu having information regarding data processing according to the display position will be described with reference to Fig. 23A. Fig. 23A is a top view of the display device 2 placed flat. In Fig. 23A, users 250, 251 are present at the top and bottom of the drawing, respectively, and two users can simultaneously operate operation menus 301, 302 from the two directions, the bottom and the top. However, there is no problem if there is only one user. The display device 2 displays the operation menu according to the operation button definition data. The number of operation menus will be the same as the number of expected users.

[0182] Each of the operation menus 301 and 302 has information about data processing corresponding to the display position of the operation menu. In this embodiment, the operation menus 301 and 302 have a rotation angle corresponding to the display position of the operation menu. The design (outline of the icons) of the operation menus 301 and 302 is the same, but each has information about data processing corresponding to the display position of the operation menu.

[0183] Each operation menu 301, 302 has a pen operation panel 2001, 2008 that allows the user to select a pen type and perform pen operations. Each operation menu 301, 302 has a page operation panel 2002, 2009 that allows the user to perform page operations. Each operation menu 301, 302 has a page navigation operation panel 2003, 2010 that allows the user to operate a list display of file names or pages. Each operation menu 301, 302 has a page navigation 2004, 2011 that switches pages from a list display of file names or page names. For ease of use by the user, the operation menu 301 for the lower user is positioned near the bottom of the screen (along the bottom edge of the display 220), and the operation menu 301 for the upper user is positioned near the top of the screen (along the top edge of the display 220).

[0184] The user operation menu 301 on the lower side has a pen operation panel 2001, a page operation panel 2002, a page navigation operation panel 2003, and a page navigation 2004. The page navigation 2004 extends from top to bottom in a page navigation window area 2005. The page navigation window area 2005 may be simply called a page navigation window.

[0185] Similarly, the operation menu 302 for the upper user has a pen operation panel 2008, a page operation panel 2009, a page navigation operation panel 2010, and a page navigation 2011. The page navigation 2011 extends downward when the page navigation window area 2012 is viewed from the upper user.

[0186] Handwritten data and character string data are displayed near the center of the display 220. Handwritten data 2006 was written by the user at the bottom with a black pen (solid black). Character string data 2007 is a character string that was written by the user at the bottom with a black pen and recognized. Handwritten data 2013 is handwritten data that was written by the user at the top with a blue pen (solid white with black border).

[0187] Pen operation panels 2001, 2008, page operation panels 2002, 2009, page navigation operation panels 2003, 2010, and page navigations 2004, 2011 are included in the operation menu 302, with one for the lower user and the other for the upper user. Icons (buttons), file names, and page names for the upper user are displayed rotated 180 degrees. This angle of rotation is preset in the operation button definition data (described later).

[0188] When the user presses either the button area of ​​the operation menus 301 and 302, or the file name or page name of the page navigation 2004 and 2011 with the pen 2500, the display device 2 stores the rotation angle set in the operation button definition data in the pen ID control data of the pen 2500. This rotation angle becomes information related to data processing.

[0189] <<English translation of Figure 23A>> The display device 2 also supports languages ​​other than Japanese. Here, as an example, FIG. 23A will be described for the case where the display device 2 is used in an English environment. The configuration of the display device 2 is the same as when it is used in Japanese, except that the conversion dictionary and various definition data support English.

[0190] Fig. 23B is a top view of the display device 2 laid flat when the display device 2 converts handwritten data into English. When handwritten data is converted into English, the display device 2 also displays operation menus 301, 302. In Fig. 23B, users 250, 251 are present at the top and bottom of the drawing, respectively, and the two users can simultaneously operate operation menus 301, 302 from the top and bottom. The display device 2 displays the operation menus in accordance with the operation button definition data. The number of operation menus is the same as the number of expected users.

[0191] In the operation menus 301 and 302, the pen operation panels 2001 and 2008, the page operation panels 2002 and 2009, the page navigation operation panels 2003 and 2010, and the page navigations 2004 and 2011 are written in English.

[0192] The content handwritten on display 3 depends on what the user writes by hand, but in Fig. 23B it is handwritten in English. English handwritten data and character string data are displayed near the center of display 220. Handwritten data 2006 was written by the user on the lower side with a black pen (solid black). Character string data 2007 is a character string that was written by the user on the lower side with a black pen and recognized. Handwritten data 2013 is handwritten data written by the user on the upper side with a blue pen (solid white with black border).

[0193] Although the present embodiment has been described taking English as an example, the display device 2 can also support languages ​​other than English (Chinese, Hindi, Spanish, Arabic, Portuguese, Russian, etc.).

[0194] <<Pen operation panel>> 24 is an enlarged view of the pen operation panels 2001, 2008. The pen operation panels 2001, 2008 have buttons for the user to select colors, etc. Buttons 2101 to 2108 represent thin or thick opaque pens in black, red, blue, and green, while buttons 2109 and 2110 represent thick pens in transparent magenta and cyan.

[0195] Buttons 2111 and 2112 represent undo (return to original state) / redo (redo). Button 2113 represents page deletion, and button 2114 represents page sweep (a button for clearing out and redisplaying a degraded screen in high-speed drawing mode). Button 2115 accepts turning on or off the display of character string candidates 539 (a button for accepting the setting of the handwriting recognition candidate off state (RecommendMenuOff) of the pen ID control data).

[0196] As shown in the operation button definition data in Fig. 27 to Fig. 30, for each button, a button identifier (ButtonId), a button display position (Position), an icon image (Icon) of the button, a rotation angle (Angle) of the button, and a command (Command) that the display device 2 executes when the button is pressed are defined. When the user presses a button 2101 to 2115 with the pen 2500, the display device 2 saves the rotation angle defined for that button in the pen ID control data of that pen 2500, and executes the command defined for that button. It is assumed that buttons 2101 to 2115 correspond to ButtonId=1 to 15 in the operation button definition data in order from the left.

[0197] The commands of operation button definition data 2401-2410 corresponding to buttons 2101-2110 specify the ColorId of color definition data in the ChangePen command, which is the same as that of operation command definition data 709-711 and 720-726 in Fig. 11. The display unit 22 displays a frame 260 on the button of the pen last selected so that it can be seen that it has been selected. In Fig. 24, the thin black pen of button 2101 has been selected.

[0198] Operation button definition data 2411, 2412 corresponding to buttons 2111, 2112 define the Undo / Redo commands. Buttons 2111, 2112 (Undo / Redo) each accept the execution of an Undo / Redo command. Undo is a command to return to the previous operation state, and Redo is a command to advance the undone operation state one step forward.

[0199] The operation button definition data 2413 corresponding to the button 2113 defines a command of EragePage. The button 2113 (erase page) accepts the execution of a command of EragePage. EragePage erases all data inputted on the current page and redisplays it.

[0200] The operation button definition data 2414 corresponding to the button 2114 defines a SweepPage command. The button 2114 (Sweep button) accepts execution of the SweepPage command. In a display device such as electronic paper, some of the image may remain undisplayed in high-speed drawing mode. The display device redisplays this image and clears it away.

[0201] The operation button definition data 2415 corresponding to the button 2115 defines the command ToggleRecommendMenu. The button 2115 (handwriting recognition candidate on / off) accepts execution of the command ToggleRecommendMenu. ToggleRecommendMenu switches the display of the character string candidates 539 obtained by handwriting recognition on and off. As a result, when it is turned off, RecommendMenuOff="True" is added to the pen ID control data of the pressed pen 2500. When the display of the character string candidates 539 is turned on, all the character string candidates 539 are displayed, but when it is turned off, only the operation commands are displayed. If there are no operation commands, the entire selectable candidates are not displayed, allowing the user to concentrate on handwriting. When the display device 2 executes ToggleRecommendMenu, the current on / off state is returned, and the icon is switched according to that state.

[0202] It should be noted that buttons 2101 to 2110 are defined as buttons separate from the pen color selection buttons 81 to 86 in FIG. 18, and since the ColorId may be set in duplicate by the pen color selection buttons 81 to 86, in such a case the result of pressing buttons 2101 to 2110 is given priority and set in the pen ID control data.

[0203] <<Page Operation Panel>> FIG. 25 shows an example of the page operation panels 2002 and 2009. The page operation panels 2002 and 2009 are operation menus related to page operations. The button 2201 is a button that allows the user to rotate the page being displayed by 90 degrees at a time. The buttons 2202 and 2204 are buttons that allow the user to switch to the previous page or next page, respectively. The current page 2203 indicates the current page number (numerator) and the total number of pages (denominator). The button 2205 is a button that allows the user to open or close the page navigation window. The file name 2206 indicates the name of the file currently being displayed. The buttons 2201, 2202, 2203, and 2205 are defined in the operation button definition data 2416 to 2419 and 2440 to 2443.

[0204] Button 2201 (Rotate Page Display) accepts execution of the RotatePage command. RotatePage rotates only the handwritten data 2006 and character string data 2007 of the lower user and the handwritten data 2013 of the upper user shown in FIG. 30 by 90 degrees counterclockwise and displays them. The operation menu including the page navigation is not rotated. In other words, in the example of FIG. 30, the handwritten data 2013 of the upper user is upside down and difficult to read when viewed from the lower user, so by executing the RotatePage command, the display device 2 temporarily rotates the page to make it easier to read.

[0205] Buttons 2202 and 2204 (page switching) accept the execution of the commands PageBack and PageNext, which switch the current page to the previous page or the next page.

[0206] A button 2205 (opens or closes the page navigation window) accepts execution of a TogglePageNavi command, which opens or closes the page navigation window areas 2005 and 2012 in FIG.

[0207] Current page 2203 and file name 2206 are only displayed and do not accept any operations. Current page 2203 displays the current page number (numerator) and the total number of pages (denominator). In this example, display device 2 is displaying the second page of a four-page file. File name 2206 displays the name of the file currently being displayed. In this example, display device 2 is displaying "Technical Review Meeting.pdf".

[0208] <<Page navigation operation panel>> 26 shows an example of the page navigation operation panels 2003 and 2010. The page navigation operation panels 2003 and 2010 display the page names for each file in page order. In other words, if a page name is selected from character string data, the user can easily identify what is written on which page later.

[0209] Buttons 2301 and 2305 allow the user to switch to the previous page navigation window or the next page navigation window, respectively. Buttons 2302 and 2304 allow the user to switch to the previous file or the next file, respectively (the first page of the switched file is displayed on the display 220). Current page navigation 2303 displays the current number (numerator) and total number (denominator) of page navigation windows. Buttons 2301, 2302, 2304, and 2305 are defined in operation button definition data 2420-2423 and 2444-2447.

[0210] The page navigation 2306 displays character strings in file order and page order. The page navigation 2306 displays the file names of multiple loaded files and the names of each page in the file in a tree format. When a file name or page name is pressed, the display device 2 displays that page on the display 220 based on the page navigation data in FIG. 38. For example, if the page navigation has 200 lines and the number of lines in the page navigation window is 50 lines, the current number of the current page 2203 can take values ​​from 1 to 4, and the total number is 4. When the user presses button 2301, the display device 2 decrements the current number, and when the user presses button 2305, the display device 2 increments the current number and redisplays the page navigation window. For ease of use by the user, the page navigation 2306 is arranged at the bottom of the display 220 for the lower user and at the top of the display 220 for the upper user.

[0211] In the page navigation 2306 in Fig. 26, two file names 2310, 2308 are displayed, and in each file, page names 2307, 2309 of each page are displayed in a tree structure. The page names are character string data specified by the user or character strings extracted automatically. The page names are displayed indented by the user's operation.

[0212] Buttons 2301 and 2305 (page navigation window switching) accept execution of the commands PageNaviWinBack / PageNaviWinNext. PageNaviWinBack / PageNaviWinNext switch the current page navigation window to the previous or next page navigation window. If the number of lines that can be displayed in the page navigation window is 50, pressing the buttons 2301 and 2305 causes the display device 2 to switch and display page navigation windows of 50 lines.

[0213] Buttons 2302 and 2304 (file switching) accept execution of the commands PageNaviFileBack / PageNaviFileNext, which switch the currently displayed page to the first page of the previous or next file.

[0214] Page navigation 2306 is defined by the page navigation data in Fig. 31. Tree open icon 2311 is a downward triangle and means that the tree is being displayed. Tree closed icon 2312 is a sideways triangle and means that the tree is being closed. When the tree is closed, the page names of the rows below the row where the tree is located are no longer displayed.

[0215] When the Tree in the page navigation data is "Opened", the display device 2 displays an open tree icon 2311, which, when pressed, changes the Tree to "Closed" and closes the tree. Conversely, when Tree="Closed", the display device 2 displays a closed tree icon 2312, which, when pressed by the user, changes the Tree to "Opened" and opens the tree.

[0216] The page name 2313 indicates the indented page. The indent level is controlled by the Level of the page navigation data. The file name and page name have the initial value of Level="0" and Level="1", respectively. When the user presses the file name or page name to indent to the right, the indent level Level value increases. The page name 2313 indicates an indent of Level="2". To change the Level, the user presses and holds the page name and moves it left or right. For example, when the user moves it to the right, the display device 2 searches for the same Level (the Level before the move) above the current line and adds Tree="Opened" to the line above. It is possible to not display the indented line, and the display or non-display of the indented line is controlled by the Tree of the line at the same Level as the line before the indentation. The display device 2 also increments the Level of the indented line and displays it again.

[0217] Page name 2314 indicates that it is a blank page, and indicates the blank page following the last page of the file (page name 2313). This means that nothing has been handwritten on a blank page yet. Blank pages continue up to the maximum page number, but the "blank page" displayed in the page navigation is only the first blank page. For example, if the maximum page number is 100 and there are four pages in the file, pages 5 to 100 are blank pages, but only the first blank page 5 is displayed in the page navigation 2306 as a "blank page."

[0218] A page name 2315 is the page currently being displayed on the display 220. Here, the page name 2315 is highlighted by being surrounded by a frame 262.

[0219] Page name 2316 is the default page name. When there is only handwriting and no handwriting-recognized character string, the display device 2 displays the page name with "nnnn page" (nnnn is an integer) added like this. This shows that page names other than the default page name of page name 2316, the blank page of page name 2314, and the file names 2310 and 2308 have handwriting-recognized character strings.

[0220] In the page navigation data described later, if AutoName="True" is set, the display device 2 will automatically set the handwritten recognized character string as the page name if there is one. In the screen layout example of FIG. 30, the character string "Hardware Configuration" is automatically set as the page name. There are various possible logics for determining the page name from the handwritten recognized character string. For example, there is a method of combining character strings in the order of the time when the character strings were input and extracting a fixed number of characters from the beginning, or a method of determining the priority of the position of the character strings, combining character strings in order of the highest priority, and extracting a fixed number of characters from the beginning.

[0221] As shown in the example of the operation command display in Figure 20, when the user selects confirmed data that has already been handwritten and recognized by enclosing or crossing lines, "Set as page name" is displayed in the operation commands. When this is selected, the SetPageName command is executed. SetPageName changes the current page name in the page navigation 2306 to the selected character string, and changes AutoName in the page navigation data to "False."

[0222] When a file name or page name on the page navigation operation panel 2003, 2010 is selected with the pen 2500, "Angle" is also set in the pen ID control data. This angle is the angle by which the page navigation window area is rotated. The display device 2 saves the angle (Angle) by which the page navigation window area is rotated, which is obtained from the operation button definition data for display purposes, and when a file name or page name is selected with the pen 2500, the stored rotation angle is set as "Angle" in the pen ID control data. Thereafter, data is processed with the angle set in the pen ID control data.

[0223] <Operation button definition data> 27 to 30 show examples of operation button definition data 2401 to 2448. The operation button definition data 2401 to 2448 define the buttons of the pen operation panels 2001 and 2008, the page operation panels 2002 and 2009, and the page navigation operation panels 2003 and 2010. One line of the operation button definition data defines one button. Also, Figs. 27 and 28 define the operation menu 301 for the lower user, and Figs. 29 and 30 define the operation menu 302 for the upper user.

[0224] Each operation button definition data 2401-2448 defines a button identifier (ButtonId), a display position (Position) of the button, an icon image (Icon) of the button, an angle (Angle) which is information related to data processing, and a command (Command) executed by the display device 2 when the button is pressed. The angle (Angle) also represents the user's operation position and the rotation angle of the operation menu 302. When the user presses the buttons 2101-2115, 2201, 2202, 2203, 2205, 2301, 2302, 2304, 2305 with the pen 2500, the display device 2 saves the angle (Angle) defined for that button in the pen ID control data of that pen, and executes the command defined for that button.

[0225] In this way, the operation menu is associated with data processing according to the display position of the operation menu. Figures 27 to 30 show an example in which character string data is rotated according to the display position. If the administrator changes the Command of the same button on the operation menu for lower users 301 and the operation menu for upper users 302, various data processing becomes possible.

[0226] Operation button definition data 2401-2415 define the pen operation panel 2001 for the lower user. Operation button definition data 2416-2419 define the page operation panel 2002 for the lower user. Operation button definition data 2420-2424 define the page navigation operation panel 2003 for the lower user. Operation button definition data 2425-2439 define the pen operation panel 2008 for the upper user. Operation button definition data 2440-2443 define the page operation panel 2009 for the upper user. Operation button definition data 2444-2448 define the page navigation operation panel 2010 for the upper user.

[0227] The operation button definition data 2415, 2439 must indicate the current on / off state of the handwriting recognition candidates, so define an icon image for on (IconOn) and an icon image for off (IconOff). The operation button definition data 2419, 2443 must represent a button for opening or closing the page navigation window, so define an icon image for on (IconOn) and an icon image for off (IconOff). The display device 2 displays the icon image for on / off based on the return value of the execution result of the Command.

[0228] The operation button definition data 2424, 2448 is definition data for the page navigation window area, and defines only the position and angle of the rectangular area. When the user presses these rectangular areas with the pen 2500, the display device 2 stores the defined angle in the pen ID control data of the pen 2500 used for the press.

[0229] <Page navigation data> Figure 31 shows an example of page navigation data. Page navigation data is an example of information in which the page name is recorded for each page of a file. The page navigation data defines the display content of the page navigation window areas 2005, 2012. Each line of the page navigation data defines one line of the page navigation. The page navigation data defines the file name (FileName), page number (Page), indent level (Level), page name (PageName), automatic name (AutoName), tree state (Tree), and current page (CurrentPage). Page="0" and Level="0" are the file name. As the Level increases, the page name on the page navigation is indented to the right.

[0230] The default is PageName="nnnn page" (nnnn is the page number) or "blank page", AutoName="True", Tree="Opened". When AutoName="True", the display device 2 automatically sets the handwritten recognized character string to PageName. When the user specifies the confirmed data that was handwritten and selects "Set to page name" from the operation command, the page name is changed to that confirmed data and AutoName="False". In other words, the page name set by the user takes precedence. "Tree" is set so that when Tree="Opened", all page names of rows below the current row that are one level lower than the current row are displayed in a tree, and when Tree="Closed", no tree display is performed (only the page name of the current row is displayed). When the page name is displayed, the same process is repeated for the displayed page name. For this reason, when the next row has a higher level than the current row, the display device 2 sets "Opened" to the Tree of the current row. CurrentPage indicates the page currently displayed on the display 220.

[0231] <Operation procedure> The operation of the display device 2 will be described with reference to the above configuration and Figs. 32 to 39. Figs. 32 to 39 are sequence diagrams illustrating an example of the process in which the display device 2 displays character string candidates and operation command candidates. The process in Fig. 32 starts when the display device 2 is started (when an application is started). Note that, due to space limitations, the functions in Fig. 6 are indicated by symbols in Figs. 32 to 39.

[0232] Before starting to input handwritten data, the user has already selected the buttons 2101 to 2110 on the pen operation panels 2001 and 2008 (the PenId has been specified). a. The pen button ID, ColorId and Angle are specified in the operation button definition data, b. Pen ID, Color ID, and Angle are registered in the pen ID control data.

[0233] S1: First, handwriting input display control unit 23 transmits a handwriting data start command to handwriting input storage unit 25. Handwriting input storage unit 25 secures a handwriting data area (a memory area for storing handwriting data). The handwriting data area may be secured after the user brings the pen into contact with handwriting input unit 21.

[0234] S2: Next, the user touches the pen to handwriting input unit 21. Handwriting input unit 21 detects pen-down and transmits the result to handwriting input display control unit 23.

[0235] S3: The handwritten input display control unit 23 transmits a stroke start to the handwritten input storage unit 25, and the handwritten input storage unit 25 secures a stroke area.

[0236] S4: When the user moves the pen while keeping it in contact with the handwriting input unit 21, the handwriting input unit 21 transmits the pen coordinates to the handwriting input display control unit 23.

[0237] S5: The handwriting input display control unit 23 specifies the pen ID received from the pen 2500 at the same time as the coordinates are input, and acquires the current pen ID control data stored in the pen ID control data storage unit 36. Since the pen ID is transmitted when the coordinates are input, the strokes and the pen ID are associated with each other. The pen ID control data storage unit 36 ​​transmits the pen ID control data to the handwriting input display control unit 23. Note that there is no AccountId because the user has not signed in.

[0238] S6: The handwriting input display control unit 23 transmits pen coordinate complement display data (data for interpolating discrete pen coordinates) to the display unit 22. The display unit 22 interpolates the pen coordinates using the pen coordinate complement display data, specifies the line type and thickness from the color definition data based on the ColorId, and displays the stroke.

[0239] S7: Handwriting input display control unit 23 transmits the pen coordinates, the reception time thereof, ColorId, and angle information to handwriting input storage unit 25. Handwriting input storage unit 25 adds the pen coordinates to the stroke. While the user is moving the pen, handwriting input unit 21 periodically repeats transmission of the pen coordinates to handwriting input display control unit 23, so that the processes of steps S4 to S7 are repeated until the pen is lifted.

[0240] S8: When the user removes the pen from the handwriting input unit 21, the handwriting input unit 21 transmits a pen-up signal to the handwriting input display control unit 23.

[0241] S9: The handwriting input display control unit 23 transmits a stroke end signal to the handwriting input storage unit 25, and the handwriting input storage unit 25 confirms the pen coordinates of the stroke. After the pen coordinates of the stroke are confirmed, it becomes impossible to add pen coordinates to the stroke.

[0242] S10: Next, the handwriting input display control unit 23 transmits an overlap state acquisition of the handwriting data rectangular area and the stroke rectangular area to the handwriting input storage unit 25 based on the handwriting data rectangular area 403. The handwriting input storage unit 25 calculates the overlap state and transmits the overlap state to the handwriting input display control unit 23.

[0243] The subsequent steps S11 to S17 are executed if the handwritten data rectangular area and the stroke rectangular area do not overlap.

[0244] S11: If the handwritten data rectangular area and the stroke rectangular area do not overlap, one piece of handwritten data is confirmed. Therefore, the handwritten input display control unit 23 transmits a retained data clear command to the handwriting recognition control unit 26.

[0245] S12 to S14: The handwriting recognition control unit 26 transmits a retained data clear command to the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32. The data relating to character string candidates and operation command candidates retained up to that point in time by the handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 are cleared. Note that at the time of clearing, the last handwritten stroke has not been added to the handwritten data.

[0246] S15: Handwriting input display control unit 23 transmits a handwriting data end notification to handwriting input saving unit 25. Handwriting input saving unit 25 confirms the handwritten data. Confirming handwritten data means that one piece of handwritten data has been completed (no more strokes will be added).

[0247] S16: Handwritten input display control unit 23 transmits a handwritten data start signal to handwritten input storage unit 25. In preparation for the start of handwriting of the next handwritten data (pen down), handwritten input storage unit 25 secures a new handwritten data area.

[0248] S17: Next, the handwriting input display control unit 23 transmits a stroke addition for the stroke completed in step S9 to the handwriting input saving unit 25. If steps S11 to S17 have been executed, the stroke to be added is the first stroke of the handwritten data, and the handwriting input saving unit 25 adds the stroke data to the starting handwritten data. If steps S11 to S17 have not been executed, the stroke to be added is added to the handwritten data already being handwritten.

[0249] S18: Next, the handwriting input display control unit 23 transmits a stroke addition to the handwriting recognition control unit 26. The handwriting recognition control unit 26 adds the stroke data to a stroke data holding area (area where stroke data is temporarily stored) in which character string candidates are stored.

[0250] S19: The handwriting recognition control unit 26 executes gesture handwriting recognition on the stroke data storage area. Gesture handwriting recognition refers to recognizing angle information from a straight line. Since gesture handwriting recognition is performed inside the operation guide 500, the handwriting recognition control unit 26 detects straight lines inside the operation guide 500. Position information of the operation guide 500 is transmitted to the handwriting recognition control unit 26 in step S67 described later.

[0251] S20: When a straight line is detected in the operation guide 500, the angle α between the straight line 522 drawn from the starting point of the straight line in the 6 o'clock direction and the straight line 521 input by the user in the counterclockwise direction is determined in units of 45 degrees. Then, the handwriting recognition control unit 26 stores the determined angle information in the pen ID control data storage unit 36 ​​in association with the pen ID contained in the stroke data of the straight line 521. Note that step S20 is executed when a straight line is detected in the operation guide 500. Therefore, it is possible to set a different angle in preference to the angle determined by pressing the buttons 2101 to 2115 on the pen operation panels 2001 and 2008.

[0252] S20-2: The handwriting recognition control unit 26 clears the selectable candidate display rectangle.

[0253] S21: Next, the handwriting recognition control unit 26 acquires angle information of the current pen ID control data from the pen ID control data storage unit 36 ​​by specifying the pen ID received from the handwriting input unit 21.

[0254] S22: The handwriting recognition control unit 26 rotates the stroke data in the stroke data storage area clockwise using the acquired angle information. In this way, the display device 2 can rotate the stroke data in accordance with the information on data processing according to the display position to perform character recognition.

[0255] S23: The handwriting recognition control unit 26 transmits the rotated stroke data to the handwritten signature authentication control unit 38. In this way, the stroke data is always transmitted to the handwritten signature authentication control unit 38 in a state where it is unclear whether it is a handwritten signature or not.

[0256] S24: The handwritten signature authentication control unit 38 receives the stroke data, and receives the registered handwritten signature data from the handwritten signature data storage unit 39. The handwritten signature authentication control unit 38 then compares (matches) the stroke data with the handwritten signature data, and stores the authentication result of the handwritten signature so that the authentication result of the handwritten signature can be obtained in the subsequent step S61. If the authentication is successful, the AccountId is registered in the pen ID control data.

[0257] S25: Next, the handwriting recognition control unit 26 performs handwriting recognition on the stroke data, and if there is a "check mark" or "X" in the registration or cancellation column of the form, processes the form, otherwise performs normal handwriting recognition processing.

[0258] S26: If there is a "check mark" in the registration or cancellation field of the handwritten signature data registration form, the handwritten signature data (stroke data) entered by the user into the handwritten signature registration form is sent by the handwritten signature recognition control unit 26 to the handwritten signature authentication control unit 38. The handwritten signature registration form is generated in the handwritten input storage unit 25 by the handwritten input display control unit 23 in step S86, which will be described later.

[0259] S27: The handwritten signature authentication control unit 38 registers the received handwritten signature data (stroke data) in the handwritten signature data storage unit 39. As a result, a SignatureId is assigned a number. The SignatureId is returned to the handwriting recognition control unit 26. If the SignatureId and the name entered in the name input field of the handwritten signature registration form are not present in the user-defined data, the handwriting recognition control unit 26 adds new user-defined data. The handwriting recognition control unit 26 assigns a number to an AccountId and saves the SignatureId in the user-defined data. If the name entered in the name input field is present in the user-defined data, the SignatureId is saved in the user-defined data. This process links the AccountId and SignatureId.

[0260] S28: By registering the handwritten signature data, the handwritten recognition control unit 26 deletes the handwritten signature registration form from the handwritten input storage unit 25.

[0261] S29: If there is a check mark in the registration or cancellation field of the user-defined data change form, handwriting recognition control unit 26 transmits the change value entered in user-defined data change form 562 to operation command definition unit 33. User-defined data change form 562 is generated in handwriting input storage unit 25 by handwriting input display control unit 23 in step S86, which will be described later.

[0262] S30: When the user-defined data change is executed, the handwriting recognition control unit 26 deletes the user-defined data change form 562 from the handwriting input storage unit 25.

[0263] S31: If the registration or cancellation column of the form added in step S86 described later has an "X" mark, the handwriting recognition control unit 26 deletes from the handwriting input storage unit 25 the form added in step S86.

[0264] S33: If the processing is not form processing, the handwriting recognition control unit 26 transmits handwriting recognition character string candidates handwritten by the user to the handwriting recognition dictionary unit 27. The handwriting recognition dictionary unit 27 transmits language character string candidates that are linguistically likely to be correct to the handwriting recognition control unit 26.

[0265] If "RecommendMenuOff="True"" is currently set in the pen ID control data and operation commands are not displayed, the handwriting recognition control unit 26 does not perform the control related to the recognition in steps S33 to S47. This allows the display device 2 to reduce the processing load.

[0266] S34: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidate and the received language character string candidate to the character string conversion control unit 28.

[0267] S35: The character string conversion control unit 28 transmits the handwriting recognition character string candidates and the language character string candidates to the character string conversion dictionary unit 29. The character string conversion dictionary unit 29 transmits the converted character string candidates to the character string conversion control unit 28.

[0268] S36 : The character string conversion control unit 28 transmits the received converted character string candidate to the predictive conversion control unit 30 .

[0269] S37: The predictive conversion control unit 30 transmits the received converted character string candidate to the predictive conversion dictionary unit 31. The predictive conversion dictionary unit 31 transmits the predicted character string candidate to the predictive conversion control unit 30.

[0270] S38 : The predictive conversion control unit 30 transmits the received predicted character string candidate to the operation command recognition control unit 32 .

[0271] S39: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. In this way, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) matching the predicted character string candidate.

[0272] Thereafter, the display device 2 performs similar processes up to the transmission of operation command candidates shown in steps S40 to S47. S 40 : The character string conversion control unit 28 transmits the received converted character string candidate to the operation command recognition control unit 32 .

[0273] S41: The operation command recognition control unit 32 transmits the received converted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. In this way, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) matching the converted character string candidate.

[0274] S42: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidates and the language character string candidates to the predictive conversion control unit 30.

[0275] S43: The predictive conversion control unit 30 transmits the handwritten recognition character string candidate and the received language character string candidate to the predictive conversion dictionary unit 31. The predictive conversion dictionary unit 31 transmits the predicted character string candidate to the predictive conversion control unit 30.

[0276] S 44 : The predictive conversion control unit 30 transmits the received predicted character string candidates to the operation command recognition control unit 32 .

[0277] S45: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. In this way, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) matching the predicted character string candidate.

[0278] S46: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidates and the received language character string candidates to the operation command recognition control unit 32.

[0279] S47: The operation command recognition control unit 32 transmits the handwriting recognition character string candidate and the received language character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidates to the operation command recognition control unit 32. In this way, the operation command recognition control unit 32 can acquire operation command candidates corresponding to the operation command definition data having a character string (String) matching the language character string candidate.

[0280] S 48 : Next, the handwriting recognition control unit 26 transmits a stroke addition to the operation command recognition control unit 32 .

[0281] S49: The operation command recognition control unit 32 transmits a request to acquire position information of the confirmed data to the handwritten input storage unit 25. The handwritten input storage unit 25 transmits the position information of the confirmed data to the operation command recognition control unit 32.

[0282] S50: The operation command recognition control unit 32 judges the selected data. The operation command recognition control unit 32 judges whether or not the position information of the stroke received from the handwriting recognition control unit 26 in the stroke addition in step S48 and the position information of the confirmed data received from the handwriting input storage unit 25 have a predetermined relationship based on the crossing line judgment condition 406 and the encircling line judgment condition 407. If there is confirmed data that can be judged as selected, the operation command recognition control unit 32 stores the confirmed data as selected data. In this case, since the selected data is specified, candidates for operation commands when there is selected data are obtained from the operation command definition unit 33.

[0283] In addition, the handwriting recognition control unit 26, the string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 each hold handwriting recognition string candidates, language string candidates, conversion string candidates, predicted string candidates, operation command candidates, and data related to selected data so that they can be acquired in subsequent steps S55 to S58.

[0284] S18-2: Immediately after transmitting the stroke addition command to the handwriting recognition control unit 26 in step S18, the handwriting input display control unit 23 transmits a command to start a selectable candidate display timer to the candidate display timer control unit 24. The candidate display timer control unit 24 starts this timer.

[0285] The following steps S51 to S53 are executed if a pen-down occurs before a certain period of time has elapsed (before the timer times out).

[0286] S51: If the user touches the pen to the handwriting input unit 21 before the timer times out, the handwriting input unit 21 transmits a pen-down signal (the same event as in step S2) to the handwriting input display control unit 23.

[0287] S52: The handwritten input display control unit 23 transmits a stroke start (same as step S3) to the handwritten input storage unit 25. The sequence thereafter is the same as that after step S3.

[0288] S53: Furthermore, the handwriting input display control unit 23 transmits a selectable candidate display timer stop command to the candidate display timer control unit 24. The candidate display timer control unit 24 stops the timer, because the timer is no longer necessary since pen-down has been detected.

[0289] Steps S54 to S103 are executed if the pen-down does not occur before a certain time has elapsed (before the timer times out), and therefore the operation guide 500 shown in FIG.

[0290] S54: If the user does not bring the pen into contact with the handwriting input unit 21 while the selectable candidate display timer is started, the candidate display timer control unit 24 transmits a timeout to the handwriting input display control unit 23.

[0291] S55: The handwriting input display control unit 23 transmits a handwriting recognition character string / language character string candidate acquisition to the handwriting recognition control unit 26. The handwriting recognition control unit 26 transmits the handwriting recognition character string / language character string candidate currently held to the handwriting input display control unit 23.

[0292] S56: The handwriting input display control unit 23 transmits a conversion character string candidate acquisition to the character string conversion control unit 28. The character string conversion control unit 28 transmits the conversion character string candidates currently held to the handwriting input display control unit 23.

[0293] S57: The handwriting input display control unit 23 transmits a predicted character string candidate acquisition request to the predictive conversion control unit 30. The predictive conversion control unit 30 transmits the predicted character string candidates currently held to the handwriting input display control unit 23.

[0294] S58: The handwriting input display control unit 23 transmits the operation command candidate acquisition to the operation command recognition control unit 32. The operation command recognition control unit 32 transmits the operation command candidates and the selection data currently held to the handwriting input display control unit 23.

[0295] S59: Furthermore, the handwriting input display control unit 23 transmits the estimated writing direction acquisition to the handwriting input storage unit 25. The handwriting input storage unit 25 judges the estimated writing direction from the stroke addition time and the horizontal and vertical distances of the handwritten data rectangular area, and transmits the estimated writing direction to the handwriting input display control unit 23.

[0296] S60: Next, the handwriting input display control unit 23 specifies the pen ID received from the handwriting input unit 21 and acquires the ColorId, AccountId (if none exists) and Angle of the current pen ID control data from the pen ID control data storage unit 36.

[0297] S61: The handwriting input display control unit 23 obtains the authentication result of the handwritten signature from the handwritten signature authentication control unit 38. As a result, the SignatureId of the user is obtained, and when an operation command described later is executed, the AccountId is associated with the pen ID control data and registered. When the user signs in, color definition data associated with the user-defined data identified by the AccountId is identified. The handwriting input display control unit 23 can display handwritten data in which the color information of the user-defined data is highlighted in black and white, or the text data converted from the handwritten data.

[0298] S62: The handwriting input display control unit 23 determines whether or not "RecommendMenuOff="True"" is set in the pen ID control data of the pen 2500 used by the user, and if "RecommendMenuOff="True"", does not display the character string candidate 539. In this case, only the operation commands are displayed, but it is also possible not to display the operation commands either. In the case of "RecommendMenuOff="False", the handwriting input display control unit 23 creates selectable candidate display data as shown in FIG. 19 from these handwriting recognition character string candidates ("gi" in FIG. 19), language character string candidates (for example, "gi" although not displayed in FIG. 19), converted character string candidates ("minutes" and "skill test" in FIG. 19), predicted character string candidates ("Settle the skill test" and "Destination for minutes" in FIG. 19), operation command candidates ("Load minutes template" and "Save in minutes folder" in FIG. 19), each selection probability, and estimated writing direction. In addition, the handwriting input display control unit 23 rotates the selectable candidate display data (operation guide 500) counterclockwise using the Angle acquired in step S60, and transmits the rotated selectable candidate display data (operation guide 500) to the display unit 22 for display.

[0299] S63: The handwriting input display control unit 23 also rotates the rectangular area display data (rectangular frame) of the handwritten data and selected data (handwritten data rectangular area display 503 in FIG. 19) counterclockwise using the angle information acquired in step S60, and transmits it to the display unit 22 for display.

[0300] S64: The handwriting input display control unit 23 transmits a selectable candidate display deletion timer start to the candidate display timer control unit 24 in order to delete the selectable candidate display data a certain time after it is displayed. The candidate display timer control unit 24 starts this timer. If the operation guide 500 including the operation commands is not displayed because "RecommendMenuOff="True"", the selectable candidate display deletion timer is not started.

[0301] Steps S65 to S70 are executed when the user erases the selectable candidate display displayed on the display unit 22 while the selectable candidate deletion timer is started, or when a change occurs in the handwritten data (i.e., when a stroke of the handwritten data is added, deleted, moved, transformed or divided), or when no candidate is selected before the timer times out.

[0302] Furthermore, steps S65 to S67 are executed when the candidate display is erased or when a change occurs in the handwritten data.

[0303] S65: The handwriting input unit 21 transmits to the handwriting input display control unit 23 a notice of the deletion of the selectable candidate display or the occurrence of a change in the handwritten data.

[0304] S66: The handwritten input display control unit 23 transmits a command to stop the selectable candidate deletion timer. The candidate display timer control unit 24 stops the timer. This is because an operation has been performed on the handwritten data within a certain period of time, making the timer unnecessary.

[0305] S67: The handwriting input display control unit 23 stores the position information of the operation guide 500 in the handwriting recognition control unit 26 so that it can be used in the gesture judgment of the gesture handwriting recognition in step S19. The position information is, for example, the coordinates of the upper left corner and the lower right corner or coordinates equivalent thereto. This allows the handwriting recognition control unit 26 to judge whether the straight line used to input the angle information is within the operation guide 500.

[0306] S69: The handwriting input display control unit 23 erases the display by transmitting a selectable candidate display data erasure command to the display unit 22. In the case of "RecommendMenuOff="True", it is not necessary to erase only the operation command or to erase nothing.

[0307] S70: The handwriting input display control unit 23 erases the display by transmitting a command to erase the rectangular area display data of the handwritten data and the selected data to the display unit 22. Therefore, when the display of the operation command candidate is erased under a condition other than the selection of the operation command candidate, the handwritten data remains displayed as it is.

[0308] S68: On the other hand, if the selectable candidate display is not erased or the handwritten data is not changed while the selectable candidate deletion timer is started (if the user does not perform a pen operation), the candidate display timer control unit 24 sends a timeout to the handwritten input display control unit 23.

[0309] Similarly, after the selectable candidate display deletion timer times out, the handwriting input display control unit 23 executes steps S69 and S70. This is because the display unit 22 may delete the selectable candidate display data and the rectangular area display data of the handwritten data and selected data after a certain period of time has elapsed.

[0310] When the user selects a selectable candidate while the selectable candidate deletion timer is in progress, steps S71 to S103 are executed.

[0311] S71: When the user selects a selectable candidate while the selectable candidate deletion timer is starting, the handwriting input unit 21 transmits the character string candidate or the operation command candidate selection to the handwriting input display control unit 23.

[0312] S71-2: The handwriting input display control unit 23 transmits a selectable candidate display deletion timer stop command to the candidate display timer control unit 24. The candidate display timer control unit 24 stops this timer.

[0313] S72: Next, the handwriting input / display control unit 23 transmits a command to clear the stored data to the handwriting recognition control unit 26.

[0314] S73: The handwriting recognition control unit 26 transmits a command to clear the stored data to the character string conversion control unit 28.

[0315] S74: The handwriting recognition control unit 26 transmits a command to clear the stored data to the predictive conversion control unit 30.

[0316] S75: The handwriting recognition control unit 26 transmits a retained data clear command to the operation command recognition control unit 32. The handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 clear the data relating to the character string candidates and operation command candidates that have been retained thus far.

[0317] S76: Next, the handwriting input display control unit 23 transmits a command to erase the selectable candidate display data to the display unit 22, thereby erasing the display.

[0318] S77: The handwriting input display control unit 23 transmits a command to erase the rectangular area display data of the handwritten data and the selected data to the display unit 22, thereby erasing the display.

[0319] S78: The handwriting input display control unit 23 erases the display by transmitting a command to erase the display data and the pen coordinate complement display data transmitted in step S6 to the display unit 22. This is because the handwritten data, etc. are no longer necessary since a character string candidate or an operation command candidate has been selected.

[0320] S79: The handwritten input display control unit 23 transmits a command to the handwritten input storage unit 25 to delete the handwritten data.

[0321] When a character string candidate is selected, steps S80 to S82 are executed.

[0322] S80: When a character string candidate is selected, the handwritten input display control unit 23 transmits a character string data addition to the handwritten input storage unit 25.

[0323] S81: Furthermore, the handwriting input display control unit 23 transmits a character string data font acquisition to the handwriting input storage unit 25. The handwriting input storage unit 25 specifies the font name of the character string data associated with the PenId of the pen ID control data from the pen color selection button definition data, and transmits an estimated character size of the handwritten data to the handwriting input display control unit 23.

[0324] S82: Next, the handwriting input display control unit 23 uses the defined font received from the handwriting input storage unit 25 to transmit character string data display data to be displayed in the same position as the handwritten data to the display unit 22, thereby displaying the character string data. The handwriting input display control unit 23 specifies the line type and thickness from the color definition data based on the ColorId of the pen ID control data, and displays the text (character string data) rotated by the angle information. Therefore, the display device 2 can display character string data whose top and bottom are aligned vertically as viewed from the user's operation position.

[0325] When a candidate for an operation command is selected, steps S83 to S101 are executed. An operation command for the selected data for editing or modification is executed in steps S83 to S85.

[0326] S83: When a candidate for an operation command for selected data is selected (when selected data exists), the handwriting input display control unit 23 erases the display by transmitting a command to erase the selected data display data to the display unit 22. This is because the display unit 22 erases the original selected data.

[0327] S84: Next, the handwriting input display control unit 23 transmits an operation command for executing the selected data to the handwriting input saving unit 25. The handwriting input saving unit 25 transmits display data of the newly selected data (display data after editing or modification) to the handwriting input display control unit 23.

[0328] S85: Next, the handwriting input display control unit 23 transmits selected data display data to the display unit 22 to re-display the selected data after the operation command is executed.

[0329] Although there is selected data, the operation command "add page name" is executed in steps S83-2 to S84-2.

[0330] S83-2: When a candidate for the operation command “Add page name” is selected (when selected data exists), the handwriting input display control unit 23 requests the handwriting input saving unit 25 to acquire the selected data. The handwriting input saving unit 25 transmits the selected data to the handwriting input display control unit 23.

[0331] S84-2: The handwritten input display control unit 23 checks the page number currently being displayed on the display 220 using the page navigation data, and adds the selected data to the handwritten input storage unit 25 as a page name.

[0332] S86: When “register handwritten signature” in the operation command definition data 713 or “change settings” in the operation command definition data 716 is specified, the handwritten input display control unit 23 adds a handwritten signature registration form or a user-defined data change form to the handwritten input storage unit 25.

[0333] S87: When the operation command candidate “save file” or “print” is selected, the handwriting input display control unit 23 transmits a file transmission request to the file transmission / reception control unit 37.

[0334] S88: The file transmission / reception control unit 37 transmits to the handwritten input storage unit 25 a request to acquire the handwritten input storage data of the file transmission target.

[0335] S89: Handwriting input storage unit 25 determines whether the destination is a color-compatible device using the MIB, etc. Handwriting input storage unit 25 processes the file depending on whether the destination is a color-compatible device, a black-and-white compatible device, or a black-and-white emphasis compatible device.

[0336] S90: The file transmission / reception control unit 37 transmits the handwritten input data received from the handwritten input storage unit to a destination or writes it to a file.

[0337] S91: When the operation command candidate “read file” is selected, the handwriting input display control unit 23 transmits a file list information acquisition request to the file transmission / reception control unit 37.

[0338] S92: The file transmission / reception control unit 37 receives file list information from a storage medium such as a USB memory, a network storage, or a Web server, or from an external device.

[0339] S93: The file transmission / reception control unit 37 transmits file list information to the handwritten input display control unit 23.

[0340] S94: The handwriting input display control unit 23 transmits file list display data to the display unit 22. As a result, the display unit 22 displays the file list on the display.

[0341] S95: When the user selects a file and the handwriting input unit 21 accepts it, the handwriting input unit 21 transmits the file selection to the handwriting input display control unit 23.

[0342] S96: The handwritten input display control unit 23 transmits a file reception request for the selected file to the file transmission / reception control unit 37.

[0343] S97: The file transmission / reception control unit 37 acquires the file from the external device.

[0344] S98: The file transmission / reception control unit 37 stores the file in the handwritten input storage unit 25.

[0345] S99: The handwriting input saving unit 25 analyzes the file received from the file transmission / reception control unit 37 and converts the handwriting input originated data into handwriting input saved data (black and white emphasis / color convertible data). That is, the handwriting input saving unit 25 judges whether or not there is metadata, and if there is metadata, judges whether or not it is possible to convert the data into handwriting input saved data (whether or not there is a ColorId, etc.), and saves the data as handwriting input saved data. Then, the handwriting input saving unit 25 reads the ColorId for the handwriting input originated data, refers to the color definition data, and converts the data into black and white emphasis display associated with the ColorId. The handwriting input saving unit 25 transmits display data of the handwritten data to the handwriting input display control unit 23. In the case of data that is not handwriting input originated data, the handwriting input saving unit 25 reads color information, font, character size, etc. according to the file format and saves the data as handwriting input saved data.

[0346] S100: The handwriting input display control unit 23 displays the display data of the handwriting input data on the display unit 22. As a result, the display unit 22 displays the handwriting input originated data as handwritten data with black and white emphasis, and displays other data in black and white using conventional luminance conversion.

[0347] When operation command 512 for signing in is executed, handwriting input display control unit 23 acquires the pen ID received by display device 2 when operation command 512 is executed. Handwriting input display control unit 23 identifies the user-defined data having SignatureId acquired in step S61 and acquires AccountId from the user-defined data. Then, handwriting input display control unit 23 registers AccountId in the pen ID control data in association with the pen ID. This associates pen 2500 with the user, and display device 2 becomes able to process using the user-defined data.

[0348] When the user writes by hand or loads a file after signing in, the handwriting input display control unit 23 obtains the AccountId associated with the pen ID received by the display device 2 when the operation command is executed from the pen ID control data. The handwriting input display control unit 23 identifies the user-defined data by this AccountId, and sets color definition data, etc. in %~% of the operation command and executes it.

[0349] S101: When an input / output operation command is selected, the handwritten input display control unit 23 executes the operation command character string (Command) of the operation command definition data corresponding to the operation command selected by the user.

[0350] S102: If the user manually inputs angle information, the handwriting input display control unit 23 stores the received angle information in the pen ID control data storage unit 36 ​​in association with the pen ID received from the pen 2500 when the rotation operation button 511 is pressed.

[0351] S103: For the next handwritten data, handwritten input display control unit 23 transmits a handwritten data start signal to handwritten input storage unit 25. Handwritten input storage unit 25 secures a handwritten data area. After this, the processes of steps S2 to S103 are repeated.

[0352] <Display operation menu> Below, we will explain the details of the processing that could not be fully explained in the sequence diagram.

[0353] First, FIG. 40 is a flowchart showing an example of a procedure for the display device 2 to display the operation menus 301 and 302. In FIG.

[0354] When the user, for example, turns on the power of the display device 2, the display device 2 starts booting up (S201). The display device 2 reads and executes an OS (Operating System) stored in the SSD 204, and the OS reads and executes a program for the display device 2 stored in the SSD 204.

[0355] The display unit 22 displays operation menus 301, 302 having information on data processing according to the display position. First, the display unit 22 displays the user's operation menu 301 on the lower side (S202). The display unit 22 may display the operation menu 302 first, or may display them simultaneously. The display unit 22 reads out the operation button definition data 2401-2424 and acquires the icon file. The display unit 22 determines the position based on the Position. The display unit 22 determines the rotation angle based on the Angle. The display unit 22 displays icons for each of the buttons 2101-2115, 2201, 2202, 2203, 2205, 2301, 2302, 2304, 2305 on the display 220. Since the lower user angle is 0 degrees, the icons of buttons 2101 to 2115, 2201, 2202, 2203, 2205, 2301, 2302, 2304, and 2305 are displayed without rotation. Note that the file name 2206 is not displayed until a file is loaded, and the current page 2203 is "1 / 1."

[0356] Next, the display unit 22 displays the operation menu 302 for the upper user (S203). The display unit 22 reads out the operation button definition data 2425-2448 and acquires the icon file. The display unit 22 determines the position based on the Position. The display unit 22 determines the rotation angle based on the Angle. The display unit 22 displays the icons of the buttons 2101-2115, 2201, 2202, 2203, 2205, 2301, 2302, 2304, and 2305 on the display 220. Since the Angle for the upper user is 180 degrees, the icons of the buttons 2101-2115, 2201, 2202, 2203, 2205, 2301, 2302, 2304, and 2305 are displayed rotated by 180 degrees. This allows buttons to be displayed that are easy for each user to see.

[0357] <Executing commands in operation button definition data> A case where one of the buttons 2101 to 2115 on the pen operation panels 2001 and 2008 is selected will be described with reference to Fig. 41. Fig. 41 is an example of a flow chart illustrating the process when one of the buttons 2101 to 2115 on the pen operation panels is pressed.

[0358] First, the handwriting input unit 21 accepts the selection of any one of the buttons 2101 to 2115 (S301). Here, it is assumed that the button 2101 has been pressed. The handwriting input unit 21 detects the coordinates touched by the pen 2500, and the buttons 2101 to 2115 are registered in the operation button definition data 2401 to 2415, so the pressed button 2101 is identified.

[0359] Next, the handwriting input display control unit 23 executes “ChangePen ThinBlack” of the operation button definition data 2401 corresponding to the pressed button 2101, so the handwriting input display control unit 23 sets the PenId and ColorId="ThinBlack" of the pen 2500 used by the user to the pen ID control data (S302).

[0360] Furthermore, the handwriting input display control unit 23 sets the "Angle" of the operation button definition data 2401 corresponding to the pressed button 2101 in the pen ID control data (S303).

[0361] Since the pen color is set in the pen ID control data, when the display unit 22 displays the handwritten data and character string data on the display 220, the data can be displayed in the line type defined in the color definition data. In addition, since the angle is set in the pen ID control data, the handwritten data can be rotated before character recognition.

[0362] Although the buttons on the pen operation panels 2001 and 2008 have been used as examples in FIG. 41, the PenId and Angle can be set in the pen ID control data by pressing any of the page operation panels 2002 and 2009, the page navigation operation panels 2003 and 2010, and the page navigation panels 2004 and 2011.

[0363] <Execute the operation command "Set to page name"> FIG. 42 is an example of a flowchart illustrating the process when the user executes the operation command "set to page name."

[0364] First, the user selects the confirmed data (text data after handwriting recognition) already displayed on the display 220 by drawing a crossing line or a surrounding line. This causes the display unit 22 to display an operation command "Set as page name." When the user presses this operation command, the handwriting input unit 21 accepts the selection of the operation command (S401). Since the coordinates of the operation guide 500 are known, the selected operation command is identified by the coordinates touched by the tip of the pen 2500.

[0365] The handwritten input display control unit 23 obtains the operation command definition data of the selected operation command, Name="Set to page name", and executes Command=SetPageName (S402).

[0366] The handwriting input display control unit 23 sets the selected data specified by the straddling or encircling lines to the PageName of the page being displayed on the display 220 (the page where CurrentPage=True in the page navigation data) (S403). This allows the user to assign a page name to the page by the simple operation of selecting the confirmed data.

[0367] Note that until the user assigns a page name, a default value (for example, page X, blank page, etc.) is set as the page name of the page when the user starts to write by hand on the page.

[0368] <Display page navigation> FIG. 43 is a flowchart illustrating an example of a method for displaying a page navigation.

[0369] The display unit 22 determines whether the page navigation window button (button 2115) has been operated to open (S501). Since coordinates are set in the operation button definition data 2439 of the page navigation window button, it is possible to determine whether the button has been pressed from the coordinates touched by the pen 2500.

[0370] When an operation to open the page navigation window button is input, the display unit 22 opens the page navigation window areas 2005 and 2012 (S502). Since the page navigation window area is closed (there is no page navigation window area) until it is opened, the display unit 22 reduces the entire display 220 horizontally to leave the right hand side of the display 220 empty. Note that it is either or both of the page navigation window areas 2005 and 2012 operated by the user that are opened.

[0371] The number of page navigation window areas is at most the same as the number of operation menus. The number of operation menus may be the number of types of "Angle" included in the operation button definition data. Alternatively, the number of operation menus may be directly set as a numerical value in the display device 2. The page navigation window area 2012 is also displayed at a position rotated by the "Angle". Therefore, for example, the right hand side of the user becomes the page navigation window area.

[0372] Next, the display unit 22 acquires the page navigation data 2501 to 2511 (S503). The display unit 22 processes the page navigation data 2501 to 2511 line by line.

[0373] First, the display unit 22 identifies the file name (S504). The display unit 22 determines whether the file name is a file name by whether "Page=0" and "Level=0". Since the file name is always displayed, the display unit 22 displays the FileName (S505).

[0374] Next, the display unit 22 judges whether or not the "Tree" of the file in question is "Open" (S506). If there is no "Tree" setting, it may be judged as No.

[0375] If the "Tree" of the file is Open, the page navigation displays the page names in the file, so the display unit 22 displays all page names one level lower (S507).

[0376] Next, the display unit 22 determines whether or not the page that displays the page name has a Tree setting (S508).

[0377] For a page with a Tree setting, the display unit 22 displays the names of all pages one level lower (S509). This process is performed for all nested pages.

[0378] Next, the display unit 22 determines whether the processing of one file is completed (S510).

[0379] If processing of one file has not been completed, the process returns to step S508, and the display unit 22 repeats the process.

[0380] If processing of one file has been completed, the display unit 22 determines whether or not all page navigation data has been processed (S511), and if processing has not been completed, the process returns to step S504 to process the next file.

[0381] If an operation to close the page navigation window button is input (Yes in S512), the display unit 22 closes the page navigation window area (S513). The display unit 22 erases the page navigation window area, expands the entire display 220 horizontally, and displays handwritten data, etc., on the entire display 220.

[0382] In this way, by displaying the page navigation, characters that were not visible in conventional thumbnails become easier to see, making it easier for the user to determine which page to open. In addition, the page navigation window area can be opened and closed, allowing for more use of the display 220. The user can display a page on the display 220 simply by selecting the file name or page name in the page navigation.

[0383] <Example of pen holder placement> Next, a pen holder that holds the pen 2500 will be described with reference to Fig. 44. Fig. 44 is a diagram for explaining an example of the arrangement of the pen holder. Fig. 44 shows an example of the arrangement of the pen holder when the display device 2 is placed flat on a desk (installed horizontally facing upward). Note that handwritten data and the like similar to those in Fig. 30 are displayed on the display 220.

[0384] The display device 2 has penholders 2603 and 2604 for the user on the lower side. Penholder 2603 is arrangement example 1, and penholder 2604 is arrangement example 2. In other words, it is sufficient to have either one of the penholders, but both penholders may be present. In addition, the display device 2 has penholders 2612 and 2613 for the user on the upper side. Penholder 2612 is arrangement example 1, and penholder 2613 is arrangement example 2.

[0385] Each of the pen holders 2603 and 2612 allows a pen to be taken out from the edge of the display 220 along which the operation menu is displayed, and each of the pen holders 2604 and 2613 stores the pen 2500 parallel to the edge.

[0386] Both layout examples 1 and 2 are located in positions that are easy for the user on the lower side or the user on the upper side to use, and are located close to the operation menus 301, 302 for the user on the lower side or the user on the upper side. For this reason, it is highly likely that the user will unconsciously use the operation menus 301, 302 to automatically set information (rotation angle) related to data processing according to the display position in the pen ID control data.

[0387] Once the information (angle) regarding data processing according to the display position is set in the pen ID control data, it remains valid even after the next startup. When the user removes the pen 2500 from the pen holder and starts writing by hand without using the operation menus 301 and 302, the information regarding data processing according to the display position has already been set in the pen ID control data. The user can use the pen 2500 as if it were a dedicated pen for the lower user or the upper user.

[0388] <Determining the layout of the operation menu> The layout of the operation menu may be defined in advance by the operation button definition data, or may be set by an administrator or the like.

[0389] FIG. 45 is an example of an arrangement decision screen that allows the administrator to decide the arrangement of the operation menu. The administrator causes the arrangement decision screen to be displayed on the display device 2 with a specified operation. On the arrangement decision screen, the operation menu with the default or current settings is displayed. In FIG. 45, the operation menus for the lower user (pen operation panel 2001, page operation panel 2002, and page navigation operation panel 2003) are displayed from the beginning. The administrator inputs operations to duplicate these with the pen 2500 or the like, and moves them to any desired location. In FIG. 45, pen operation panels 2008 and 2015 are arranged for users on the left and top. The page operation panel 2002 and page navigation operation panel 2003 can also be arranged in a similar manner.

[0390] When the administrator finishes the arrangement, the display device 2 creates operation button definition data based on the arrangement of the operation menu set by the administrator. As a result, the next time the device is started, the operation menu is displayed in the arrangement set by the administrator.

[0391] <Major Effects> As described above, the display device 2 of this embodiment displays an operation menu having information regarding data processing according to the display position, making it possible to process data according to the operation position of the user who operates the operation menu. In addition, a page navigation is displayed, allowing the user to understand what content is included in each page. Furthermore, by selecting a page name displayed in the page navigation, the user can easily switch the page displayed on the display 220. EXAMPLES

[0392] In the first embodiment, the display device 2 is described as having a large touch panel, but the display device is not limited to having a touch panel. In this embodiment, a projector type display device will be described.

[0393] <<Another configuration example 1 of the display device>> Fig. 46 is a diagram showing another example of the configuration of a display device. In Fig. 46, a projector 411 is installed on the upper side of a normal whiteboard 413. This projector 411 corresponds to the display device. The normal whiteboard 413 is not a flat panel display integrated with a touch panel, but a whiteboard on which a user directly writes by hand with a marker. The whiteboard may be a blackboard, and may be any flat surface large enough to project an image.

[0394] The projector 411 has an ultra-short focal length optical system and can project an image with little distortion from a distance of about 10 cm onto the whiteboard 413. This image may be transmitted from the PC 400-1 connected wirelessly or by wire, or may be stored in the projector 411.

[0395] A user writes by hand on the whiteboard 413 using a dedicated electronic pen 2700. The electronic pen 2700 has a light-emitting part, for example at the tip, that is switched on and emits light when the user presses it against the whiteboard 413 to write by hand. The light has a wavelength of near infrared or infrared, so it is invisible to the user's eyes. The projector 411 has a camera that captures the light-emitting part, analyzes the image, and identifies the direction of the electronic pen 2700. The electronic pen 2700 also emits sound waves along with the light emission, and the projector 411 calculates the distance based on the arrival time of the sound waves. The projector 411 can identify the position of the electronic pen 2700 based on the direction and distance. A stroke is drawn (projected) at the position of the electronic pen 2700.

[0396] Since the projector 411 projects the menu 430, when the user presses a button with the electronic pen 2700, the projector 411 identifies the pressed button by the position of the electronic pen 2700 and the ON signal of the switch. For example, when the save button 431 is pressed, the strokes (a set of coordinates) handwritten by the user are saved by the projector 411. The projector 411 saves the handwritten information in a predetermined server 412 or USB memory 2600 or the like. The handwritten information is saved for each page. Since the coordinates are saved instead of image data, the user can re-edit it. However, in this embodiment, the menu 430 does not need to be displayed because the operation command can be called up by handwriting.

[0397] <<Another configuration example 2 of the display device>> 47 is a diagram showing another example of the configuration of the display device 2. In the example of FIG.

[0398] The terminal device 600 is connected by wire to the image projection device 700A and the pen motion detection device 810. The image projection device 700A projects image data input by the terminal device 600 onto the screen 800.

[0399] The pen operation detection device 810 communicates with the electronic pen 820 and detects the operation of the electronic pen 820 in the vicinity of the screen 800. Specifically, the electronic pen 820 detects coordinate information indicating a point indicated by the electronic pen 820 on the screen 800 and transmits the coordinate information to the terminal device 600.

[0400] The terminal device 600 generates image data of a stroke image input by the electronic pen 820 based on the coordinate information received from the pen operation detection device 810, and causes the image projection device 700A to draw the stroke image on the screen 800.

[0401] Furthermore, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by combining the background image projected by the image projection device 700A and the stroke image input by the electronic pen 820.

[0402] <<Another configuration example 3 of the display device>> 48 is a diagram showing a configuration example of a display device. In the example of Fig. 48, the display device 2 has a terminal device 600, a display 800A, and a pen motion detection device 810.

[0403] The pen action detection device 810 is disposed near the display 800A, detects coordinate information indicating a point indicated by the electronic pen 820A on the display 800A, and transmits the coordinate information to the terminal device 600. In the example of Fig. 48, the electronic pen 820A may be charged by the terminal device 600 via a USB connector.

[0404] Based on the coordinate information received from the pen operation detection device 810, the terminal device 600 generates image data of a stroke image input by the electronic pen 820A, and displays the image data on the display 800A.

[0405] <<Another Configuration Example 4 of the Display Device>> Fig. 49 is a diagram showing a configuration example of a display device. In the example of Fig. 49, the display device 2 includes a terminal device 600 and an image projection device 700A.

[0406] The terminal device 600 performs wireless communication (Bluetooth (registered trademark) or the like) with the electronic pen 820B and receives coordinate information of a point indicated by the electronic pen 820B on the screen 800. Then, based on the received coordinate information, the terminal device 600 generates image data of a stroke image input by the electronic pen 820B and causes the image projection device 700A to project the stroke image.

[0407] Furthermore, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by combining the background image projected by the image projection device 700A and the stroke image input by the electronic pen 820.

[0408] As described above, each of the above-described embodiments can be applied to various system configurations.

[0409] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0410] For example, the display method of the present embodiment can be suitably applied to any information processing device having a touch panel. In addition, devices having the same functions as a display device are also called electronic blackboards, electronic whiteboards, electronic information boards, interactive boards, etc. Information processing devices equipped with a touch panel may be, for example, PJ (Projector), output devices such as digital signage, HUD (Head Up Display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, network home appliances, notebook PCs (Personal Computers), mobile phones, smartphones, tablet terminals, game consoles, PDAs (Personal Digital Assistants), digital cameras, wearable PCs, desktop PCs, etc.

[0411] In this embodiment, the server may perform part of the processing performed by the display device 2. For example, the display device transmits stroke information to the server, and obtains information to be displayed on the operation guide 500 from the server and displays the information.

[0412] In this embodiment, the display device 2 detects the coordinates of the pen by detecting the coordinates of the pen tip with a touch panel, but the coordinates of the pen tip may be detected by ultrasonic waves. The pen also emits ultrasonic waves together with light, and the display device 2 calculates the distance based on the arrival time of the ultrasonic waves. The display device 2 can identify the position of the pen based on the direction and distance. The projector draws (projects) the trajectory of the pen as a stroke.

[0413] In this embodiment, when there is selected data, candidates for editing, modification, and "add to page name" operation commands are displayed, and when there is no selected data, candidates for input / output operation commands are displayed. However, the display device 2 may simultaneously display candidates for editing, modification, and "add to page name" operation commands and candidates for input / output operation commands.

[0414] Furthermore, the user's handwritten signature data does not have to be stored in the display device 2. It may be stored in the cloud or in an in-house information processing device.

[0415] Moreover, the configuration examples in Fig. 6 and the like are divided according to main functions in order to facilitate understanding of the processing by the display device 2. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing by the display device 2 can be divided into more processing units according to the processing contents. Also, it can be divided so that one processing unit includes more processes.

[0416] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0417] In addition, in this embodiment, even if a threshold is exemplified as a comparison target, the threshold is not limited to the exemplified value. Therefore, in this embodiment, for all thresholds, the descriptions "less than threshold" and "below threshold" have the same meaning, and the descriptions "exceeding threshold" and "above threshold" have the same meaning. For example, the description "less than threshold" when the threshold is 11 has the same meaning as the description "below threshold" when the threshold is 10. Furthermore, the description "exceeding threshold" when the threshold is 10 has the same meaning as the description "below threshold" when the threshold is 11. [Explanation of symbols]

[0418] Display device 2 [Prior art documents] [Patent documents]

[0419] [Patent Document 1] JP 2018-026185 A

Claims

An information processing apparatus that displays a first operation menu including a first operation button and a second operation menu including a second operation button on a display, a display control unit that displays the first operation menu along an edge of a first side among four sides of the display, and displays the second operation menu along an edge of a second side opposite to the first side among the four sides of the display; a storage unit that stores, in association with each other, information of a pen used by a user and angle information used for character recognition; a first input reception unit that receives an input by pressing the pen from the user with respect to the first operation button included in the first operation menu displayed on the display or the second operation button included in the second operation menu; when receiving an input by pressing the pen with respect to the first operation button, sets the angle information to a first rotation angle; a storage unit that, when receiving an input by pressing the pen with respect to the second operation button, sets the angle information to a second rotation angle different from the first rotation angle; a second input reception unit that receives a handwritten input using the pressed pen from the user; when the first rotation angle is set in association with the information of the pressed pen, performs character recognition on handwritten data generated based on the handwritten input using the pen, using the first rotation angle set in association with the information of the pressed pen; a character recognition unit that, when the second rotation angle is set in association with the information of the pressed pen, performs character recognition on handwritten data generated based on the handwritten input using the pen, using the second rotation angle set in association with the information of the pressed pen; An information processing apparatus, characterized by comprising the above. According to claim 1, the display control unit: displays the first operation menu including the first operation button in a direction where a user who performs a handwritten input from the first side exists, according to the first rotation angle; displays the second operation menu including the second operation button in a direction where a user who performs a handwritten input from the second side exists, according to the second rotation angle. The information processing apparatus according to claim 1. According to claim 1 or 2, the information processing apparatus, characterized in that the first rotation angle is 0 degrees and the second rotation angle is 180 degrees.

4. Receive identification information of the pen for inputting handwritten data, When receiving an input by pressing the pen on the first operation button included in the first operation menu or the second operation button included in the second operation menu, set the identification information of the pen in association with the first rotation angle or the second rotation angle, The information processing apparatus according to any one of claims 1 to 3, wherein the handwritten data written with the pen is processed and displayed at the first rotation angle or the second rotation angle associated with the identification information of the pen.

5. The information processing apparatus according to claim 4, wherein the display control unit causes the character string obtained by recognizing the handwritten data to be displayed according to the first rotation angle or the second rotation angle.

6. The information processing apparatus according to any one of claims 1 to 5, further comprising a pen holder from which the pen can be taken out from the edge along which the first operation menu and the second operation menu are displayed, or a pen holder that stores the pen parallel to the edge.

7. When acquiring one or more files, The information processing apparatus according to any one of claims 1 to 6, wherein the file name of the file and the page name of each page included in the file are displayed.

8. When receiving a selection of the page name, The information processing apparatus according to claim 7, wherein the selected page is displayed using the information in which the page name is recorded for each page of the file.

9. The information processing apparatus according to claim 7 or 8, wherein the character string obtained by recognizing the handwritten data is displayed as the page name of the page on which the character string is displayed.

10. The information processing apparatus according to claim 7 or 8, wherein when receiving a selection of the character string obtained by recognizing the handwritten data, the character string is displayed as the page name of the page on which the character string is displayed.

11. The information processing apparatus according to any one of claims 8 to 10, wherein the file name and the page name are displayed in the same number as the first operation menu and the second operation menu. **Claim 12**: The information processing apparatus according to any one of claims 7 to 11, wherein the display control unit causes the file name and the page name to be displayed according to the first rotation angle or the second rotation angle. **Claim 13**: Receiving identification information of a pen for receiving a user's handwritten input from the pen, when the file name or the page name is pressed, associating and storing the identification information of the pen with the first rotation angle or the second rotation angle, The information processing apparatus according to claim 12, wherein the display control unit causes the character string obtained by recognizing the handwritten data to be displayed according to the first rotation angle or the second rotation angle. **Claim 14**: The information processing apparatus according to any one of claims 1 to 13, wherein the display control unit causes a third button for receiving a setting as to whether or not to display candidates for character strings obtained by recognizing the handwritten data to be displayed. **Claim 15** When receiving a press of the third button, even if the handwritten data is input, candidates for character strings obtained by recognizing the handwritten data are not displayed, and candidates for operation commands to be executed on the display device are displayed. The information processing apparatus according to claim 14. **Claim 16** The information processing apparatus according to any one of claims 1 to 15, wherein a layout determination screen for receiving the layout of the first operation menu and the second operation menu is displayed. **Claim 17**: An information processing method performed by an information processing apparatus that displays a first operation menu including a first operation button and a second operation menu including a second operation button on a display, wherein information of a pen used by a user and angle information used for character recognition are stored in a storage unit in association with each other, a process of displaying the first operation menu along an edge of a first side among four sides of the display, and displaying the second operation menu along an edge of a second side opposite to the first side among the four sides of the display; a process of receiving an input by pressing the pen from the user for the first operation button included in the first operation menu displayed on the display or the second operation button included in the second operation menu; when receiving an input by pressing the pen for the first operation button, setting the angle information to a first rotation angle, When receiving an input by pressing the pen against the second operation button, a process of setting the angle information to a second rotation angle different from the first rotation angle; A process of receiving a handwritten input using the pressed pen from the user; When the first rotation angle is set in association with the information of the pressed pen, performing character recognition on the handwritten data generated based on the handwritten input using the pen, using the first rotation angle set in association with the information of the pressed pen; When the second rotation angle is set in association with the information of the pressed pen, a process of performing character recognition on the handwritten data generated based on the handwritten input using the pen, using the second rotation angle set in association with the information of the pressed pen; An information processing method characterized by performing the above.

18. An information processing apparatus that displays a first operation menu including a first operation button and a second operation menu including a second operation button on a display, An information processing apparatus having a storage unit in which information of a pen used by a user and angle information used for character recognition are associated with each other, A process of displaying the first operation menu along an edge of a first side among the four sides of the display, and displaying the second operation menu along an edge of a second side opposite to the first side among the four sides of the display; A process of receiving an input by pressing the pen from the user against the first operation button included in the first operation menu displayed on the display or the second operation button included in the second operation menu; When receiving an input by pressing the pen against the first operation button, setting the angle information to a first rotation angle; When receiving an input by pressing the pen against the second operation button, a process of setting the angle information to a second rotation angle different from the first rotation angle; A process of receiving a handwritten input using the pressed pen from the user; When the first rotation angle is set in association with the information of the pressed pen, performing character recognition on the handwritten data generated based on the handwritten input using the pen, using the first rotation angle set in association with the information of the pressed pen; When the second rotation angle is set in association with the information of the pressed pen, a process of performing character recognition on the handwritten data generated based on the handwritten input using the pen, using the second rotation angle set in association with the information of the pressed pen, and A program for causing the execution.

Images