Information processing device, information processing method, program

Abstract

To provide a display device that can perform data processing depending on a display position of an operation menu.SOLUTION: In a display device 2 that displays an operation menu which is related to data processing performed for handwriting data, the operation menu is characterized to have information associated with the data processing depending on a display position of the operation menu.SELECTED DRAWING: Figure 30

Description

Technical Field

[0006] ,

[0004] , , , , technology ,

[0005] , Information processing device , , and a second operation menu including a second operation button ,

[0001] The present invention relates to Information processing device, information processing method and a program.

Background Art

[0002] A display device that displays data handwritten on a touch panel with a pen or a finger is known. A display device equipped with a relatively large touch panel is installed in a conference room or the like and used as an electronic blackboard or the like by a plurality of users.

[0003] The display device may display a menu for a user to select the color of a pen or the like (see, for example, Patent Document 1). Patent Document 1 discloses a handwriting input device in which menus for color setting, transparency setting, thickness setting, line type setting, stamp setting, and operation setting are displayed by pressing a pen button.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the conventional display device has a problem that information related to data processing is the same even if the display position of the operation menu is different. For example, when a user writes on the display device, each user operates the operation menu near himself / herself. However, regardless of the display position of the operation menu operated by the user, the data processing performed by the display device on the handwritten data is the same.

[0005] In view of the above problems, the present invention aims to provide technology that can perform data processing according to the display position of an operation menu.

Means for Solving the Problems

[0006] In view of the above problems, the present invention First operation button including the first displays an operation menu and a second operation menu including a second operation button by On the display displaying Information processing deviceAnd, A display control unit that displays the first operation menu along the edge of the first of the four sides of the display, and displays the second operation menu along the edge of the second of the four sides of the display opposite to the first side; and a storage unit that stores information of the pen used by the user and angle information used for character recognition in association with each other. , A first input receiving unit that receives input from a user by pressing the pen to the first operation button included in the first operation menu or the second operation button included in the second operation menu displayed on the display; a storage unit that, when input by pressing the pen to the first operation button is received, sets the angle information to a first rotation angle, and when input by pressing the pen to the second operation button is received, sets the angle information to a second rotation angle different from the first rotation angle; a second input receiving unit that receives handwriting input from the user using the pressed pen; a character recognition unit that, when the first rotation angle is set in association with the information of the pressed pen, recognizes the handwriting data generated based on the handwriting input using the pen using the first rotation angle set in association with the information of the pressed pen, and when the second rotation angle is set in association with the information of the pressed pen, recognizes the handwriting data generated based on the handwriting input using the pen using the second rotation angle set in association with the information of the pressed pen; It is characterized by having the following: [Effects of the Invention]

[0007] It is possible to perform data processing according to the display position of the operation menu. technology We can provide this. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of an operation menu displayed by a display device. [Figure 2] This is an example of a display device placed on a flat surface. [Figure 3] This is a diagram showing an example of a pen in an oblique view. [Figure 4] This figure shows an example of an overall configuration diagram of a display device. [Figure 5] This is an example of a hardware configuration diagram for a display device. [Figure 6] This diagram illustrates the functions of a display device and a pen. [Figure 7] This figure shows an example of predefined control data. [Figure 8] This figure shows an example of dictionary data for the handwriting recognition dictionary section. [Figure 9] This figure shows an example of dictionary data for the string conversion dictionary section. [Figure 10] This figure shows an example of dictionary data for the predictive text dictionary section. [Figure 11] This figure shows an example of operation command definition data and system definition data held by the operation command definition unit. [Figure 12] This figure shows an example of user-defined data held by the operation command definition unit. [Figure 13] This figure shows an example of handwritten signature data held by the handwritten signature data storage unit. [Figure 14]It is a diagram showing an example of the handwritten input storage data stored by the handwritten input storage unit. [Figure 15] It is a diagram explaining the pen ID control data stored by the pen ID control data storage unit. [Figure 16] It is a diagram showing an example of color definition data. [Figure 17] It is a diagram showing an example of pen color selection button definition data. [Figure 18] It is a diagram showing an example of the pen color selection button displayed on the display. [Figure 19] It is a diagram showing an example of the operation guide and selectable candidates displayed by the operation guide. [Figure 20] It is a diagram showing a display example of candidates for operation commands based on operation command definition data when there is handwritten data. [Figure 21] [[ID=二十一]]It is a diagram explaining the input method of 90-degree angle information. [Figure 22] It is a diagram showing an example of the operation guide displayed when the user handwrites "Suzuki", which is the handwritten signature data registered by the user. [Figure 23A] It is an example of a top view of a flat display device. [Figure 23B] It is a top view of a flat display device when the display device converts handwritten data into English. [Figure 24] It is a diagram showing an example of a pen operation panel. [Figure 25] It is a diagram showing an example of a page operation panel. [Figure 26] It is a diagram showing an example of a page navigation operation panel. [Figure 27] It is a diagram showing an example of the operation button definition data of the operation menu operated by the lower user (Part 1). [Figure 28] It is a diagram showing an example of the operation button definition data of the operation menu operated by the lower user (Part 2). [Figure 29] It is a diagram showing an example of the operation button definition data of the operation menu operated by the upper user (Part 1). [Figure 30] This figure (part 2) shows an example of the operation button definition data for the operation menu operated by the user at the top. [Figure 31] This figure shows an example of page navigation data. [Figure 32] This is a sequence diagram (part 1) illustrating an example of the process by which a display device shows string candidates and operation command candidates. [Figure 33] This is a sequence diagram (part 2) illustrating an example of the process by which a display device shows string candidates and operation command candidates. [Figure 34] This is a sequence diagram (part 3) illustrating an example of the process by which a display device shows string candidates and operation command candidates. [Figure 35] This is a sequence diagram (part 4) illustrating an example of the process by which a display device shows string candidates and operation command candidates. [Figure 36] This is a sequence diagram (part 5) illustrating an example of the process by which a display device shows string candidates and operation command candidates. [Figure 37] This is an example sequence diagram (part 6) illustrating the process by which a display device shows string candidates and operation command candidates. [Figure 38] This is a sequence diagram (part 7) illustrating an example of the process by which a display device shows string candidates and operation command candidates. [Figure 39] This is an example sequence diagram (part 8) illustrating the process by which a display device shows string candidates and operation command candidates. [Figure 40] This is an example of a flowchart illustrating the procedure for a display device to show an operation menu. [Figure 41] This is an example of a flowchart illustrating the process that occurs when a button on the pen control panel is pressed. [Figure 42] This is an example of a flowchart illustrating the process that occurs when a user executes the operation command "Set as page name". [Figure 43] This is an example of a flowchart illustrating how to display page navigation. [Figure 44]This is a diagram illustrating an example of pen holder placement. [Figure 45] This is an example of a screen for determining the layout. [Figure 46] This figure shows another example of a display device configuration. [Figure 47] This figure shows another example of a display device configuration. [Figure 48] This figure shows another example of a display device configuration. [Figure 49] This figure shows another example of a display device configuration. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment for carrying out the present invention will be described with reference to the drawings, including a display device and a display method performed by the display device. [Examples]

[0010] The input method can be any method that allows handwriting by specifying coordinates on a touch panel. Examples include a pen, a person's finger or hand, or a rod-shaped object. Eye-tracking input is also possible. Handwritten data is data displayed as a trajectory of coordinate points reached by the user continuously moving the input method on the touch panel. The series of operations performed by the user, such as pressing the input method against the touch panel, moving it continuously, and then lifting it off the touch panel, is called a stroke. Data written by hand through strokes is called stroke data. Handwritten data has one or more stroke data. Handwritten input refers to the input of handwritten data by the user.

[0011] <Comparative Examples of Display Devices> Before describing the display device of this embodiment, we will first briefly explain an example of the display of the operation menu of the display device.

[0012] Figure 1 shows an example of an operation menu displayed by a display device. Because this display device 2 is propped up, the operation menu 102 is displayed on the right side of the display. The user can move the operation menu 102 to the left side of the display, but conventionally, only one operation menu 102 could be displayed at a time. Therefore, for example, even if the user was on the left side facing the display, it was difficult for that user to press the operation menu 102.

[0013] Next, Figure 2 shows an example of a flat-lying display device 2. In Figure 2, the users are facing each other, but even in this case, it would be difficult to operate if there is only one operation menu. Therefore, it is considered that the display device 2 should display operation menus for each user. The operation menu does not contain information about data processing according to the display position.

[0014] In this case, the user can press an operation menu near them, but the operation menu does not contain information about data processing according to the display position, making it difficult to perform data processing according to the display position even when the user presses an operation menu. For example, when the display device 2 recognizes handwritten characters, it cannot recognize them correctly unless the handwritten data is rotated according to the orientation of the characters, but conventionally, the content of data processing has not been changed according to the operation menu that is pressed.

[0015] Therefore, in this embodiment, a display device that displays an operation menu having information regarding data processing according to the display position will be described.

[0016] Furthermore, as shown in Figure 1, the display device stores page data, including handwritten data, on a page-by-page basis and displays thumbnails 106 of each page at the bottom of the display. However, the thumbnails 106 are small, making it difficult to discern their contents. Also, there is an upper limit to the number of pages that can be displayed as thumbnails 106, making it impossible to display all the content contained in each page.

[0017] Therefore, in this embodiment, the text selected by the user, or the text automatically selected by the display device, is extracted for each page, and a page navigation display showing these texts is shown for each page.

[0018] <About Terminology> Handwritten data refers to data displayed on a screen as a trajectory, representing a sequence of coordinate points created by a user continuously moving an input device. The series of operations—pressing the input device against the screen, moving it continuously, and then lifting it—is called a stroke, and the data created by this stroke is called stroke data. Handwritten data can consist of one or more stroke data points.

[0019] Information regarding data processing refers to the processing of handwritten data or handwritten input-induced data displayed by the display device 2. Examples include rotation of handwritten data for handwriting recognition and rotation after character recognition. The display device 2 can rotate handwritten data according to the display position of the operation menu. Furthermore, if it is a color display, it can control whether to display in color or monochrome, and which colors to use. The display device 2 can also control which font to use.

[0020] Black and white compatible devices are devices that can only output white, black, or grayscale. Examples include e-paper and monochrome printers. They can also be described as devices that do not support color.

[0021] Electronic paper is also called E-paper. Electronic paper is a general term for displays that are about 1 / 10th the thickness of a millimeter and can display and erase data using electrical means. The representative technology that makes electronic paper possible is the microcapsule electrophoresis method, which is called E-ink (registered trademark) technology. Each microcapsule contains white particles and black particles, with the white particles being positively charged and the black particles being negatively charged.

[0022] Color-compatible devices are devices that can output not only black and white but also color. Examples include devices with LCD or OLED displays, and color printers. Color-compatible devices can also be used as black and white devices.

[0023] A monochrome enhancement device is a device that can only output white, black, or grayscale, but performs monochrome enhancement display based on the color information associated with handwritten input data. An example of this device is the display device of this embodiment. Note that the display device is capable of handwritten input and may also be called an input device or handwritten input device.

[0024] Handwriting-input-derived data refers to data entered by handwriting on a touch panel. It does not matter whether the handwriting remains handwritten after input or is converted to text data. Furthermore, data acquired from external devices is also considered to have originated from handwriting input. In addition to text data converted through character recognition, handwriting-input-derived data may also include data converted based on user operations, such as stamps displayed as fixed characters or marks like "Completed" or "Secret," shapes like circles and stars, and lines.

[0025] Data that is not handwritten input-derived data refers to data other than data entered by handwriting on a touch panel, such as image data, handwritten data entered on a device that does not support black and white enhancement, or text data.

[0026] <An example of the pen's appearance> Figure 3 shows an example of a perspective view of the pen 2500. Figure 3 shows an example of a multi-functional pen 2500. A pen 2500 that has a built-in power supply and can send commands to the display device 2 is called an active pen (a pen that does not have a built-in power supply is called a passive pen). The pen 2500 in Figure 3 has one physical switch at the tip, one at the end of the pen, and two on the side of the pen. The tip is for writing, the end of the pen is for erasing, and the side of the pen is for assigning user functions. In this embodiment, the pen 2500 also has a non-volatile memory and stores a pen ID that does not overlap with other pens.

[0027] Furthermore, using a pen with a switch can reduce the number of steps required for the user to operate the display device 2. A pen with a switch mainly refers to an active pen, but in the case of electromagnetic induction, even passive pens that do not have a built-in power supply can generate power using only an LC circuit, so this includes not only active pens but also electromagnetic induction passive pens. Pens with switches using optical, infrared, and capacitive methods other than electromagnetic induction are active pens.

[0028] The hardware configuration of the Pen 2500 is assumed to be similar to that of a general control system equipped with communication functions and a microcontroller. The Pen 2500 may use electromagnetic induction or active electrostatic coupling methods. In addition, the Pen 2500 may have functions such as pressure sensitivity, tilt detection, and hover function (displaying the cursor before the pen touches the surface).

[0029] <Overall configuration of the device> The overall configuration of the display device 2 according to this embodiment will be explained using Figure 4. Figure 4 is a diagram showing the overall configuration of the display device 2. Figure 4(a) shows an example of the display device 2, which is used as a horizontally oriented electronic whiteboard suspended on a wall.

[0030] As shown in Figure 4(a), a display 220, which is an example of a display device, is installed at the top of the display device 2. User U can use the pen 2500 to handwrite (input or draw) characters and other things on the display 220.

[0031] Figure 4(b) shows a display device 2 used as a vertically oriented electronic whiteboard suspended from a wall.

[0032] Figure 4(c) shows the display device 2 placed flat on the desk 230. Since the display device 2 is about 1 cm thick, there is no need to adjust the height of the desk when it is placed flat on a regular desk. In addition, the user can easily move the display device 2.

[0033] Furthermore, the tilt sensor automatically detects which orientation the display device 2 is being used in.

[0034] <Device Hardware Configuration> Next, the hardware configuration of the display device 2 will be explained using Figure 5. As shown in the figure, the display device 2 has the configuration of an information processing device or a computer. Figure 5 is an example of a hardware configuration diagram of the display device 2. As shown in Figure 5, the display device 2 is equipped with a CPU (Central Processing Unit) 201, ROM (Read Only Memory) 202, RAM (Random Access Memory) 203, and SSD (Solid State Drive) 204.

[0035] Of these components, the CPU 201 controls the overall operation of the display device 2. The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and the IPL (Initial Program Loader). The RAM 203 is used as the work area for the CPU 201. The SSD 204 stores various data, such as programs for the display device 2.

[0036] The display device 2 also includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, a power switch 227, a tilt sensor 217, a serial interface 218, a speaker 219, a microphone 221, a wireless communication device 222, an infrared interface 223, a power control circuit 224, an AC adapter 225, and a battery 226.

[0037] The display controller 213 controls and manages the screen display in order to output the output image to the display 220, etc. The touch sensor 216 detects when the pen 2500 or the user's hand (the pen or the user's hand serves as an input means) touches the display 220. The touch sensor 216 also receives the pen ID.

[0038] The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 performs coordinate input and coordinate detection. An example of how to input and detect coordinates is described below. For example, in the case of an optical system, two light-emitting and receiving devices installed at both ends of the upper part of the display 220 emit multiple infrared rays parallel to the display 220. Reflective members provided around the display 220 reflect multiple infrared rays, and the light-receiving element receives the light that returns along the same optical path as the light emitted. The touch sensor 216 outputs position information of the infrared rays emitted by the two light-emitting and receiving devices that are blocked by the object to the touch sensor controller 215, and the touch sensor controller 215 identifies the coordinate position, which is the contact position of the object. The touch sensor controller 215 also has a communication unit 215a and can communicate wirelessly with the pen 2500. For example, if communication is using a standard such as Bluetooth®, a commercially available pen can be used. If one or more pens 2500 are registered in the communication unit 215a beforehand, the user can communicate with the display device 2 without having to perform connection settings to allow the pen 2500 to communicate with the display device 2.

[0039] The power switch 227 is a switch for turning the power of the display device 2 ON / OFF. The tilt sensor 217 is a sensor that detects the tilt angle of the display device 2. The tilt sensor 217 is mainly used to detect whether the display device 2 is being used in one of the installation states shown in Figure 4(a), Figure 4(b), or Figure 4(c), and can automatically change the thickness of characters, etc., according to the installation state.

[0040] The serial interface 218 is a communication interface with external devices such as USB and LAN interfaces. The serial interface 218 is used for inputting information from external sources. The speaker 219 is used for audio output, and the microphone 221 is used for audio input. The wireless communication device 222 communicates with the user's portable terminal and relays connections to the internet, for example. The wireless communication device 222 communicates using Wi-Fi or Bluetooth (registered trademark), but the communication standard is not specified. The wireless communication device 222 forms an access point, and when the user sets the SSID (Service Set Identifier) ​​and password obtained by the user on the user's portable terminal, the user can connect to the access point.

[0041] It is preferable that the wireless communication device 222 has two access points. a. Access point → Internet b. Access point → Internal network → Internet Access point a is for external users; users cannot access the internal network but can use the internet. Access point b is for internal users; users can use both the internal network and the internet.

[0042] The infrared interface 223 detects the adjacent display device 2. The infrared interface 223 can detect only the adjacent display device 2 by utilizing the directional properties of infrared light. Preferably, one infrared interface 223 is provided on each side, so that it is possible to detect in which direction the other display device 2 is positioned relative to the display device 2. The adjacent display device 2 can display handwritten information that has been previously written (handwritten information from another page, with the area of ​​one display 220 considered as one page).

[0043] The power control circuit 224 controls the AC adapter 225 and battery 226, which are the power sources for the display device 2. The AC adapter 225 converts the alternating current (AC) shared by the commercial power supply to direct current (DC).

[0044] If the display 220 is so-called electronic paper, it consumes little to no power to maintain the image after it has been drawn, so it can be powered by a battery 226. This makes it possible to use the display device 2 for applications such as digital signage even in places where it is difficult to connect to a power source, such as outdoors.

[0045] Furthermore, the display device 2 is equipped with a bus line 210. The bus line 210 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 201 shown in Figure 5.

[0046] The touch sensor 216 is not limited to optical; it may also be a capacitive touch panel that identifies the contact position by detecting changes in capacitance. The touch sensor 216 may also be a resistive touch panel that identifies the contact position by voltage changes between two opposing resistive films. The touch sensor 216 may use various detection means, such as an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by contact between an object and the display. The touch sensor 216 does not require an electronic pen to detect the presence or absence of a pen tip touch. In this case, a fingertip or a pen-shaped object can be used for touch operation. The pen 2500 does not need to be a long, slender pen shape.

[0047] <About the device's functions> Next, the functions of the display device 2 and the pen 2500 will be explained using Figure 6. Figure 6(a) is an example of a functional block diagram showing the functions of the display device 2 in a block-like manner. The display device 2 includes a handwriting input unit 21, a display unit 22, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a string conversion control unit 28, a string conversion dictionary unit 29, a predictive conversion control unit 30, a predictive conversion dictionary unit 31, an operation command recognition control unit 32, an operation command definition unit 33, a pen ID control data storage unit 36, a file transmission / reception control unit 37, a handwriting signature authentication control unit 38, and a handwriting signature data storage unit 39. Each function of the display device 2 is a function or means realized by any of the components shown in Figure 5 operating according to instructions from the CPU 201 in accordance with a program deployed from the SSD 204 onto the RAM 203.

[0048] The handwriting input unit 21 is implemented using a touch sensor 216 or the like, and accepts handwriting input from the user and receives a pen ID from the pen 2500. The handwriting input unit 21 converts the user's pen input d1 into pen operation data d2 (pen up, pen down, or pen coordinate data) with the pen ID and transmits it to the handwriting input display control unit 23. The pen coordinate data is transmitted periodically as discrete values, and the coordinates between the discrete values ​​are interpolated.

[0049] The display unit 22 is implemented by a display 220 or the like, and displays handwritten data, operation menus, etc. The display unit 22 converts the drawing data d3 written to the video memory by the handwriting input display control unit 23 into data according to the characteristics of the display 220, and transmits it to the display 220.

[0050] The handwriting input display control unit 23 performs overall control over handwriting input and display. The handwriting input display control unit 23 processes pen operation data d2 from the handwriting input unit 21 and transmits it to the display unit 22 for display. Details of the processing of pen operation data d2 and the display of strokes will be explained in Figures 32 to 39 below. The handwriting input display control unit 23 holds operation button definition data, which will be described later, and detects the operation menu (button) operated based on the pen operation data d2 from the handwriting input unit 21, and performs control according to the operation menu.

[0051] The Candidate Display Timer Control Unit 24 is a timer that controls the display of selectable candidates. The Candidate Display Timer Control Unit 24 generates the timing to start or stop the timer to begin displaying selectable candidates and to erase the display. Selectable candidates are the handwriting recognition string / language string candidates, conversion string candidates, string / predictive conversion candidates, and operation command candidates that are displayed as selectable in the operation guide (see Figure 19) described later. The Candidate Display Timer Control Unit 24 receives a timer start request d4 (or a timer stop request) from the handwriting input display control unit 23 and sends a timeout event d5 to the handwriting input display control unit 23.

[0052] The handwriting input storage unit 25 has a storage function for storing user data (handwritten data / string data). The handwriting input storage unit 25 receives and holds user data d6-1 from the handwriting input display control unit 23. The handwriting input storage unit 25 receives an acquisition request d6-2 from the handwriting input display control unit 23 and transmits the user data d7 stored in the handwriting input storage unit 25. The handwriting input storage unit 25 transmits the location information d36 of the confirmed data (string data that has already been recognized or handwritten data that has been confirmed not to be converted) to the operation command recognition control unit 32.

[0053] The handwriting recognition control unit 26 is a recognition engine that performs online handwriting recognition. Unlike typical OCR (Optical Character Reader), it recognizes characters (not only Japanese but also multiple languages ​​such as English), numbers, symbols (%), $, &, etc., and shapes (lines, circles, triangles, etc.) in parallel with the user's pen input. Various algorithms have been devised for recognition methods, but in this embodiment, known technologies can be used, so details are omitted.

[0054] The handwriting recognition control unit 26 receives pen operation data d8-1 from the handwriting input display control unit 23, performs handwriting recognition, and stores handwriting recognition string candidates. The handwriting recognition control unit 26 also stores language string candidates converted from handwriting recognition string candidates d12 using the handwriting recognition dictionary unit 27. Separately, when it receives an acquisition request d8-2 from the handwriting input display control unit 23, the handwriting recognition control unit 26 transmits the stored handwriting recognition string candidates and language string candidates d9 to the handwriting input display control unit 23.

[0055] The handwriting recognition dictionary unit 27 is dictionary data for language conversion in handwriting recognition. The handwriting recognition dictionary unit 27 receives the handwriting recognition string candidate d12 from the handwriting recognition control unit 26, converts it into a linguistically plausible language string candidate d13, and transmits it to the handwriting recognition control unit 26. For example, in the case of Japanese, it converts hiragana to kanji or katakana.

[0056] The string conversion control unit 28 controls the conversion of conversion string candidates into strings. A conversion string is a string that is highly likely to be generated by including a handwritten recognition string or a language string. The string conversion control unit 28 receives the handwritten recognition string and language string candidate d11 from the handwritten recognition control unit 26, converts them into conversion string candidates using the string conversion dictionary unit 29, and stores them. Separately, if the string conversion control unit 28 receives an acquisition request d14 from the handwritten input display control unit 23, it transmits the stored conversion string candidate d15 to the handwritten input display control unit 23.

[0057] The string conversion dictionary unit 29 is dictionary data for string conversion. The string conversion dictionary unit 29 receives the handwritten recognition string and language string candidate d17 from the string conversion control unit 28 and transmits the conversion string candidate d18 to the string conversion control unit 28.

[0058] The predictive text control unit 30 receives the handwritten recognition string and language string candidate d10 from the handwritten recognition control unit 26, and the conversion string candidate d16 from the string conversion control unit 28. The predictive text control unit 30 converts each of the handwritten recognition string, language string candidate, and conversion string candidate into predictive string candidates using the predictive text dictionary unit 31. A predictive text string is a string that is highly likely to be generated by including the handwritten recognition string, language string, or conversion string. Separately, if the predictive text control unit 30 receives an acquisition request d19 from the handwritten input display control unit 23, it sends the predictive string candidate d20 to the handwritten input display control unit 23.

[0059] The predictive text dictionary unit 31 contains dictionary data for predictive text conversion. The predictive text dictionary unit 31 receives handwritten recognition strings, language string candidates, and conversion string candidates d21 from the predictive text control unit 30, and transmits predictive string candidates d22 to the predictive text control unit 30.

[0060] The operation command recognition control unit 32 receives the handwritten recognition string and language string candidate d30 from the handwritten recognition control unit 26, and the conversion string candidate d28 from the string conversion control unit 28. The operation command recognition control unit 32 receives the prediction string candidate d29 from the prediction conversion control unit 30. Then, the operation command recognition control unit 32 sends an operation command conversion request d26 to the operation command definition unit 33 for each of the handwritten recognition string, language string candidate, conversion string candidate, and prediction string candidate d29, and receives the operation command candidate d27 from the operation command definition unit 33. The operation command recognition control unit 32 stores the operation command candidate d27.

[0061] If the operation command definition unit 33 finds that the operation command conversion request d26 partially matches the operation command definition, it sends a candidate operation command d27 to the operation command recognition control unit 32.

[0062] Furthermore, the operation command recognition control unit 32 receives pen operation data d24-1 from the handwriting input display control unit 23. The operation command recognition control unit 32 sends a request d23 to the handwriting input storage unit 25 to acquire location information of previously entered and confirmed confirmed data, and stores the confirmed data specified by the pen operation data as selected data (including location information). The operation command recognition control unit 32 identifies the location of the pen operation data d24-1 and the selected data that satisfies predetermined criteria. Separately, if the operation command recognition control unit 32 receives an acquisition request d24-2 from the handwriting input display control unit 23, it sends the held candidate operation command and the identified selected data d25 to the handwriting input display control unit 23.

[0063] The pen ID control data storage unit 36 ​​holds the pen ID control data (it can also be called a storage means). Before the handwriting input display control unit 23 transmits display data to the display unit 22, the pen ID control data storage unit 36 ​​transmits the pen ID control data d41 to the handwriting input display control unit 23. The handwriting input display control unit 23 draws the display data according to the operating conditions stored in association with the pen ID. Also, before the handwriting recognition control unit 26 performs handwriting recognition, the pen ID control data storage unit 36 ​​transmits the angle information d44 of the pen ID control data to the handwriting recognition control unit 26, and the handwriting recognition control unit 26 rotates the stroke according to the angle information stored in association with the pen ID and performs handwriting recognition.

[0064] Furthermore, after recognizing a straight line for setting angle information when the user handwrites characters, the handwriting recognition control unit 26 transmits the angle information d43 of the pen ID control data to the pen ID control data storage unit 36. The handwriting recognition control unit 26 stores the angle information d43 in association with the pen ID. Also, after the handwriting input display control unit 23 executes an operation command to set the angle information, the handwriting input display control unit 23 transmits the pen ID control data d42 to the pen ID control data storage unit 36. The handwriting input display control unit 23 stores the execution result of the operation command (angle information set by the user) in association with the pen ID. From then on, the stroke of that pen ID is rotated by the set angle information before handwriting recognition is performed.

[0065] Furthermore, the handwriting recognition control unit 26 transmits stroke data d49, which has been rotated clockwise using the angle information of the pen ID control data, to the handwriting signature authentication control unit 38. This enables authentication of handwritten signatures regardless of the user's operating position (from which direction the handwriting is performed relative to the display device 2).

[0066] The handwritten signature data storage unit 39 stores the handwritten signature data. When the handwritten signature data storage unit 39 receives a handwritten signature data acquisition request d45 from the handwritten signature authentication control unit 38, it transmits the handwritten signature data d46 to the handwritten signature authentication control unit 38. The format of the handwritten signature data depends on the handwritten signature authentication algorithm of the handwritten signature authentication control unit 38. The data stored in the handwritten signature data storage unit 39 is explained in Figure 13.

[0067] When the handwriting signature authentication control unit 38 receives stroke data d49 rotated clockwise from the handwriting recognition control unit 26, it sends a handwriting signature data acquisition request d45 to the handwriting signature data storage unit 39. The handwriting signature data storage unit 39 then sends the handwriting signature data d46 to the handwriting signature authentication control unit 38.

[0068] The handwritten signature authentication control unit 38 authenticates the user based on the handwritten signature data. Various algorithms have been devised for user authentication based on handwritten signature data, but in this embodiment, a technology that can recognize signatures with a recognition rate that does not impede practical use is used. For example, the handwritten signature authentication control unit 38 generates a feature vector whose elements include coordinates, pen pressure, and the time it takes to write each stroke, which constitute the handwritten signature data. The handwritten signature authentication control unit 38 appropriately weights the elements and compares the feature vector of the registered handwritten signature data with the feature vector of the username etc. that the user handwritten during sign-in. If the degree of match is above a threshold, the handwritten signature authentication control unit 38 determines that authentication is successful, and if it is below the threshold, it determines that authentication is failed.

[0069] The handwritten signature authentication control unit 38 stores the authentication result of the handwritten signature, which is the result of comparing the stroke data d49 and the handwritten signature data d46. Separately, when it receives an acquisition request d48 from the handwritten input display control unit 23, the handwritten signature authentication control unit 38 transmits the stored handwritten signature authentication result d47 to the handwritten input display control unit 23. The authentication result of the handwritten signature indicates whether the stroke data d49 and the handwritten signature data d46 can be considered to match, and if they are considered to match, it has a SignatureId, which will be described later, associated with the matched handwritten signature data d46.

[0070] If the handwriting recognition result of the handwriting recognition control unit 26 matches the operation command that instructs the execution of handwritten signature registration, the handwriting recognition control unit 26 retrieves the data d52 entered into the handwritten signature registration form (a frame into which handwritten signature data is entered, as described later) from the handwriting input storage unit 25. The handwriting recognition control unit 26 sends the handwritten signature data d50 from the data d52 to the handwriting signature authentication control unit 38. The handwriting signature authentication control unit 38 sends the received handwritten signature data d50 to the handwriting signature data storage unit 39 for registration.

[0071] If the handwriting recognition result from the handwriting recognition control unit 26 is an instruction to cancel or register a handwritten signature, the handwriting recognition control unit 26 sends a request d51 to the handwriting input storage unit 25 to delete the handwritten signature registration form, and deletes the handwritten signature registration form from the handwriting input storage unit 25.

[0072] If the handwriting recognition result from the handwriting recognition control unit 26 is an execution instruction for user-defined data modification, the handwriting recognition control unit 26 retrieves the data d53 entered into the user-defined data modification form from the handwriting input storage unit 25. The handwriting recognition control unit 26 sends the modified value d54 from the data d53 to the operation command definition unit 33 to modify the user-defined data. User-defined data is explained in Figure 12.

[0073] If the handwriting recognition result from the handwriting recognition control unit 26 indicates a cancellation instruction or execution of registration for a user-defined data change form, the handwriting recognition control unit 26 sends a deletion request d55 for the user-defined data change form to the handwriting input storage unit 25, and deletes the user-defined data change form from the handwriting input storage unit 25.

[0074] The file transmission / reception control unit 37 performs tasks such as saving and retrieving handwritten input-derived data or non-handwritten input-derived data files to and from a storage medium, and communicating with external devices (print requests, display requests, etc.). The file transmission / reception control unit 37 receives a file transmission / reception execution request d64 from the handwritten input display control unit 23. When saving or printing a file, the handwritten input display control unit 23 sends a file transmission request to the file transmission / reception control unit 37, and the file transmission / reception control unit 37 sends a request to retrieve handwritten input saved data d61 to the handwritten input storage unit 25. • If the data is not derived from handwritten input, the handwritten input storage unit 25 transmits the data as it is, without altering the color information it contains. If the data originates from handwritten input, the handwritten input storage unit 25 sends the handwritten input storage data d62 converted to color to the file transmission / reception control unit 37 if the destination is a color-compatible device (e.g., a color printer). • In the case of handwritten input-derived data, the handwritten input storage unit 25 sends the handwritten input storage data d62 converted to black and white to the file transmission / reception control unit 37 if the destination is a black and white compatible device (e.g., a monochrome printer). Since black and white compatible devices may also be able to convert to grayscale, it may be possible to convert to color before transmission. • If the destination is a device that supports black and white enhancement, the handwritten input and saved data described later will be sent to the file transmission / reception control unit 37. Furthermore, when the handwriting input storage unit 25 saves handwriting input-derived data to a file, it converts the handwriting input storage data d62 to color according to the file format and attaches the handwriting input storage data as metadata to the file. When saving data that is not handwriting input-derived to a file, it converts the handwriting input storage data d62 to color according to the file format.

[0075] Whether a device is color-compatible or monochrome-compatible is stored, for example, in the MIB (Management Information Base) maintained by the network device, and the file transmission / reception control unit 37 can determine this by retrieving the MIB. Similarly, whether a device is monochrome-compatible can also be determined from the model name and other information published in the MIB.

[0076] When a file is read, the handwriting input display control unit 23 sends a file list acquisition request d65 to the file transmission / reception control unit 37. The file transmission / reception control unit 37 sends a file list acquisition request to an external device, acquires the file list d63, and sends it to the handwriting input display control unit 23. The handwriting input display control unit 23 displays the file list on the display unit 22, the handwriting input unit 21 sends the display position of the selected file to the handwriting input display control unit 23, and the handwriting input display control unit 23 sends a file reception request d66 to the file transmission / reception control unit 37. The file transmission / reception control unit 37 acquires the file from the external device and sends this file d67 to the handwriting input storage unit 25. The handwriting input storage unit 25 analyzes the file's metadata to determine whether it is handwriting input-derived data, and if it is, it extracts handwriting input storage data (data that can be converted to grayscale / highlighted, as described later). The file transmission / reception control unit 37 converts the handwriting input-derived data to grayscale highlighting display, and if it is not handwriting input-derived data, it sends it to the handwriting input display control unit 23 without conversion (it is displayed in grayscale). The handwriting input display control unit 23 transmits the display data of the handwritten data to the display unit 22.

[0077] Figure 6(b) is a functional block diagram showing the functions of the pen 2500 in a block-like manner. The pen 2500 has a pen event transmission unit 41. The pen event transmission unit 41 transmits event data such as pen up, pen down, and pen coordinates to the display device 2 with a pen ID attached.

[0078] <About predefined control data> Next, using Figure 7, we will explain the predefined control data used by the display device 2 for various processes. Figure 7 shows an example of predefined control data. The predefined control data in Figure 7 shows the control data for each control item.

[0079] The selectable candidate display timer 401 defines the time until selectable candidates are displayed. This is because the display device 2 does not display selectable candidates while handwriting is in progress. In Figure 7, it means that if a pen-down occurs within TimerValue = 500 [ms] from pen-up, selectable candidates will be displayed. The selectable candidate display timer 401 is held by the candidate display timer control unit 24. The selectable candidate display timer 401 is used when the selectable candidate display timer starts in step S18-2 of Figure 34, which will be described later.

[0080] The selectable candidate clearing timer 402 defines the time until the displayed selectable candidates are cleared. This is to clear the selectable candidates if the user does not select one. In Figure 7, it means that if a selectable candidate is not selected within TimerValue = 5000 [ms] from the time the selectable candidates are displayed, the selectable candidate display data will be cleared. The selectable candidate clearing timer 402 is held by the candidate display timer control unit 24. The selectable candidate clearing timer 402 is used when the selectable candidate display clearing timer starts in step S64 of Figure 36.

[0081] The handwritten data rectangular area 403 defines a rectangular area that is considered to be the vicinity of the handwritten data. In the example in Figure 7, the handwritten data rectangular area 403 is a rectangular area that is expanded horizontally by 50% of the estimated character size (Horizontal) and vertically by 80% of the estimated character size (Vertical). In the example in Figure 7, it is specified as a percentage of the estimated character size, but it is also possible to make it a fixed length by specifying the unit as "mm", etc. The handwritten data rectangular area 403 is held by the handwritten input storage unit 25. The font data 405 is used in step S10 of Figure 33 to determine the overlap between the handwritten data rectangular area and the stroke rectangular area.

[0082] The estimated writing direction / character size determination condition 404 defines constants for determining the measurement direction of writing direction and character size. In the example in Figure 7, • If the difference between the time when the first stroke was added and the time when the last stroke was added within the handwritten data rectangular area is MinTime = 1000 [ms] or more, • The difference between the horizontal distance (width) and vertical distance (height) of the handwritten data rectangular area is MinDiff = 10 [mm] or more. • If the horizontal distance is longer than the vertical distance, This means that the estimated writing direction is determined to be "horizontal" and the estimated character size is determined to be the vertical distance. If the horizontal distance is shorter than the vertical distance, this means that the estimated writing direction is determined to be "vertical" and the estimated character size is determined to be the horizontal distance. If none of the above conditions are met, the estimated writing direction is determined to be "horizontal" (DefaultDir="Horizontal") and the estimated character size is determined to be the longer of the horizontal distance and the vertical distance. The estimated writing direction / character size determination condition 404 is held by the handwriting input storage unit 25. The estimated writing direction / character size determination condition 404 is used for obtaining the estimated writing direction in step S59 of Figure 36 and for obtaining the string data font in step S81 of Figure 38.

[0083] Font data 405 defines data for estimating the size of characters, etc. In the example in Figure 7, the estimated character size determined by the estimated writing direction / character size determination condition 404 is compared with the smaller character 405a (hereinafter referred to as the minimum font size) and the larger character 405c (hereinafter referred to as the maximum font size) in font data 405. If the estimated character size is smaller than the minimum font size, the estimated character size is determined to be the minimum font size. If the estimated character size is larger than the maximum font size, the estimated character size is determined to be the maximum font size. Otherwise, it is determined to be the character size of the medium-sized character 405b. Font data 405 is held by the handwriting input storage unit 25. Font data 405 is used for acquiring the string data font in step S81 in Figure 38.

[0084] Specifically, the handwriting input storage unit 25 compares the estimated character size determined by the estimated writing direction / character size determination condition 404 with the FontSize of the font data 405 and uses the font of the closest size. For example, the handwriting input storage unit 25 determines that the estimated character size is 25 [mm] (FontSize for small characters) or less, and that it is "small characters". The handwriting input storage unit 25 determines that the estimated character size is greater than 25 mm but 50 mm (FontSize for medium characters) or less, and that it is "medium characters". The handwriting input storage unit 25 determines that the estimated character size is greater than 100 mm (FontSize for large characters), and that it is "large characters". For "small text" 405a, a 25mm Mincho font (FontStyle="Mincho" FontSize="25mm") is used; for "medium text" 405b, a 50mm Mincho font (FontStyle="Mincho" FontSize="50mm") is used; and for "large text" 405c, a 100mm Gothic font (FontStyle="Gothic" FontSize="100mm") is used. To address the need for more font size or style options, the display device provider can increase the number of font data types 405.

[0085] The crossover line determination condition 406 defines the data used to determine whether multiple display data items have been selected. This data is a single stroke. In the example in Figure 7, • The length of the longest side of the handwritten data is 100 mm or more (MinLenLongSide="100mm"), and, • The length of the shorter side is 50 [mm] or less (MaxLenShortSide="50mm"), and, • Overlap rate of handwritten data in the long and short directions is 80% or more (MinOverLapRate="80%") If there is display data, it is determined that multiple display data have been selected (selected data). The crossover line determination condition 406 is held by the operation command recognition control unit 32. The crossover line determination condition 406 is used for crossover line determination in the selection data determination in step S50 of Figure 35.

[0086] The border line determination condition 407 defines the data used to determine whether handwritten data is bordered or not. In the example in Figure 7, the operation command recognition control unit 32 determines confirmed data where the overlap rate between the long and short sides of the handwritten data is 100% or more (MinOverLapRate="100%") as selected data. The border line determination condition 407 is held by the operation command recognition control unit 32. The border line determination condition 407 is used for border line determination in the selection data determination in step S50 of Figure 35.

[0087] Note that the overlap line detection condition 406 and the enclosure line detection condition 407 can be judged in any order. For example, if the overlap line detection condition 406 is made lenient (making it easier to select overlap lines) and the enclosure line detection condition 407 is made strict (setting a value so that only enclosure lines can be selected), the operation command recognition control unit 32 should prioritize the enclosure line detection condition 407.

[0088] <Example of dictionary data> The dictionary data will be explained using Figures 8 to 10. Figure 8 is an example of the dictionary data of the handwriting recognition dictionary unit 27. Figure 9 is an example of the dictionary data of the string conversion dictionary unit 29. Figure 10 is an example of the dictionary data of the predictive text dictionary unit 31. These dictionary data are used in steps S33 to S41 of Figure 35, respectively.

[0089] In this embodiment, the conversion results from the dictionary data of the handwriting recognition dictionary unit 27 in Figure 8 are called language string candidates, the conversion results from the dictionary data of the string conversion dictionary unit 29 in Figure 9 are called conversion string candidates, and the conversion results from the dictionary data of the predictive conversion dictionary unit 31 in Figure 10 are called predictive string candidates. For each dictionary data, "Before Conversion" represents the string to search the dictionary data, "After Conversion" represents the converted string corresponding to the search string, and "Probability" represents the probability that the user will select it. The probability is calculated from the results of users selecting each string in the past. Therefore, the probability may be calculated for each user. Various algorithms have been devised for calculating the probability, but it is sufficient to calculate it using an appropriate method as appropriate, and the details are omitted. In this embodiment, string candidates are displayed in descending order of selection probability from the estimated writing direction.

[0090] In the dictionary data of the handwritten recognition dictionary unit 27 in FIG. 8, it shows that the handwritten "ぎ" is converted to "議" with a probability of 0.55 and "技" with a probability of 0.45, and the handwritten "ぎし" is converted to "技士" with a probability of 0.55 and "技師" with a probability of 0.45. The same applies to other "before conversion" character strings. In FIG. 8, the "before conversion" character string is handwritten hiragana, but other characters than hiragana may be registered as "before conversion".

[0091] In the dictionary data of the character string conversion dictionary unit 29 in FIG. 9, it shows that the character string "議" is converted to "議事録" with a probability of 0.95 and the character string "技" is converted to "技量試" with a probability of 0.85. The same applies to other "before conversion" character strings.

[0092] In the dictionary data of the prediction conversion dictionary unit 31 in FIG. 10, it shows that the character string "議事録" is converted to "議事録の送付先" with a probability of 0.65 and the character string "技量試" is converted to "技量試を決裁" with a probability of 0.75. In the example of FIG. 10, all the character strings before conversion are Chinese characters, but other characters than Chinese characters may be registered.

[0093] Note that the dictionary data has no language dependency, and any character strings may be registered before and after conversion.

[0094] <The operation command definition data held by the operation command definition unit> Next, the operation command definition data used by the operation command recognition control unit 32 will be described using FIG. 11. FIG. 11 shows an example of the operation command definition data and the system definition data held by the operation command definition unit 33.

[0095] Figure 11(a) shows an example of operation command definition data. The operation command definition data in Figure 11(a) is an example of operation command definition data when no selection data has been selected by handwritten data, and covers all operation commands that can be executed on the display device 2. The operation command in Figure 11(a) has an operation command name (Name), a string that partially matches the string candidate (String), and an operation command string to be executed (Command). "%~%" in the operation command string is a variable and is associated with system definition data as shown in Figure 11(b). In other words, "%~%" is replaced with the system definition data shown in Figure 11(b).

[0096] First, operation command definition data 701 indicates that the operation command name is "Load meeting minutes template", the string that partially matches the string candidate is "Meeting minutes" or "Template", and the operation command string to be executed is "ReadFile https: / / %username%:%password%@server.com / template / minutes.pdf". In this example, the operation command string to be executed contains the system definition data "%~%", and "%username%" and "%password%" are replaced by system definition data 704 and 705, respectively. Therefore, the final operation command string to be executed is "ReadFile https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf", which indicates that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf" will be read (ReadFile).

[0097] Operation command definition data 702 indicates that the operation command name is "Save to meeting minutes folder", the string that partially matches the string candidate is "Meeting minutes" or "Save", and the operation command string to be executed is "WriteFile https: / / %username%:%password%@server.com / minutes / %machinename%_%yyyy-mm-dd%.pdf". Similar to operation command definition data 701, "%username%", "%password%", and "%machinename%" in the operation command string will be replaced with system definition data 704-706, respectively. Note that "%yyyy-mm-dd%" will be replaced with the current date. For example, if the current date is September 26, 2018, it will be replaced with "2018-09-26". The final command to be executed is "WriteFile https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf", which indicates that the meeting minutes will be saved (WriteFile) to the file "https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf".

[0098] Operation command definition data 703 indicates that the operation command name is "Print", the string that partially matches the string candidate is "Print" or "Print", and the operation command string to be executed is "PrintFile https: / / %username%:%password%@server.com / print / %machinename%-"%yyyy-mm-dd%.pdf". If the operation command string is replaced in the same way as operation command definition data 702, the operation command to be executed will ultimately be "PrintFile https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf", indicating that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf" will be printed (PrintFile). In other words, the file will be sent to the server. When the user makes the printer communicate with the server and specifies the file, the printer will print the contents of the file onto the paper.

[0099] In this way, operation command definition data 701-703 can be identified from the string candidates, allowing the user to display the operation command by writing it manually. Furthermore, if user authentication is successful, the "%username%" and "%password%" in the operation command definition data will be replaced with user information, enabling file input and output associated with the user.

[0100] If user authentication is not performed (including cases where authentication fails but the user can still use display device 2), display device 2 will be replaced with the pre-configured "%username%", "%password%", etc. for display device 2. Therefore, file input and output can be performed in association with display device 2 even without user authentication.

[0101] Operation command definition data 709, 710, 711, 720, 721, 722, 723, 724, 725, and 726 are operation commands for changing the pen color. The pen color is the color of the handwritten data entered with the pen the user is using. The operation command names for operation command definition data 709, 710, 711, 720, 721, 722, 723, 724, 725, and 726 are "Black Pen", "Red Pen", "Blue Pen", "Green Pen", "Magenta Pen", "Cyan Pen", "Fine Black Pen", "Fine Red Pen", "Fine Blue Pen", and "Fine Green Pen". For example, in the case of "Black Pen" and "Fine Black Pen", if the string that partially matches the string candidate is "kuro" or "pen", and the user writes "kuro", only "Black Pen" and "Fine Black Pen" will be displayed as operation command candidates. On the other hand, "pen" is a string that partially matches the string candidates, such as "red pen," so if the user types "pen," the operation command candidates will display "black pen" to "cyan pen," "black fine pen" to "green fine pen," etc. When these operation commands are executed, as shown in Figure 15, the control data associated with the pen ID of the pen 2500 used by the user is updated, and the color of this pen ID is set as the ColorId of the pen ID control data (for example, Black in "Command="ChangePen Black"" becomes the ColorId).

[0102] The operation command definition data 719 has the operation command name "Read File", the strings that partially match the string candidates are "File", "Read", and "Read", and the operation command to be executed is "ReadFile https: / / %username%:%password%@server.com / files / ". When the handwriting input display control unit 23 replaces the system definition data in the same way as the operation command definition data 702, the operation command becomes "ReadFile https: / / taro.tokkyo:x2PDHTyS@server.com / files / ", indicating that the file at this address (folder) will be read (ReadFile). When this operation command is executed, the display device 2 displays a file selection window.

[0103] Operation command definition data 712 is an operation command for aligning the orientation of text data to a single direction. The operation command name of operation command definition data 712 is "Align Text Direction", the string that partially matches the string candidate is "Text", "Orientation", or "Direction", and the operation command string is "AlignTextDirection". Text data written by the user from directions other than vertical has various orientations, making it difficult to read all of it from a single direction. When the user executes operation command definition data 712, the display device 2 aligns the handwritten recognized strings to the same direction (for example, vertical direction). In this case, aligning means rotating the text data only by the angle information.

[0104] The operation command definition data 713 indicates that the operation command name is "Register handwritten signature", the strings that partially match the string candidates are "signature" and "register", and the operation command string is "RegistSignature". When the RegistSignature command is executed, a handwritten signature registration form is added to the handwritten input storage unit 25, and the handwritten signature registration form for the user to register their handwritten signature is displayed on the operation screen 101.

[0105] The operation command definition data 714 indicates that the operation command name is "Sign in with handwriting", the string to be partially positioned with the string candidate is "%signature%", and the operation command is "Signin". Here, "%signature%" is a reserved word in the system definition data and represents the fact that the registered handwritten signature data and stroke data such as the username match. In other words, if a match is found, the operation command 512 based on the operation command definition data 714 will be displayed in the operation guide 500 (see Figure 22).

[0106] When the Signin command is executed, the AccountId of the user with the matching SignatureId of the handwritten signature data is stored in the pen ID control data of the pen 2500, which has handwritten stroke data such as the username. This associates the pen ID with the AccountId, and the display device 2 can use the user-defined data identified by this AccountId (see Figure 15(a)).

[0107] The operation command definition data 715 indicates that the operation command name is "Sign out by handwriting", the string that partially matches the string candidate is "Sign" or "Out", and the operation command is "Signout". When the Signout command is executed, the AccountId is removed from the pen ID control data of the pen 2500 that performed the handwriting signout operation. This removes the association between the pen ID and the AccountId, allowing any user to use the pen 2500.

[0108] The operation command definition data 716 indicates that the operation command name is "Change Settings", the string that partially matches the string candidate is "Settings" or "Change", and the operation command is "ConfigSettings". When the ConfigSettings command is executed, a user-defined data change form is added to the handwriting input storage unit 25, and the user-defined data change form is displayed on the operation screen 101.

[0109] The command definition data for operations when handwritten data is present is omitted here, but when handwritten data is selected, the operation commands for when handwritten data is present are displayed (see Figure 20). In this case, the operation commands for when handwritten data is present are displayed when the already displayed handwritten data and the bounding rectangle of the stroke written by the user overlap.

[0110] <User-defined data> Next, user-defined data will be explained using Figure 12. Figure 12 shows an example of user-defined data held by the operation command definition unit 33. The user-defined data in Figure 12 is an example of definition data for one user. In user-defined data 717, AccountId is user identification information that is automatically assigned to each user, AccountUsername and AccountPassword are the username and password, SignatureId is identification information for handwritten signature data that is automatically assigned when handwritten signature data is registered, and username, password, and machinename are strings that are set in operation command definition data 701 to 703 in place of system definition data 704 to 706, respectively. This makes it possible to execute operation commands using user-defined data.

[0111] When a user signs in by handwriting their username, etc., the system utilizes the fact that the pen ID and AccountId are associated with the pen ID control data (see Figure 15(a)). The user-defined data string associated with the AccountId linked to the pen ID of the pen 2500 used by the user is used when executing operation commands. After the user signs out, even if the user uses the same pen 2500 that they used to sign in, the system-defined data string is used when executing operation commands.

[0112] User-defined data 718 is the data used in the user-defined data modification form. `Name` is the field name for AccountUsername, AccountPassword, username, password, or machinename in user-defined data 717, and `Data` is the changed value for AccountUsername, AccountPassword, username, password, or machinename. In this example, the data for "Name" is "%AccountName%", the data for "Password" is "%AccountPassword%", the data for "Folder Username" is "%username%", the data for "Folder Password" is "%password%", and the data for "Folder Filename" is "%machinename%", corresponding to each field in user-defined data 717. These fields entered in the user-defined data modification form are reflected in user-defined data 717.

[0113] It is also possible for users to register the color definition data described later in user-defined data 717, and users can input using the color definition data they have defined.

[0114] <Handwritten signature data> Next, we will explain the handwritten signature data using Figure 13. Figure 13 shows an example of handwritten signature data held by the handwritten signature data storage unit 39. The handwritten signature data has Data that represents the handwritten signature associated with SignatureId. SignatureId is identification information that is automatically assigned when the handwritten signature data is registered, and Data is data calculated by the handwritten signature authentication algorithm of the handwritten signature authentication control unit 38 from the stroke data received from the handwritten signature authentication control unit 38.

[0115] <Handwriting input data saved by the handwriting input storage unit> Next, we will explain the handwritten input saved data using Figure 14. Figure 14 shows an example of handwritten input saved data stored by the handwritten input saving unit 25. Each line in Figure 14 represents one stroke, one sentence (text), or one image.

[0116] When Type is Stroke, one handwritten input saved data has the following items: DataId, Type, PenId, ColorId, Angle, StartPoint, StartTime, EndPoint, EndTime, Point, and Pressure. DataId is the identification information for the stroke. Type is the type of handwritten input saved data. Types include Stroke, Group, Text, and Image. Handwritten input saved data 801 and 802 are of type Stroke. Handwritten input saved data 803 is of type Group. Handwritten input saved data 804, 805, and 806 are of type Text. Handwritten input saved data 807 is of type Image.

[0117] Group means grouping other strokes together. For handwriting input data of type Group, DataId specifies the strokes to be grouped. PenId, ColorId, and Angle are the pen ID control data described below. StartPoint is the coordinates of the starting point of the stroke, and StartTime is the time the stroke started. EndPoint is the coordinates of the ending point of the stroke, and EndTime is the time the stroke ended. Point is a sequence of coordinates from the starting point to the ending point, and Pressure is the pen pressure from the starting point to the ending point. As shown in Angle, handwriting input data 804 and 805 indicate that they were rotated clockwise by 180 degrees and 270 degrees, respectively, before being recognized as handwritten.

[0118] Additionally, handwritten input data 804, 805, and 806, whose Type is Text, contain FontName, FontSize, and Text. FontName is the font name of the text, FontSize is the font size, and Text is the content of the text (character code).

[0119] If the pen ID control data does not contain FontName, the handwriting input storage unit 25 copies the FontStyle from the font data 405 of the predefined control data shown in Figure 7 to the handwriting input storage data. The FontSize is copied from the font data 405 of the predefined control data shown in Figure 7.

[0120] The handwritten input data 801-805 should be attached as metadata when saving the file, as it is data originating from handwritten input. Therefore, the display device 2 can acquire the handwritten input data 801-805 when reading the file. When the display device 2 transmits the handwritten input data to an external device, it can convert the handwritten input data to color for color-compatible devices, or transmit it without conversion to color or after conversion for monochrome-compatible devices. When transmitting to a monochrome-enhanced device, it is sufficient to transmit the handwritten input data. Thus, the handwritten input data 801-805 supports both monochrome enhancement and color conversion.

[0121] While handwritten input saved data 801-805 are data originating from handwritten input, handwritten input saved data 806 and 807 are not. These are files read using the file read command. This can be determined by whether or not the ColorId contains a color value defined by the color definition data described later. For example, the ColorId of handwritten input saved data 801 is "Black", but the ColorId of handwritten input saved data 806 is "#e6001200". This is represented by # and 8 hexadecimal digits, with every two digits representing the values ​​of R (red), G (green), B (blue), and A (transparency).

[0122] Thus, in handwritten text data, the ColorId contains a string representing color information, while in text data not created by handwriting input, the ColorId is "#color value". In this way, whether or not text data was created by handwriting input is determined by focusing on the ColorId.

[0123] The handwriting input storage unit 25 stores the color of the text data of the file read by the file transmission / reception control unit 37 as ColorId="#e6001200". Therefore, the handwriting input storage data 806 does not support black and white emphasis, and only supports color conversion.

[0124] The handwriting input display control unit 23 determines whether the data is due to handwriting input using ColorId. If there is no data due to handwriting input, it calculates a grayscale value from RGBA and displays it. If the display device 2 is a color-compatible device, it displays RGBA as is.

[0125] The handwritten input saved data 807 is also an image file read from the storage medium using the file read command. When the Type is Image, the handwritten input saved data 807 has a FileId and a FileName. The FileId is the management number within the handwritten input saved data, and the FileName is the original file name.

[0126] In Figure 14, the ColorId value indicates whether or not the data originates from handwritten input, but a dedicated flag or similar indicator could also be used to show whether or not the data originates from handwritten input. Furthermore, although Type is limited to handwritten data of strokes, machine learning can also be used to determine whether or not the data originates from handwritten input based on its shape. In this case, the learning device learns the correspondence between handwritten input-derived data and character shapes using deep learning or similar methods, and outputs whether or not the shape of the input character originates from handwritten input.

[0127] Furthermore, handwriting input-derived data may include not only text data converted from handwritten data through character recognition, but also stamps displayed as fixed characters or marks such as "Completed" or "Secret," shapes such as circles and stars, straight lines, and other data converted based on user operations.

[0128] Handwritten input-generated data is used in steps such as step S7 (pen coordinates and reception time) in Figure 32, step S88 (acquisition of saved handwritten input data) in Figure 39, and step S96 (file saving) in Figure 39.

[0129] <Pen ID control data stored by the Pen ID control data storage unit> Next, we will explain the pen ID control data using Figure 15. Figure 15 is a diagram illustrating the pen ID control data 901 to 904 stored by the pen ID control data storage unit 36. The pen ID control data controls the color of handwritten input-induced data, etc. Figure 15 shows that there are four pens 2500. Four is just an example; one or more pens are acceptable.

[0130] Each row in Figure 15(a) represents the pen ID control data for one pen. Figure 15(b) is a diagram illustrating the angle information when a user writes on the display device 2. The angle information corresponds to the operating position and can be said to be the angle of the direction in which the user is located, the angle of the direction in which the pen is used, or the angle related to the rotation of the characters written by the user. With a predetermined direction of the display device 2 (e.g., up and down) as 0 degrees (reference), the angle information for each user is 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees, and 315 degrees counterclockwise.

[0131] Note that the angle information is not automatically determined by the user's position; each user sets the angle information. There are two methods for setting the angle information: inputting a straight line (described later) and selecting from the operation menu. The resolution of the angle information that can be specified by inputting a straight line (45 degrees in Figure 15) is just an example; smaller degrees such as 5 to 30 degrees are also acceptable. However, it is assumed that users can read characters rotated by around 45 degrees.

[0132] Pen ID control data 901-904 can contain PenId, ColorId, Angle, FontName (font specification), handwriting recognition candidate off state (RecommendMenuOff), and AccountId. All except PenId are optional. AccountId is associated with the pen used by the user to sign in. PenId is identification information stored inside the pen 2500. ColorId indicates which color the user using this pen 2500 selected (this can be changed by the user). The color ID from the color definition data described later is set to ColorId. The details of each color (thickness, etc.) are defined in the color definition data.

[0133] Angle is the stroke angle information set for this Pen 2500 (which can be changed by the user). In the example in Figure 15(a), the angle information for each pen is 0 degrees, 90 degrees, 180 degrees, and 270 degrees counterclockwise. AccountId is the user's identification information. By associating AccountId with Pen ID, the AccountId associated with the Pen ID of the Pen 2500 used by the user can be identified, and operation commands can be executed using user-defined data.

[0134] FontName is added when the user presses the pen color selection button in Figure 18. Once FontName is added, in step S81 of Figure 38, when acquiring the font for the string data, FontName is acquired instead of FontStyle, which is present in the font data 405 of the predefined control data shown in Figure 7. The string data is then displayed using the font of this FontName.

[0135] Pen ID control data 901 is a pen ID control data with PenId=1, color Black, angle information 0 degrees, AccountId=1, no FontName, and no RecommendMenuOff. The user with AccountId=1 is the user in user-defined data 717 in Figure 12. This indicates that this user signed in by handwriting their username etc. with the pen with PenId=1 and selected black. Pen ID control data without an AccountId indicates a signed-out state (not associated with a user). Similarly, pen ID control data 902 is a pen ID control data with PenId 2. The color is Red. The angle is 90 degrees. There is no AccountId. The FontName is Pop. There is no RecommendMenuOff.

[0136] The pen ID control data 903 is set to the handwriting recognition candidate off state (RecommendMenuOff). The handwriting recognition candidate off state (RecommendMenuOff) is control data that controls the operation guide 500 to either not display the string candidate 539 (displaying only operation commands), or not even display the operation command candidate 510. In the latter case, the operation guide 500 itself is not displayed. The handwriting recognition candidate off state (RecommendMenuOff) is set by pressing the handwriting recognition candidate on / off button (see Figure 24), which will be described later. When a user inputs handwritten data that does not require character recognition, such as shapes, some users may find the operation guide 500 to be a hindrance. In this embodiment, since the handwriting recognition candidate can be controlled to the off state, the operation guide 500 can be hidden, and the processing load can be reduced when the display device 2 does not recognize characters. The pen ID control data 903 has a PenID of 3. The color is blue. The angle is 180 degrees. There is no AccountId. There is no FontName.

[0137] Pen ID control data 904 indicates that PenId is 4, the color is green, the angle is 270 degrees, there is no AccountId, no FontName, and no RecommendMenuOff.

[0138] If RecommendMenuOff="True" is added to the pen ID control data, and a control that does not display operation commands is adopted, the operation commands will also not be executed. Therefore, in order to remove RecommendMenuOff="True", a button operation as shown in Figure 24 is required.

[0139] Furthermore, when the user presses the pen color selection button in Figure 18 or the button in Figure 24, the pen ID control data of pen 2500 is updated with the ColorId, Angle, FontName, and RecommendMenuOff defined for that button.

[0140] The pen ID control data is used in step S5 (acquisition of pen ID control data) in Figure 32, step S20 (storage of angle information of pen ID control data) in Figure 34, step S21 (acquisition of angle information of pen ID control data) in Figure 34, and step S60 (acquisition of pen ID control data) in Figure 36.

[0141] <Color definition data> Figure 16 shows an example of color definition data. Each line of color definition data defines one color. The color definition data defines the black and white highlighting method (how to display the black and white pattern, width (Pattern), and border (Edged)) and the color conversion method (how to display the color information (Color) and width (Width) on a color-compatible device) for a given ColorId. The color information is represented by # and an 8-digit hexadecimal number. Every two digits of the color information represent #R (red), G (green), B (blue), and A (transparency), and px represents the pixel width. Note that the color definition data is only applicable to data generated by handwritten input.

[0142] Color definition data 1001 is an example of a line type or color definition with ColorId "Black". For monochrome-emphasized devices that cannot display color, the color is represented by the line type. Pattern represents the contents of the stroke or text border. Edged indicates whether or not there is a border. Color represents the RGBA color information. Width represents the line width. Color definition data 1001 indicates that on monochrome-emphasized devices, handwritten input saved data will be displayed as solid black 5 pixels wide + no border, and on color-enabled devices, it will be displayed as black with 0% transparency + 5 pixels wide. Similarly, color definition data 1002 is an example of a line type or color definition with ColorId "Red". Color definition data 1002 indicates that on monochrome-emphasized devices, handwritten input saved data will be displayed as solid black 3 pixels wide + a black border 1 pixel wide from the outside + a white border 2 pixels wide, and on color-enabled devices, it will be displayed as the red of the color chart with 0% transparency + 5 pixels wide. Color definition data 1003 is an example of a line type or color definition with ColorId "Blue". It indicates that on a monochrome-enhanced device, it will be displayed as solid white 4 pixels wide + black border 1 pixel wide, and on a color-enabled device, it will be displayed as blue with 0% transparency + 5 pixels wide on the color chart. Color definition data 1004 is an example of a line type or color definition with ColorId "Green". Color definition data 1004 indicates that on a monochrome-enhanced device, handwritten input saved data will be displayed as a black dot pattern 4 pixels wide + black border 1 pixel wide, and on a color-enabled device, it will be displayed as green with 0% transparency + 5 pixels wide on the color chart. Color definition data 1005 is an example of a line type or color definition with ColorId "Magenta". Color definition data 1005 indicates that on a monochrome-enhanced device, it will be displayed as a black upper right diagonal line pattern with white transparent 5 pixels wide + no border, and on a color-enabled device, it will be displayed as magenta with 60% transparency on the color chart. Color definition data 1006 is an example definition of a line type or color with ColorId "Cyan". Color definition data 1006 indicates that on a monochrome-compatible device, it will be displayed as a black horizontal line pattern with white being transparent and 5 pixels wide, and on a color-compatible device, it will be displayed as cyan with 60% transparency on the color chart.

[0143] Color definition data 1007 defines a line type or color with ColorId "ThinBlack". Color definition data 1007 indicates that on a monochrome-enhanced device, it will be displayed as solid black 3 pixels wide + no border, and on a color-enabled device, as black with 0% transparency + 3 pixels wide. Color definition data 1008 defines a line type or color with ColorId "ThinRed". Color definition data 1008 indicates that on a monochrome-enhanced device, it will be displayed as solid black 1 pixel wide + black border 1 pixel wide from the outside + white border 1 pixel wide, and on a monochrome-enhanced device, as the red of the color chart with 0% transparency + 3 pixels wide. Color definition data 1009 defines a line type or color with ColorId "ThinkBlue". Color definition data 1009 indicates that on a monochrome-enhanced device, it will be displayed as solid white 2 pixels wide + black border 1 pixel wide, and on a color-enabled device, as the blue of the color chart with 0% transparency + 3 pixels wide. Color definition data 1010 defines a line type or color with ColorId "ThinGreen". Color definition data 1010 indicates that on monochrome-compatible devices, it will be displayed as a 2-pixel wide black dot pattern + 1-pixel black border, and on color-compatible devices, as a 3-pixel wide green with 0% transparency from the color chart.

[0144] Thus, the color definition data includes both grayscale-enhanced and color-convertible data. The color definition data is held by the handwriting input storage unit 25 and is used for converting the handwriting input storage data.

[0145] By defining color definition data in both system-defined data and user-defined data, and adding a form to change the color definition data in the user-defined data, users can change the color definition to their personal preference after signing in.

[0146] <Pen color selection button definition data> Next, we will explain how the user selects the pen color using Figures 17 and 18. Figure 17 is an example of pen color selection button definition data. The pen color selection button definition data is data that associates the pen button displayed on the display 220 with its ColorId. The pen color selection button definition data includes a pen button ID (PenButtonId), an icon file (Icon), a color ID (ColorId), and a font name (FontName).

[0147] Figure 18 shows an example of the pen color selection buttons 81-86 displayed on the screen. The pen color selection buttons 81-86 are displayed using the icon file mentioned above. The pen color selection buttons 81-86 are displayed from left to right in order of increasing pen button ID. When a user presses one of the pen color selection buttons 81-86 with the pen 2500, ColorId and FontName are added to the pen ID control data, and thereafter, handwritten data or text data entered with that pen 2500 is displayed using ColorId and FontName.

[0148] Pen color selection button definition data 1101 is the definition data for the leftmost pen color selection button 81. When the user presses the pen color selection button 81 with pen 2500, the stroke data written by that pen 2500 will have a ColorId of "Black", and the font of the text after handwriting recognition will be Mincho. Pen color selection button definition data 1102 is for the second pen color selection button 82 from the left, with a stroke data ColorId of "Red" and a text font of Pop after handwriting recognition. Similarly, pen color selection button definition data 1103 to 1106 define the display position, ColorId, and text font for pen color selection buttons 83 to 86.

[0149] Figure 18(a) shows an example of text font display highlighted in black and white based on the pen color selection button definition data. While this example is for illustrative purposes, it may be displayed when the user hovers over the pen 2500. Figure 18(b) shows an example of text font display in color on a color-compatible device based on the pen color selection button definition data (it is actually in color, but is shown in black and white for illustrative purposes).

[0150] As shown in Figure 18(a), by changing the font in addition to using black and white emphasis, black and white emphasis can be displayed more effectively.

[0151] The pen color selection button definition data can also be defined in system-defined data and user-defined data. By adding a form to change the pen color selection button definition data in the user-defined data, users can change the pen color selection button to their preference after handwritten signature authentication. Therefore, the correspondence between ColorId and font is just one example.

[0152] The pen color selection button is held by the handwriting input display control unit 23 and is used in step S6 (coordinate interpolation display data) in Figure 32, step S82 (string data display data) in Figure 38, and step S100 (display data) in Figure 39.

[0153] <Example of displaying selectable options> Figure 19 shows an example of an operation guide and the selectable candidates 530 displayed by the operation guide. When the user writes handwritten data 504 (due to a timeout of the selectable candidate display timer), the operation guide 500 is displayed. The operation guide 500 has an operation header 520, operation command candidates 510, handwritten recognition string candidates 506, conversion string candidates 507, string / predictive conversion candidates 508, and a handwritten data rectangular area display 503. The selectable candidates 530 are the operation command candidates 510, handwritten recognition string candidates 506, conversion string candidates 507, and string / predictive conversion candidates 508. In this example, there may be cases where no language conversion string is displayed. Also, the selectable candidates 530 excluding the operation command candidates 510 are called string candidates 539.

[0154] The operation header 520 has buttons 501, 509, 502, and 505. Button 501 accepts switching operations between predictive text and "Kana conversion". In the example in Figure 19, when the user presses button 509 which is displayed as "Predictive", the handwriting input unit 21 receives the input and notifies the handwriting input display control unit 23, and the display unit 22 changes the display to button 509 which is "Kana". After the handwriting data 504 is recognized, the string candidates 539 are arranged in descending order of probability of "Kana conversion".

[0155] Button 502 controls the page of the candidate display. In the example in Figure 19, there are three candidate display pages, and currently the first page is displayed. Button 505 accepts the deletion of the operation guide 500. When the user presses button 505, the handwriting input unit 21 receives the notification and notifies the handwriting input display control unit 23, and the display unit 22 deletes all displays except the handwritten data. Button 509 accepts the deletion of all displays at once. When the user presses button 509, the handwriting input unit 21 receives the notification. The handwriting input unit 21 notifies the handwriting input display control unit 23. The display unit 22 deletes all displays shown in Figure 19, including the handwritten data, allowing the user to start handwriting again from the beginning.

[0156] The handwritten data 504 is the character "ぎ" written by the user. A handwritten data rectangular area display 503 is displayed surrounding the handwritten data 504. The display procedure is explained in the sequence diagrams in Figures 32 to 39. In the example in Figure 19, the handwritten data rectangular area display 503 is displayed with a dotted line frame.

[0157] The handwritten recognition string candidates 506, the converted string candidates 507, and the string / predictive conversion candidates 508 are sorted in descending order of probability. The handwritten recognition string candidate 506, "ぎ", is a candidate for the recognition result. In this example, the display device 2 correctly recognizes "ぎ".

[0158] The conversion string candidate 507 is a conversion string candidate converted from a language string candidate. In this example, "skill test" is an abbreviation for "mass production technology trial production". The string / predicted conversion candidate 508 is a predicted string candidate converted from a language string candidate or a conversion string candidate. In this example, "approve the skill test" and "destination of the minutes" are displayed.

[0159] The operation command candidate 510 is a candidate for the operation command selected based on the operation command definition data 701 to 703, 709 to 716 in Fig. 11(a). In the example of Fig. 19, the leading character "》" 511 indicates that it is a candidate for the operation command. In Fig. 19, there is no selection data for the handwritten data 504 "ぎ", and the string candidate "minutes" of "ぎ" partially matches the operation command definition data 701 and 702 in Fig. 11(a), so it is displayed as the operation command candidate 510

[0160] When the user selects "Load the minutes template", the operation command defined in the operation command definition data 701 is executed. When the user selects "Save to the minutes folder", the operation command defined in the operation command definition data �02 is executed. Thus, the operation command candidate is displayed when operation command definition data containing the converted string is found, so it is not always displayed.

[0161] As shown in Fig. 19, since the string candidate and the operation command candidate are displayed simultaneously (both), the user can arbitrarily select either the string candidate or the operation command that the user intends to input.

[0162] <Example of specifying selection data> In the display device 2 of this embodiment, the user can specify the selection data by selecting the confirmed data by handwriting. The selection data becomes the object of editing or decoration, or the page name.

[0163] <Display example of operation command candidate> Figure 20 shows an example of displaying candidate operation commands based on operation command definition data when handwritten data is available. Figure 20(a) shows candidate editing operation commands, and Figure 20(b) shows candidate modification operation commands. Figure 20(a) also shows an example where selected data is specified in the handwritten data 11a.

[0164] As shown in Figures 20(a) and 20(b), the candidate operation commands displayed following the bullet point "》" 511 constitute the main menu 550. The main menu 550 displays the name of the last executed operation command or the first operation command name in the operation command definition data. The bullet point "》" 511a on the first line is a candidate for an editing operation command, and the bullet point "》" 511b on the second line is a candidate for a modification operation command. The bullet point "》" 511c on the third line is a candidate for the "Set as page name" operation command.

[0165] The ">" 512 at the end of a line indicates the presence of a submenu (an example of a submenu button). The ">" 512a on the first line displays the submenu (the last selected one) as a candidate for editing operation commands. The ">" 512b on the second line displays the remaining submenus as candidates for modification operation commands. When the user presses the ">" 512, submenu 560 is displayed to its right. Submenu 560 displays all operation commands defined in the operation command definition data. In the display example in Figure 20(a), submenu 560 corresponding to the ">" 512a on the first line is displayed from the time the main menu is displayed. The display device 2 may also display the submenu when the ">" 512a on the first line is pressed.

[0166] Regarding the operation command for setting the page name on the third line, the operation command candidates are not grouped, so the ">" is not displayed at the end of the line.

[0167] When a user presses any of the operation command names with the pen, the handwriting input display control unit 23 executes the Command from the operation command definition data associated with the operation command name for the selected data. Specifically, if "Erase" 521 is selected, "Delete" is executed; if "Move" 522 is selected, "Move" is executed; if "Rotate" 523 is selected, "Rotate" is executed; and if "Select" 524 is selected, "Select" is executed.

[0168] For example, a user can erase the "Meeting Minutes" by pressing "Delete" 521 with the pen. When a user presses "Move" 522, "Rotate" 523, or "Select" 524, a bounding box (a rectangle enclosing the selected data) is displayed. If "Move" 522 or "Rotate" 523 is pressed, the user can move or rotate the "Meeting Minutes" by dragging with the pen. If "Select" 524 is pressed, the user can perform other bounding box operations.

[0169] In Figure 20, the string candidates other than the operation command candidates, "○" 551, "∞" 552, "0" 553, and "00" 554, are the recognition results of the enclosed line (handwritten data 11b). If the user intended to enter a string rather than an operation command, they can select a string candidate.

[0170] In Figure 20(b), pressing the ">" 512b on the second line displays the submenu shown in Figure 20(b). The display example in Figure 20(b) also shows the main menu 550 and submenu 560, similar to Figure 20(a). When "Thick" 531 is selected, the handwriting input display control unit 23 executes "Thick" on the selected data. When "Thin" 532 is selected, the handwriting input display control unit 23 executes "Thin" on the selected data. When "Large" 533 is selected, the handwriting input display control unit 23 executes "Large" on the selected data. When "Small" 534 is selected, the handwriting input display control unit 23 executes "Small" on the selected data. When "Underline" 535 is selected, the handwriting input display control unit 23 executes "Underline" on the selected data.

[0171] <Example of angle information input> Next, we will explain how to input angle information using Figure 21. Figure 21 is an example of a diagram illustrating how to input angle information. Figure 21 explains the case where a user located at the 3 o'clock position of the display device 2 inputs angle information. Since handwritten characters etc. from the 3 o'clock position are correctly recognized when rotated 90 degrees clockwise, it is best to input angle information of 90 degrees.

[0172] Figure 21(a) shows the state where the operation guide 500 is displayed because the user, who is located at the 3 o'clock position on the display device 2, has handwritten the character "ぎ" while the angle information of the pen ID control data is 0 degrees (initial value). Because the display device 2 recognizes the handwritten character "ぎ" from the 3 o'clock position with the angle information still at 0 degrees, it displays a different selection candidate 530 than expected.

[0173] When inputting angle information, the user draws a straight line by hand in the operation guide 500, from top to bottom as viewed from the user's perspective. Figure 21(b) shows an example of this straight line 521. The angle information is the angle α formed counterclockwise between the 6 o'clock direction (where the angle information is 0 degrees) and this straight line 521. In other words, the angle information is the angle α formed counterclockwise between the straight line 522 drawn from the starting point S towards the 6 o'clock direction and the straight line 521 entered by the user. Simply put, the direction of the endpoint of the straight line 521 is the angle information. Therefore, the angle information entered by the user in Figure 21(b) is 90 degrees.

[0174] One method for detecting straight lines is to convert the coordinates from a starting point S to an ending point E into a straight line using the least squares method, and then compare the resulting correlation coefficient with a threshold to determine whether or not it is a straight line.

[0175] Immediately after the user starts writing the straight line 521 (immediately after the pen 2500 touches the starting point S of the straight line 521), the display device 2 erases the operation guide 500. Also, immediately after finishing writing the straight line 521 (immediately after the pen 2500 leaves the end point E of the straight line 521), the display device 2 searches for the value closest to the included angle α among 45 degrees, 90 degrees, 135 degrees, 180 degrees, 215 degrees, 270 degrees, 315 degrees, 360 degrees and determines it as the angle information. The included angle α itself may also be the angle information. The determined angle information is set in the Angle of the pen ID control data. Since the pen event transmission unit 41 of the pen 2500 transmits the pen ID to the display device 2 when the pen tip is pressed for handwriting or the like, the display device 2 can associate the angle information with the pen ID control data.

[0176] Note that the user can input angle information by handwriting a straight line only within the operation guide 500. As a result, when the user handwrites a straight line outside the operation guide 500, it is recognized as "1" or "一", etc., and when the user handwrites a straight line within the operation guide 500, the angle information can be input. That is, the handwriting recognition control unit 26 detects a straight line from a predetermined range and converts the stroke data handwritten outside the predetermined range into text data.

[0177] FIG. 21(c) shows the operation guide 500 immediately after the operation in FIG. 21(b). Since 90 degrees is set as the angle information (Angle) in the pen ID control data, the handwriting data (stroke data) is rotated 90 degrees clockwise internally for handwriting recognition, and the operation guide 500 is rotated 90 degrees counterclockwise and displayed. Note that the angle information may be manually input by the user from the menu.

[0178] [[ID=1l]] <An example of signing in by handwriting> Next, using Figure 22, we will explain how a user signs in after registering handwritten signature data. Figure 22 is an example of the operation guide 500 displayed when a user writes "Suzuki," which is the handwritten signature data they registered. Since "Suzuki" is registered as handwritten signature data in the operation command definition unit 33, "Suzuki" matches the handwritten signature data. For this reason, the operation command 512 "Sign in by handwriting" is displayed.

[0179] Furthermore, since the handwritten signature data matched, a SignatureId representing "Suzuki" was identified, and user-defined data with an AccountId associated with this SignatureId was identified.

[0180] When the user selects the operation command 512, "Sign in by handwriting," the AccountId of "Suzuki" is associated with the Pen ID of the pen 2500 being used and added to the Pen ID control data, and the user-defined data of "Suzuki" is then used when the operation command is used.

[0181] <Example of operation menu display> Next, using Figure 23A, we will explain the operation menu that contains information regarding data processing according to the display position. Figure 23A is a top view of the display device 2 when placed flat. In Figure 23A, there are users 250 and 251 on the upper and lower sides of the diagram, respectively, and two users can operate operation menus 301 and 302 simultaneously from two directions, the lower and upper sides. However, it is possible to operate with just one user. The display device 2 displays the operation menu according to the operation button definition data. The number of operation menus will be the same as the number of expected users.

[0182] Each operation menu 301, 302 contains information related to data processing according to the display position of the operation menu. In this embodiment, operation menus 301, 302 have a rotation angle according to the display position of the operation menu. The design (appearance of the icons) of operation menus 301, 302 is the same, but they contain information related to data processing according to the display position of the operation menu.

[0183] One operation menu 301, 302 has a pen operation panel 2001, 2008 for the user to select a pen type and perform pen operations. Another operation menu 301, 302 has a page operation panel 2002, 2009 for the user to perform page operations. Another operation menu 301, 302 has a page navigation operation panel 2003, 2010 for the user to operate a list of file names or pages. Another operation menu 301, 302 has a page navigation 2004, 2011 for switching pages from a list of file names or page names. To make them easier for the user to use, the lower user operation menu 301 is positioned at the bottom of the screen (along the bottom edge of the display 220), and the upper user operation menu is positioned at the top of the screen (along the top edge of the display 220).

[0184] The user-facing operation menu 301 at the bottom includes a pen operation panel 2001, a page operation panel 2002, a page navigation operation panel 2003, and a page navigation 2004. The page navigation 2004 extends from top to bottom within the page navigation window area 2005. The page navigation window area 2005 is sometimes simply referred to as the page navigation window.

[0185] Similarly, the user operation menu 302 for the upper user includes a pen operation panel 2008, a page operation panel 2009, a page navigation operation panel 2010, and a page navigation 2011. The page navigation 2011 extends from top to bottom as viewed from the upper user within the page navigation window area 2012.

[0186] Near the center of display 220, handwritten data and text data are displayed. Handwritten data 2006 was written by the lower user with a black pen (solid black). Text data 2007 is a string of text written by the lower user with a black pen and recognized. Handwritten data 2013 is handwritten data written by the upper user with a blue pen (solid white with a black outline).

[0187] The pen operation panels 2001 and 2008, page operation panels 2002 and 2009, page navigation operation panels 2003 and 2010, and page navigation panels 2004 and 2011 all have two versions: one for the lower user and one for the upper user, but they are included in operation menu 302. The icons (buttons), file names, and page names for the upper user are displayed rotated 180 degrees. This rotation angle is pre-set in the operation button definition data (described later).

[0188] When a user presses any of the button areas in operation menus 301 and 302, or any file name or page name in page navigation 2004 and 2011 with the pen 2500, the display device 2 saves the rotation angle set in the operation button definition data to the pen ID control data of the pen 2500. This rotation angle becomes information related to data processing.

[0189] <<English expression for Figure 23A>> Display device 2 also supports languages ​​other than Japanese. Here, as an example, Figure 23A describes how display device 2 is used in an English environment. The configuration of display device 2 is the same as when used in Japanese, except that the conversion dictionary and various definition data are compatible with English.

[0190] Figure 23B is a top view of the display device 2 when it is placed flat and converting handwritten data to English. Even when the handwritten data is converted to English, the display device 2 displays operation menus 301 and 302. In Figure 23B, there are users 250 and 251 on the top and bottom of the diagram, respectively, and two users can operate operation menus 301 and 302 simultaneously from two directions, the bottom and the top. The display device 2 displays the operation menu according to the operation button definition data. The number of operation menus will be the same as the number of expected users.

[0191] Furthermore, in operation menus 301 and 302, the pen operation panels 2001 and 2008, page operation panels 2002 and 2009, page navigation operation panels 2003 and 2010, and page navigation panels 2004 and 2011 are displayed in English.

[0192] The content handwritten on display 3 depends on what the user writes, but in Figure 23B, it is handwritten in English. Handwritten English data and string data are displayed near the center of display 220. Handwritten data 2006 was written by the lower user with a black pen (solid black). String data 2007 is a string written by the lower user with a black pen and recognized. Handwritten data 2013 is handwritten data written by the upper user with a blue pen (solid white + black outline).

[0193] Although this embodiment uses English as an example, the display device 2 can also support languages ​​other than English (such as Chinese, Hindi, Spanish, Arabic, Portuguese, Russian, etc.).

[0194] <<Pen Control Panel>> Figure 24 is a magnified view of the pen operation panels 2001 and 2008. The pen operation panels 2001 and 2008 have buttons for the user to select colors, etc. Buttons 2101 to 2108 represent opaque black, red, blue, and green fine or broad pens, while buttons 2109 and 2110 represent magenta and cyan broad pens as transparent colors.

[0195] Buttons 2111 and 2112 represent undo / redo. Button 2113 represents page clear, and button 2114 represents page sweep (a button that clears and redisplays a degraded screen in high-speed drawing mode). Button 2115 accepts the on / off setting for the display of string candidates 539 (a button that accepts the setting for the handwriting recognition candidate off state (RecommendMenuOff) of the pen ID control data).

[0196] As shown in the operation button definition data in Figures 27 to 30, each button has a defined button identifier (ButtonId), button display position (Position), button icon image (Icon), button rotation angle (Angle), and command to be executed by the display device 2 when pressed (Command). When the user presses buttons 2101 to 2115 with the pen 2500, the display device 2 saves the rotation angle defined for that button in the pen ID control data of the pen 2500 and executes the command defined for that button. Note that buttons 2101 to 2115 correspond to ButtonId=1 to 15 in the operation button definition data, from left to right.

[0197] The commands for the operation button definition data 2401-2410 corresponding to buttons 2101-2110 are the same as the ChangePen command in the operation command definition data 709-711 and 720-726 in Figure 11, with the ColorId from the color definition data specified. The display unit 22 displays a frame 260 around the button of the last selected pen to indicate that it has been selected. In Figure 24, the black fine pen of button 2101 is selected.

[0198] The operation button definition data 2411 and 2412 corresponding to buttons 2111 and 2112 define the Undo / Redo commands. Buttons 2111 and 2112 (Undo / Redo) accept the execution of the Undo / Redo commands, respectively. Undo is a command that returns to the previous operation state, and Redo is a command that advances the operation state one step forward from the undone state.

[0199] The operation button definition data 2413 corresponding to button 2113 defines the command for EragePage. Button 2113 (Erase Page) accepts the execution of the EragePage command. EragePage erases all data currently entered on the page and redisplays it.

[0200] The operation button definition data 2414 corresponding to button 2114 defines the SweepPage command. Button 2114 (sweep button) accepts the execution of the SweepPage command. On display devices such as e-paper, residual images may remain in high-speed drawing mode. The display device clears these by redisplaying them.

[0201] The operation button definition data 2415 corresponding to button 2115 defines the ToggleRecommendMenu command. Button 2115 (handwriting recognition candidate on / off) accepts the execution of the ToggleRecommendMenu command. ToggleRecommendMenu switches the display of the string candidates 539 obtained by handwriting recognition on and off. As a result, if it is turned off, RecommendMenuOff="True" is added to the pen ID control data of the pressed pen 2500. When the display of string candidates 539 is on, all string candidates 539 are displayed, but when it is off, only the operation command is displayed. If there is no operation command, the entire range of selectable candidates is not displayed, allowing the user to concentrate on handwriting. When the display device 2 executes ToggleRecommendMenu, it returns the current on / off state, and switches the icon according to that state.

[0202] Note that buttons 2101-2110 define separate buttons from the pen color selection buttons 81-86 in Figure 18. Since ColorId may be set redundantly by the pen color selection buttons 81-86, in that case, the result of pressing buttons 2101-2110 takes precedence and is set as the pen ID control data.

[0203] <<Page Control Panel>> Figure 25 shows an example of the Page Operation Panel 2002,2009. The Page Operation Panel 2002,2009 is an operation menu related to page manipulation. Button 2201 is a button that allows the user to rotate the currently displayed page by 90 degrees. Buttons 2202 and 2204 are buttons that allow the user to switch to the previous page or the next page, respectively. Current page 2203 shows the current page number (numerator) and the total number of pages (denominator). Button 2205 is a button that allows the user to open or close the page navigation window. File name 2206 shows the file name of the currently displayed file. Buttons 2201, 2202, 2203, and 2205 are defined in the operation button definition data 2416~2419 and 2440~2443.

[0204] Button 2201 (Page Rotation Display) accepts the execution of the RotatePage command. RotatePage rotates only the handwritten data 2006 and text data 2007 of the lower user, and the handwritten data 2013 of the upper user, by 90 degrees counterclockwise, as shown in Figure 30. The operation menu, including the page navigation, does not rotate. In other words, in the example in Figure 30, the handwritten data 2013 of the upper user is upside down and difficult to read from the perspective of the lower user, so the display device 2 temporarily rotates the page to make it easier to read by executing the RotatePage command.

[0205] Buttons 2202 and 2204 (page switching) accept the execution of the PageBack or PageNext command. PageBack or PageNext switches the current page to display the previous page or the next page.

[0206] Button 2205 (open or close the page navigation window) accepts the execution of the TogglePageNavi command. The TogglePageNavi command opens or closes the page navigation window areas 2005 and 2012 in Figure 30.

[0207] Page 2203 and file name 2206 are display-only and do not accept any interaction. Page 2203 displays the current page number (numerator) and the total number of pages (denominator). In this example, display device 2 is displaying page 2 of a 4-page file. File name 2206 displays the file name of the currently displayed file. In this example, display device 2 is displaying "Technical Review Meeting.pdf".

[0208] <<Page Navigation Control Panel>> Figure 26 shows an example of the page navigation control panel for 2003 and 2010. The page navigation control panel for 2003 and 2010 displays page names in page order for each file. In other words, if page names are selected from string data, it is easier for the user to later identify what is written on each page.

[0209] Buttons 2301 and 2305 are for the user to switch to the previous page navigation window or the next page navigation window, respectively. Buttons 2302 and 2304 are for the user to switch to the previous file or the next file, respectively (the first page of the file after switching is displayed on display 220). Current page navigation 2303 displays the current number (numerator) and total number (denominator) of the page navigation window. Buttons 2301, 2302, 2304, and 2305 are defined in operation button definition data 2420~2423 and 2444~2447.

[0210] Page Navigation 2306 displays strings in file order and page order. Page Navigation 2306 displays the filenames of multiple loaded files and the names of each page within those files in a tree structure. When a filename or page name is pressed, the display device 2 displays that page on display 220 based on the page navigation data in Figure 38. For example, if the page navigation has 200 rows and the page navigation window has 50 rows, the current number of the current page 2203 can take values ​​from 1 to 4, with a total of 4. When the user presses button 2301, the display device 2 decrements the current number, and when the user presses button 2305, it increments the current number and redisplays the page navigation window. For ease of use, the lower user portion of Page Navigation 2306 is positioned at the bottom of display 220, and the upper user portion is positioned at the top of display 220.

[0211] In Figure 26, page navigation 2306 displays two filenames 2310 and 2308, with the page names 2307 and 2309 for each file displayed in a tree structure. The page names are either string data specified by the user or automatically extracted strings. The page names are displayed indented according to user input.

[0212] Buttons 2301 and 2305 (page navigation window switching) accept the execution of the PageNaviWinBack / PageNaviWinNext command. PageNaviWinBack / PageNaviWinNext switches the current page navigation window to the previous or next page navigation window. If the number of lines that can be displayed in the page navigation window is set to 50 lines, pressing buttons 2301 and 2305 will allow display device 2 to switch and display page navigation windows with 50 lines of text.

[0213] Buttons 2302 and 2304 (file switching) accept the execution of the PageNaviFileBack / PageNaviFileNext command. PageNaviFileBack / PageNaviFileNext switches the currently displayed page to the beginning of the previous or next file.

[0214] Page navigation 2306 is defined by the page navigation data in Figure 31. The tree opening icon 2311 is a downward-pointing triangle, indicating that the tree is currently displayed. The tree closing icon 2312 is a sideways-pointing triangle, indicating that the tree is closing. When the tree is closed, the page names of rows below the row containing the tree are no longer displayed.

[0215] If the Tree in the page navigation data is "Opened," display device 2 displays a tree opening icon 2311. When pressed, it changes the Tree to "Closed" and closes the tree. Conversely, if Tree="Closed," display device 2 displays a tree closing icon 2312. When the user presses it, it changes the Tree to "Opened" and opens the tree.

[0216] Page name 2313 indicates an indented page. The indentation level is controlled by the Level in the page navigation data. File names are initially Level="0", and page names are initially Level="1". When the user presses a file name or page name to indent to the right, the Level value of the indentation level increases. Page name 2313 indicates an indentation of Level="2". To change the Level, the user long-presses the page name and moves it left or right. For example, when the user moves to the right, the display device 2 searches for a line with the same Level (the Level before the move) above the current line and adds Tree="Opened" to the line above. It becomes possible to hide indented lines, and the display or hiding of indented lines is controlled by the Tree of the line that was at the same Level as the line before indentation. The display device 2 also increments the Level of the indented line and redisplays it.

[0217] Page name 2314 indicates a blank page, representing the blank page immediately following the last page of the file (page name 2313). A blank page means that nothing has been written on it yet. Blank pages continue up to the maximum page number, but only the first blank page is displayed as a "blank page" in the page navigation. For example, if the maximum page number is 100 and the file has 4 pages, pages 5 through 100 are blank pages, but only the first page (page 5) will be displayed as a "blank page" in page navigation 2306.

[0218] Page 2315 is the page currently displayed on display 220. Here, page 2315 is highlighted by being enclosed in frame 262.

[0219] Page name 2316 is the default page name. If there is no handwritten text recognized, the display device 2 will add "nnnn page" (where nnnn is an integer) to the page name and display it like this. This indicates that there is handwritten text recognized for all page names except the default page name 2316, the blank page 2314, and file names 2310 and 2308.

[0220] In the page navigation data described later, if AutoName="True", the display device 2 automatically sets the page name if there is a handwritten string that has been recognized. In the example screen layout in Figure 30, the string "Hardware Configuration" is automatically set as the page name. Various logics can be considered for determining the page name from a handwritten string. For example, one method is to concatenate the strings in the order in which they were entered and extract a fixed number of characters from the beginning, or one method is to determine the priority of the positions of the strings, concatenate the strings in order from the highest priority, and extract a fixed number of characters from the beginning.

[0221] As shown in the example of the operation command display in Figure 20, when the user selects confirmed data that has already been recognized by handwriting using a border or crossover line, the operation command "Set as page name" will appear. When this is selected, the SetPageName command is executed. SetPageName changes the current page name of page navigation 2306 to the selected string and changes the AutoName of the page navigation data to "False".

[0222] Even when a file name or page name in the Page Navigation Operation Panel 2003 or 2010 is selected with Pen 2500, "Angle" is set in the Pen ID control data. This Angle is the angle at which the Page Navigation window area is rotated. Display device 2 saves the angle (Angle) of the Page Navigation window area obtained from the operation button definition data for display purposes, and when a file name or page name is selected with Pen 2500, it sets the stored rotation angle to "Angle" in the Pen ID control data. From then on, data processing is performed using the Angle set in the Pen ID control data.

[0223] <Operation Button Definition Data> Figures 27 to 30 show examples of operation button definition data 2401 to 2448. Operation button definition data 2401 to 2448 defines the buttons for the pen operation panels 2001 and 2008, the page operation panels 2002 and 2009, and the page navigation operation panels 2003 and 2010. Each line of operation button definition data defines one button. Figures 27 and 28 define the operation menu 301 for the lower user, and Figures 29 and 30 define the operation menu 302 for the upper user.

[0224] Each operation button definition data 2401-2448 defines the button identifier (ButtonId), the button's display position (Position), the button's icon image (Icon), the angle (Angle) which is related to data processing, and the command that the display device 2 executes when the button is pressed (Command). The angle (Angle) also represents the user's operating position and the rotation angle of the operation menu 302. When the user presses buttons 2101-2115, 2201, 2202, 2203, 2205, 2301, 2302, 2304, or 2305 with the pen 2500, the display device 2 saves the angle (Angle) defined for that button to the pen's pen ID control data and executes the command defined for that button.

[0225] In this way, the operation menu is associated with data processing that corresponds to the display position of the operation menu. Figures 27 to 30 show examples where string data rotates according to the display position. By changing the Commands of the same buttons in operation menu 301 for the lower user and operation menu 302 for the upper user, the administrator can perform various data processing tasks.

[0226] Operation button definition data 2401~2415 defines the pen operation panel 2001 for the lower user. Operation button definition data 2416~2419 defines the page operation panel 2002 for the lower user. Operation button definition data 2420~2424 defines the page navigation operation panel 2003 for the lower user. Operation button definition data 2425~2439 defines the pen operation panel 2008 for the upper user. Operation button definition data 2440~2443 defines the page operation panel 2009 for the upper user. Operation button definition data 2444~2448 defines the page navigation operation panel 2010 for the upper user.

[0227] Operation button definition data 2415 and 2439 need to indicate the current on / off state of the handwriting recognition candidate, so icon images for on (IconOn) and off (IconOff) are defined. Operation button definition data 2419 and 2443 need to represent buttons to open or close the page navigation window, so icon images for on (IconOn) and off (IconOff) are defined. Display device 2 looks at the return value of the command execution result and displays the on / off icon image.

[0228] The operation button definition data 2424 and 2448 are definition data for the page navigation window area, defining only the position and angle of the rectangular area. When the user presses these rectangular areas with the pen 2500, the display device 2 stores the angle defined therein in the pen ID control data of the pen 2500 used for the press.

[0229] <Page navigation data> Figure 31 shows an example of page navigation data. Page navigation data is an example of information where the page name is recorded for each page of a file. Page navigation data defines the display content of the page navigation window area in 2005 and 2012. Each line of page navigation data defines one line of page navigation. Page navigation data defines the file name (FileName), page number (Page), indentation level (Level), page name (PageName), autoname (AutoName), tree state (Tree), and current page (CurrentPage). Page="0" and Level="0" is the file name. As the Level increases, the page name on the page navigation indents to the right.

[0230] The default settings are PageName="nnnn page" (where nnnn is the page number) or "blank page", AutoName="True", and Tree="Opened". When AutoName="True", display device 2 automatically sets the handwritten recognized string to PageName. If the user specifies confirmed data that has been handwritten and selects "Set as page name" from the operation command, the page name is changed to that confirmed data and AutoName="False". In other words, the page name set by the user takes precedence. "Tree" is set to display all page names in rows below the current row and at a level one level lower than the current row in a tree view when Tree="Opened", and not display in a tree view when Tree="Closed" (only the page name of the current row is displayed). If a page name is displayed, the same process is repeated for the displayed page name. Therefore, if there is a higher level in the next row, display device 2 sets the Tree of that row to "Opened". CurrentPage indicates the page currently displayed on display 220.

[0231] <Operating Procedure> The operation of the display device 2 will be explained using the above configuration and Figures 32 to 39. Figures 32 to 39 are sequence diagrams illustrating an example of the process by which the display device 2 displays string candidates and operation command candidates. The process in Figure 32 starts when the display device 2 is started (when the application is started). Note that, for space reasons, the functions in Figure 6 are indicated by symbols in Figures 32 to 39.

[0232] Before starting to input handwritten data, the user has already selected buttons 2101-2110 on the pen operation panel 2001, 2008 (PenId has been identified). Therefore, a. The operation button definition data identifies the pen button ID, ColorId, and Angle. b. The pen ID control data includes PenId, ColorId, and Angle.

[0233] S1: First, the handwriting input display control unit 23 transmits the start of handwriting data to the handwriting input storage unit 25. The handwriting input storage unit 25 reserves a handwriting data area (a memory area for storing handwriting data). The handwriting data area may be reserved after the user touches the pen to the handwriting input unit 21.

[0234] S2: Next, the user touches the pen to the handwriting input unit 21. The handwriting input unit 21 detects the pen drop and transmits it to the handwriting input display control unit 23.

[0235] S3: The handwriting input display control unit 23 transmits a stroke start signal to the handwriting input storage unit 25, and the handwriting input storage unit 25 reserves the stroke area.

[0236] S4: When the user moves the pen while keeping it in contact with the handwriting input unit 21, the handwriting input unit 21 transmits the pen coordinates to the handwriting input display control unit 23.

[0237] S5: The handwriting input display control unit 23 retrieves the current pen ID control data stored in the pen ID control data storage unit 36 ​​by specifying the pen ID received from the pen 2500 at the same time as the coordinate input. Since the pen ID is transmitted when the coordinates are input, the stroke and pen ID are associated. The pen ID control data storage unit 36 ​​sends the pen ID control data to the handwriting input display control unit 23. Note that there is no AccountId because the user is not signed in.

[0238] S6: The handwriting input display control unit 23 transmits pen coordinate interpolation display data (data that interpolates discrete pen coordinates) to the display unit 22. The display unit 22 interpolates the pen coordinates using the pen coordinate interpolation display data, identifies the line type and thickness from the color definition data based on the ColorId, and displays the stroke.

[0239] S7: The handwritten input display control unit 23 transmits the pen coordinates, the reception time thereof, the ColorId, and the angle information to the handwritten input storage unit 25. The handwritten input storage unit 25 adds the pen coordinates to the stroke. While the user is moving the pen, the handwritten input unit 21 periodically repeats the transmission of the pen coordinates to the handwritten input display control unit 23, so the processes of steps S4 to S7 are repeated until the pen is lifted up.

[0240] S8: When the user releases the pen from the handwritten input unit 21, the handwritten input unit 21 transmits a pen-up to the handwritten input display control unit 23.

[0241] S9: The handwritten input display control unit 23 transmits a stroke end to the handwritten input storage unit 25, and the handwritten input storage unit 25 determines the pen coordinates of the stroke. After the pen coordinates of the stroke are determined, pen coordinates cannot be added to the stroke anymore.

[0242] S10: Next, the handwritten input display control unit 23 transmits an acquisition of the overlapping situation between the handwritten data rectangular area and the stroke rectangular area to the handwritten input storage unit 25 based on the handwritten data rectangular area of 403. The handwritten input storage unit 25 calculates the overlapping situation and transmits the overlapping situation to the handwritten input display control unit 23.

[0243] Subsequent steps S11 to S17 are executed when the handwritten data rectangular area and the stroke rectangular area do not overlap.

[0244] S11: When the handwritten data rectangular area and the stroke rectangular area do not overlap, since one handwritten data is determined, the handwritten input display control unit 23 transmits a holding data clear to the handwritten recognition control unit 26.

[0245] S12-S14: The handwriting recognition control unit 26 sends a message to the string conversion control unit 28, the predictive text control unit 30, and the operation command recognition control unit 32 to clear the held data. The handwriting recognition control unit 26, the string conversion control unit 28, the predictive text control unit 30, and the operation command recognition control unit 32 clear the data related to the string candidates and operation command candidates that they have held up to that point. Note that the last handwritten stroke is not added to the handwriting data at the time of clearing.

[0246] S15: The handwriting input display control unit 23 transmits the end of the handwriting data to the handwriting input storage unit 25. The handwriting input storage unit 25 confirms the handwriting data. Confirmation of handwriting data means that one handwriting data is complete (no more strokes will be added).

[0247] S16: The handwriting input display control unit 23 transmits the start of handwriting data to the handwriting input storage unit 25. In preparation for the start of the next handwriting data (pen down), the handwriting input storage unit 25 reserves a new handwriting data area.

[0248] S17: Next, the handwriting input display control unit 23 sends a message to the handwriting input storage unit 25 to add a stroke regarding the stroke that was completed in step S9. If steps S11 to S17 have been performed, the added stroke is the first stroke of the handwriting data, and the handwriting input storage unit 25 adds the stroke data to the handwriting data that is currently being started. If steps S11 to S17 have not been performed, the added stroke is added to the handwriting data that is already being written.

[0249] S18: Next, the handwriting input display control unit 23 transmits the addition of a stroke to the handwriting recognition control unit 26. The handwriting recognition control unit 26 adds the stroke data to the stroke data holding area (an area where stroke data is temporarily stored) where the string candidates are stored.

[0250] S19: The handwriting recognition control unit 26 performs gesture handwriting recognition on the stroke data holding area. Gesture handwriting recognition refers to the recognition of angle information from a straight line. Since gesture handwriting recognition is performed inside the operation guide 500, the handwriting recognition control unit 26 detects a straight line inside the operation guide 500. The position information of the operation guide 500 is transmitted to the handwriting recognition control unit 26 in step S67, which will be described later.

[0251] S20: If a straight line is detected within the operation guide 500, the angle α formed by the straight line 522 drawn downwards from the starting point of the straight line at the 6 o'clock position and the straight line 521 entered by the user, in a counterclockwise direction, is determined in units of 45 degrees. The handwriting recognition control unit 26 then associates the determined angle information with the pen ID associated with the stroke data of the straight line 521 and stores it in the pen ID control data storage unit 36. Step S20 is executed when a straight line is detected within the operation guide 500. Therefore, it is possible to set a different angle in priority to the angle determined by pressing buttons 2101 to 2115 on the pen operation panels 2001 and 2008.

[0252] S20-2: The handwriting recognition control unit 26 clears the selectable candidate display rectangle.

[0253] S21: Next, the handwriting recognition control unit 26 specifies the pen ID received from the handwriting input unit 21 and obtains the angle information of the current pen ID control data from the pen ID control data storage unit 36.

[0254] S22: The handwriting recognition control unit 26 rotates clockwise using the angle information obtained from the stroke data holding area. The display device 2 can then recognize characters by rotating the stroke data according to the information related to data processing corresponding to the display position.

[0255] S23: The handwriting recognition control unit 26 transmits the rotated stroke data to the handwriting signature authentication control unit 38. In this way, the stroke data is always transmitted to the handwriting signature authentication control unit 38 without knowing whether it is a handwritten signature or not.

[0256] S24: The handwritten signature authentication control unit 38 receives stroke data and receives registered handwritten signature data from the handwritten signature data storage unit 39. The handwritten signature authentication control unit 38 then compares (matches) the stroke data with the handwritten signature data and stores the authentication result of the handwritten signature so that the authentication result of the handwritten signature can be obtained in the subsequent step S61. If authentication is successful, the AccountId is registered in the pen ID control data.

[0257] S25: Next, the handwriting recognition control unit 26 performs handwriting recognition on the stroke data. If the form has a "check mark" or "×" in the registration or cancellation field, it processes the form; otherwise, it performs normal handwriting recognition processing.

[0258] S26: If the "Register or Cancel" field in the handwritten signature data registration form is checked, the handwriting recognition control unit 26 transmits the handwritten signature data (stroke data) entered by the user into the handwritten signature registration form to the handwritten signature authentication control unit 38. The handwritten signature registration form is generated in the handwriting input storage unit 25 by the handwriting input display control unit 23 in step S86, which will be described later.

[0259] S27: The handwritten signature authentication control unit 38 registers the received handwritten signature data (stroke data) with the handwritten signature data storage unit 39. This assigns a SignatureId. The SignatureId is returned to the handwritten recognition control unit 26. If the SignatureId and the name entered in the name input field of the handwritten signature registration form are not in the user-defined data, the handwritten recognition control unit 26 adds new user-defined data. The handwritten recognition control unit 26 assigns an AccountId and saves the SignatureId to its user-defined data. If the name entered in the name input field is in the user-defined data, the SignatureId is saved to that user-defined data. This process links the AccountId and SignatureId.

[0260] S28: By registering the handwritten signature data, the handwritten recognition control unit 26 deletes the handwritten signature registration form from the handwritten input storage unit 25.

[0261] S29: When the registration or cancellation column in the user-defined data change form is a "check mark", the handwritten recognition control unit 26 sends the changed value entered in the user-defined data change form 562 to the operation command definition unit 33. The user-defined data change form 562 is generated by the handwritten input display control unit 23 in the handwritten input storage unit 25 in step S86 described later.

[0262] S30: By executing the change of the user-defined data, the handwritten recognition control unit 26 deletes the user-defined data change form 562 from the handwritten input storage unit 25.

[0263] S31: When the registration or cancellation column in the form added in step S86 described later is "×", the handwritten recognition control unit 26 deletes the form added in step S86 from the handwritten input storage unit 25.

[0264] S33: If it is not a form process, the handwritten recognition control unit 26 sends the handwritten recognition character string candidates handwritten by the user to the handwritten recognition dictionary unit 27. The handwritten recognition dictionary unit 27 sends the language string candidates that are linguistically reliable to the handwritten recognition control unit 26.

[0265] Note that if "RecommendMenuOff = \"True\"" is currently set in the pen ID control data and furthermore, the operation command is also not displayed, the handwritten recognition control unit 26 does not perform the control related to recognition in steps S33 to S47. By doing so, the display device 2 can reduce the processing load.

[0266] S34: The handwritten recognition control unit 26 sends the handwritten recognition character string candidates and the received language string candidates to the character string conversion control unit 28.

[0267] S35: The string conversion control unit 28 transmits the handwritten recognition string candidates and language string candidates to the string conversion dictionary unit 29. The string conversion dictionary unit 29 transmits the conversion string candidates to the string conversion control unit 28.

[0268] S36: The string conversion control unit 28 transmits the received conversion string candidate to the predictive conversion control unit 30.

[0269] S37: The predictive text control unit 30 transmits the received conversion string candidates to the predictive text dictionary unit 31. The predictive text dictionary unit 31 transmits the predicted string candidates to the predictive text control unit 30.

[0270] S38: The predictive text control unit 30 transmits the received predictive string candidates to the operation command recognition control unit 32.

[0271] S39: The operation command recognition control unit 32 transmits the received predicted string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can obtain the operation command candidate corresponding to the operation command definition data having a string that matches the predicted string candidate.

[0272] The display device 2 continues to process in the same manner until it sends the candidate operation command shown in steps S40 to S47. S40: The string conversion control unit 28 transmits the received conversion string candidate to the operation command recognition control unit 32.

[0273] S41: The operation command recognition control unit 32 transmits the received conversion string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can obtain an operation command candidate corresponding to the operation command definition data having a string that matches the conversion string candidate.

[0274] S42: The handwriting recognition control unit 26 transmits the handwriting recognition string candidates and language string candidates to the predictive text conversion control unit 30.

[0275] S43: The predictive text control unit 30 transmits the handwritten recognition string candidates and the received language string candidates to the predictive text dictionary unit 31. The predictive text dictionary unit 31 transmits the predicted string candidates to the predictive text control unit 30.

[0276] S44: The predictive text control unit 30 transmits the received predictive string candidates to the operation command recognition control unit 32.

[0277] S45: The operation command recognition control unit 32 transmits the received predicted string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can obtain the operation command candidate corresponding to the operation command definition data having a string that matches the predicted string candidate.

[0278] S46: The handwriting recognition control unit 26 transmits the handwriting recognition string candidates and the received language string candidates to the operation command recognition control unit 32.

[0279] S47: The operation command recognition control unit 32 transmits the handwritten recognition string candidate and the received language string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can obtain the operation command candidate corresponding to the operation command definition data having a string that matches the language string candidate.

[0280] S48: Next, the handwriting recognition control unit 26 sends a stroke addition to the operation command recognition control unit 32.

[0281] S49: The operation command recognition control unit 32 transmits the location information of the confirmed data to the handwriting input storage unit 25. The handwriting input storage unit 25 transmits the location information of the confirmed data to the operation command recognition control unit 32.

[0282] S50: The operation command recognition control unit 32 determines the selected data. Based on the crossing line determination condition 406 and the enclosing line determination condition 407, the operation command recognition control unit 32 determines whether the position information of the stroke received from the handwriting recognition control unit 26 in step S48 and the position information of the confirmed data received from the handwriting input storage unit 25 have a predetermined relationship. If there is confirmed data that can be determined to be selected, the operation command recognition control unit 32 stores the confirmed data as selected data. Also, in this case, since the selected data is identified, the operation command definition unit 33 obtains candidate operation commands for when there is selected data.

[0283] Furthermore, the handwriting recognition control unit 26, the string conversion control unit 28, the predictive text conversion control unit 30, and the operation command recognition control unit 32 each store handwriting recognition string candidates, language string candidates, conversion string candidates, predictive string candidates, operation command candidates, and data related to selected data so that they can be acquired in subsequent steps S55 to S58.

[0284] S18-2: Immediately after sending the stroke addition message to the handwriting recognition control unit 26 in step S18, the handwriting input display control unit 23 sends the selectable candidate display timer start message to the candidate display timer control unit 24. The candidate display timer control unit 24 starts this timer.

[0285] The following steps S51-S53 are executed if a pen-down occurs before a certain period of time has elapsed (before the timer times out).

[0286] S51: If the user touches the pen to the handwriting input unit 21 before the timer times out, the handwriting input unit 21 sends a pen down message (the same event as in step S2) to the handwriting input display control unit 23.

[0287] S52: The handwriting input display control unit 23 transmits a stroke start signal (same as step S3) to the handwriting input storage unit 25. The subsequent sequence is the same as from step S3 onwards.

[0288] S53: The handwriting input display control unit 23 then transmits a message to the candidate display timer control unit 24 to stop the selectable candidate display timer. The candidate display timer control unit 24 stops the timer because a pen-down has been detected and the timer is no longer needed.

[0289] Steps S54 to S103 are executed if a pen-down does not occur before a certain period of time has elapsed (before the timer times out). Therefore, the operation guide 500 shown in Figure 19 is displayed.

[0290] S54: If the user does not touch the handwriting input unit 21 with a pen while the selectable candidate display timer is running, the candidate display timer control unit 24 sends a timeout message to the handwriting input display control unit 23.

[0291] S55: The handwriting input display control unit 23 transmits the acquisition of handwriting recognition string / language string candidates to the handwriting recognition control unit 26. The handwriting recognition control unit 26 transmits the currently held handwriting recognition string / language string candidates to the handwriting input display control unit 23.

[0292] S56: The handwriting input display control unit 23 sends the acquisition of conversion string candidates to the string conversion control unit 28. The string conversion control unit 28 sends the currently held conversion string candidates to the handwriting input display control unit 23.

[0293] S57: The handwriting input display control unit 23 transmits the acquisition of predicted string candidates to the predictive conversion control unit 30. The predictive conversion control unit 30 transmits the currently held predicted string candidates to the handwriting input display control unit 23.

[0294] S58: The handwriting input display control unit 23 transmits the acquisition of candidate operation commands to the operation command recognition control unit 32. The operation command recognition control unit 32 transmits the currently held candidate operation commands and selected data to the handwriting input display control unit 23.

[0295] S59: Furthermore, the handwriting input display control unit 23 transmits the estimated writing direction to the handwriting input storage unit 25. The handwriting input storage unit 25 determines the estimated writing direction from the stroke addition time, horizontal distance, and vertical distance of the handwriting data rectangular area and transmits it to the handwriting input display control unit 23.

[0296] S60: Next, the handwriting input display control unit 23 specifies the pen ID received from the handwriting input unit 21 and obtains the current pen ID control data ColorId, AccountId (which may not be present), and Angle from the pen ID control data storage unit 36.

[0297] S61: The handwriting input display control unit 23 obtains the authentication result of the handwritten signature from the handwriting signature authentication control unit 38. This allows the user's SignatureId to be obtained, and when the operation command described later is executed, the AccountId is registered in association with the pen ID control data. When the user signs in, the color definition data associated with the user-defined data identified by the AccountId is identified. The handwriting input display control unit 23 can display handwritten data in which the color information of the user-defined data is highlighted in black and white, or the text data converted from the handwritten data.

[0298] S62: The handwritten input display control unit 23 determines whether "RecommendMenuOff = \"True\"" is set in the pen ID control data of the pen 2500 used by the user. If "RecommendMenuOff = \"True\"", the string candidates 539 are not displayed. In this case, only the operation commands may be displayed, or the operation commands may not be displayed either. If "RecommendMenuOff = \"False\"", the handwritten input display control unit 23 creates selectable candidate display data such as these handwritten recognition string candidates (\"gi\" in FIG. 19), language string candidates (not shown in FIG. 19 but for example \"yi\"), conversion string candidates (\"minutes of meeting\", \"skill test\" in FIG. 19), prediction string candidates (\"settle the skill test\", \"sender of the minutes of meeting\" in FIG. 19), candidate operation commands (\"load the minutes of meeting template\", \"save to the minutes of meeting folder\" in FIG. 19), each selection probability, and the estimated writing direction as shown in FIG. 19. Also, the handwritten input display control unit 23 rotates the selectable candidate display data (operation guide 500) obtained in step S60 counterclockwise by the Angle, and displays it by sending the rotated selectable candidate display data (operation guide 500) to the display unit 22.

[0299] S63: Further, the handwritten input display control unit 23 rotates the rectangular area display data (rectangular frame) of the handwritten data and the selection data (handwritten data rectangular area display 503 in FIG. 19) counterclockwise by the angle information obtained in step S60, and displays it by sending it to the display unit 22.

[0300] S64: The handwritten input display control unit 23 sends a signal to start the selectable candidate display erasing timer to the candidate display timer control unit 24 in order to erase the selectable candidate display data after a certain time from the display. The candidate display timer control unit 24 starts this timer. If the operation guide 500 including the operation commands is not displayed because "RecommendMenuOff = \"True\"", the signal to start the selectable candidate display erasing timer is not sent.

[0301] Steps S65 to S70 are executed if, while the selectable candidate clearing timer is running, the user clears the selectable candidate displayed on the display unit 22, a change occurs in the handwritten data (i.e., strokes in the handwritten data are added, deleted, moved, transformed, or split), or if no candidate is selected by the timeout.

[0302] Furthermore, steps S65 to S67 are executed when the candidate display is cleared or when a change occurs in the handwritten data.

[0303] S65: The handwriting input unit 21 transmits to the handwriting input display control unit 23 that the selectable candidate display has been cleared or that a change has occurred in the handwriting data.

[0304] S66: The handwriting input display control unit 23 transmits a message to stop the selectable candidate deletion timer. The candidate display timer control unit 24 stops the timer because an operation has been performed on the handwritten data within a certain time period, making the timer unnecessary.

[0305] S67: The handwriting input display control unit 23 stores the position information of the operation guide 500 in the handwriting recognition control unit 26 so that it can be used in the gesture determination of the gesture handwriting recognition in step S19. The position information is, for example, the coordinates of the upper left corner and the lower right corner or equivalent coordinates. This allows the handwriting recognition control unit 26 to determine whether the straight line used for inputting angle information is within the operation guide 500.

[0306] S69: The handwriting input display control unit 23 clears the display by sending a message to the display unit 22 to erase the selectable candidate display data. If "RecommendMenuOff="True"", only the operation command is erased, or nothing needs to be erased.

[0307] S70: The handwriting input display control unit 23 erases the display by sending a message to the display unit 22 indicating that the rectangular area display data for the handwritten data and selected data has been erased. Therefore, if the display of the operation command candidate is erased under conditions other than when an operation command candidate is selected, the handwritten data remains displayed.

[0308] S68: On the other hand, if the selectable candidate display is not cleared or the handwritten data does not change while the selectable candidate clearing timer is running (i.e., the user does not perform any pen operations), the candidate display timer control unit 24 sends a timeout message to the handwritten input display control unit 23.

[0309] Similarly, after the timer for clearing the selectable candidate display times out, the handwriting input display control unit 23 executes steps S69 and S70. This is because, after a certain period of time, the display unit 22 may clear the selectable candidate display data, the handwritten data, and the rectangular area display data for the selected data.

[0310] If the user selects a candidate while the selectable candidate deletion timer is running, steps S71 to S103 are executed.

[0311] S71: If the user selects a candidate while the selectable candidate deletion timer is running, the handwriting input unit 21 transmits the selection of a string candidate or operation command candidate to the handwriting input display control unit 23.

[0312] S71-2: The handwriting input display control unit 23 transmits a message to the candidate display timer control unit 24 to stop the selectable candidate display clear timer. The candidate display timer control unit 24 stops this timer.

[0313] S72: Next, the handwriting input display control unit 23 sends a message to the handwriting recognition control unit 26 to clear the retained data.

[0314] S73: The handwriting recognition control unit 26 sends a message to the string conversion control unit 28 to clear the retained data.

[0315] S74: The handwriting recognition control unit 26 sends a message to the predictive conversion control unit 30 to clear the retained data.

[0316] S75: The handwriting recognition control unit 26 sends a message to the operation command recognition control unit 32 to clear the retained data. The handwriting recognition control unit 26, the string conversion control unit 28, the predictive text control unit 30, and the operation command recognition control unit 32 clear the data related to the string candidates and operation command candidates that they had been holding.

[0317] S76: Next, the handwriting input display control unit 23 sends a message to the display unit 22 to erase the selectable candidate display data, thereby erasing the display.

[0318] S77: The handwriting input display control unit 23 erases the display by sending a message to the display unit 22 to erase the rectangular area display data of the handwritten data and selected data.

[0319] S78: The handwriting input display control unit 23 erases the display by sending a message to the display unit 22 erasing the display data and the pen coordinate interpolation display data sent in step S6. This is because handwritten data, etc., is no longer needed because a string candidate or operation command candidate has been selected.

[0320] S79: The handwriting input display control unit 23 transmits a message to the handwriting input storage unit 25 indicating that the handwriting data has been deleted.

[0321] If a string candidate is selected, steps S80 to S82 are executed.

[0322] S80: If a string candidate is selected, the handwriting input display control unit 23 sends a message to the handwriting input storage unit 25 to add the string data.

[0323] S81: Furthermore, the handwriting input display control unit 23 transmits the acquisition of the string data font to the handwriting input storage unit 25. The handwriting input storage unit 25 identifies the font name of the string data associated with the PenId of the pen ID control data from the pen color selection button definition data, and also transmits the estimated character size of the handwritten data to the handwriting input display control unit 23.

[0324] S82: Next, the handwriting input display control unit 23 uses a predefined font received from the handwriting input storage unit 25 to send string data to the display unit 22 to be displayed at the same position as the handwritten data. The handwriting input display control unit 23 identifies the line type and thickness from the color definition data based on the ColorId of the pen ID control data and displays the text (string data) rotated by the angle information. Therefore, the display device 2 can display string data with the top and bottom directions relative to the user's operating position.

[0325] If a candidate for an operation command is selected, steps S83 to S101 are executed. Operation commands for selected data for editing or modification are executed in steps S83 to S85.

[0326] S83: If a candidate for an operation command for selected data is selected (if selected data exists), the handwriting input display control unit 23 sends a message to the display unit 22 to erase the selected data display data, thereby erasing the display. This is because the display unit 22 erases the original selected data.

[0327] S84: Next, the handwriting input display control unit 23 transmits an operation command to the handwriting input storage unit 25 to execute an operation command on the selected data. The handwriting input storage unit 25 transmits the display data of the new selected data (display data after editing or modification) to the handwriting input display control unit 23.

[0328] S85: Next, the handwriting input display control unit 23 transmits the selected data display data to the display unit 22, thereby redisplaying the selected data after the operation command has been executed.

[0329] Although selected data exists, the "Add Page Name" operation command is executed in steps S83-2 to S84-2.

[0330] S83-2: If a candidate for the "Add Page Name" operation command is selected (if selection data exists), the handwriting input display control unit 23 requests the handwriting input storage unit 25 to acquire the selection data. The handwriting input storage unit 25 transmits the selection data to the handwriting input display control unit 23.

[0331] S84-2: The handwriting input display control unit 23 checks the page number currently displayed on the display 220 using page navigation data and adds the selected data as the page name to the handwriting input storage unit 25.

[0332] S86: If "Register handwritten signature" is specified in operation command definition data 713 or "Change settings" is specified in operation command definition data 716, the handwriting input display control unit 23 adds the handwritten signature registration form or the user-defined data change form to the handwriting input storage unit 25.

[0333] S87: If the candidate for the operation command "Save File" or "Print" is selected, the handwriting input display control unit 23 sends a file transmission request to the file transmission / reception control unit 37.

[0334] S88: The file transmission / reception control unit 37 sends a request to the handwritten input storage unit 25 to acquire the handwritten input storage data to be sent as a file.

[0335] S89: The handwriting input storage unit 25 determines whether the destination is a color-compatible device using MIB or other means. The handwriting input storage unit 25 processes the file depending on whether the destination is a color-compatible device, a monochrome-compatible device, or a monochrome-enhanced device.

[0336] S90: The file transmission / reception control unit 37 either sends the handwritten input data received from the handwritten input storage unit to the destination or writes it to a file.

[0337] S91: If a candidate for the "Load File" operation command is selected, the handwriting input display control unit 23 sends a request to the file transmission / reception control unit 37 to acquire file list information.

[0338] S92: The file transmission / reception control unit 37 receives file list information from storage media such as USB memory, network storage, or web servers, or from external devices.

[0339] S93: The file transmission / reception control unit 37 transmits file list information to the handwriting input display control unit 23.

[0340] S94: The handwriting input display control unit 23 transmits the file list display data to the display unit 22. As a result, the display unit 22 displays the file list on the display.

[0341] S95: When the user selects a file and the handwriting input unit 21 receives the selection, the handwriting input unit 21 transmits the file selection to the handwriting input display control unit 23.

[0342] S96: The handwriting input display control unit 23 sends a file reception request for the selected file to the file transmission / reception control unit 37.

[0343] S97: The file transmission / reception control unit 37 retrieves a file from an external device.

[0344] S98: The file transmission / reception control unit 37 saves the file to the handwriting input / storage unit 25.

[0345] S99: The handwriting input storage unit 25 analyzes the file received from the file transmission / reception control unit 37 and converts the handwriting input-derived data into handwriting input storage data (data that can be converted between black and white and color). That is, it determines whether metadata is present, and if metadata is present, it determines whether it can be converted into handwriting input storage data (whether there is a ColorId, etc.) and saves it as handwriting input storage data. The handwriting input storage unit 25 then reads the ColorId from the handwriting input-derived data, refers to the color definition data, and converts it to a black and white highlight display associated with the ColorId. The handwriting input storage unit 25 transmits the display data of the handwriting data to the handwriting input display control unit 23. If the data is not handwriting input-derived data, the handwriting input storage unit 25 reads the color information, font, character size, etc. according to the file format and saves it as handwriting input storage data.

[0346] S100: The handwriting input display control unit 23 displays the handwriting input data on the display unit 22. As a result, the display unit 22 displays the handwriting input-related data as grayscale enhanced handwriting data, and displays other data in grayscale using conventional brightness conversion.

[0347] When the operation command 512 for signing in is executed, the handwriting input display control unit 23 obtains the pen ID received by the display device 2 when the operation command 512 is executed. The handwriting input display control unit 23 identifies the user-defined data having the SignatureId obtained in step S61 and obtains the AccountId from the user-defined data. Then, the handwriting input display control unit 23 registers the AccountId in the pen ID control data in association with the pen ID. As a result, the pen 2500 and the user are linked, and the display device 2 can process using the user-defined data.

[0348] When a user signs in and then writes by hand or loads a file, the handwriting input display control unit 23 obtains the AccountId associated with the pen ID received by the display device 2 when the operation command is executed from the pen ID control data. The handwriting input display control unit 23 identifies user-defined data using this AccountId and sets color definition data, etc., in the %~% of the operation command and executes it.

[0349] S101: If an input / output operation command is selected, the handwriting input display control unit 23 executes the operation command string (Command) from the operation command definition data corresponding to the operation command selected by the user.

[0350] S102: If the user manually inputs angle information, the handwriting input display control unit 23 stores the received angle information in the pen ID control data storage unit 36, associating it with the pen ID received from the pen 2500 when the rotation operation button 511 is pressed.

[0351] S103: For the next handwritten data, the handwriting input display control unit 23 transmits the start of handwritten data to the handwriting input storage unit 25. The handwriting input storage unit 25 reserves the handwritten data area. From here on, steps S2 to S103 are repeated.

[0352] <Displaying the operation menu> The following sections will explain the details of the processes that could not be fully described in the sequence diagram.

[0353] First, Figure 40 is an example of a flowchart showing the procedure for the display device 2 to display operation menus 301 and 302.

[0354] For example, when the user turns on the power of display device 2, display device 2 starts up (S201). Display device 2 reads and executes the OS (Operating System) stored on SSD 204, and the OS reads and executes the display device 2 program stored on SSD 204.

[0355] The display unit 22 displays operation menus 301 and 302, which contain information related to data processing according to the display position. First, the display unit 22 displays the operation menu 301 for the lower user (S202). The display unit 22 may display operation menu 302 first or simultaneously. The display unit 22 reads the operation button definition data 2401 to 2424 and obtains the icon file. The display unit 22 determines the position based on Position. The display unit 22 determines the rotation angle based on Angle. The display unit 22 displays the icons for each button 2101 to 2115, 2201, 2202, 2203, 2205, 2301, 2302, 2304, and 2305 on the display 220. Since the user's Angle is 0 degrees, the icons for buttons 2101-2115, 2201, 2202, 2203, 2205, 2301, 2302, 2304, and 2305 will be displayed without rotation. Note that file name 2206 will not be displayed until the file is loaded, and the current page 2203 will show "1 / 1".

[0356] Next, the display unit 22 displays the operation menu 302 for the upper user (S203). The display unit 22 reads the operation button definition data 2425~2448 and obtains the icon file. The display unit 22 determines the position based on Position. The display unit 22 determines the rotation angle based on Angle. The display unit 22 displays the icons for buttons 2101~2115, 2201, 2202, 2203, 2205, 2301, 2302, 2304, and 2305 on the display 220. Since the Angle for the upper user is 180 degrees, the icons for buttons 2101~2115, 2201, 2202, 2203, 2205, 2301, 2302, 2304, and 2305 are rotated by 180 degrees and displayed. This allows each user to see the buttons in an easily viewable manner.

[0357] <Execution of command in the operation button definition data> Figure 41 illustrates the case where buttons 2101 to 2115 on the pen operation panels 2001 and 2008 are selected. Figure 41 is an example of a flowchart illustrating the process when buttons 2101 to 2115 on the pen operation panel are pressed.

[0358] First, the handwriting input unit 21 accepts the selection of one of the buttons 2101 to 2115 (S301). Here, we assume that button 2101 is pressed. The handwriting input unit 21 detects the coordinates touched by the pen 2500, and since buttons 2101 to 2115 are registered in the operation button definition data 2401 to 2415, the pressed button 2101 is identified.

[0359] Next, the handwriting input display control unit 23 executes "ChangePen ThinBlack" in the operation button definition data 2401 corresponding to the pressed button 2101, so the handwriting input display control unit 23 sets the PenId and ColorId="ThinBlack" of the pen 2500 that the user is using in the pen ID control data (S302).

[0360] Furthermore, the handwriting input display control unit 23 sets the "Angle" of the operation button definition data 2401 corresponding to the pressed button 2101 to the pen ID control data (S303).

[0361] Since the pen color is set in the pen ID control data, when the display unit 22 displays handwritten data and string data on the display 220, it can display them using the line type defined in the color definition data. Also, since the angle is set in the pen ID control data, the handwritten data can be rotated before character recognition.

[0362] Although Figure 41 uses the buttons on the pen operation panels 2001 and 2008 as an example, the PenId and Angle can be set in the pen ID control data regardless of whether the buttons on the page operation panels 2002 and 2009, the page navigation operation panels 2003 and 2010, or the page navigation buttons 2004 and 2011 are pressed.

[0363] <Execute the operation command "Set as page name"> Figure 42 is an example of a flowchart illustrating the process when a user executes the operation command "Set as page name".

[0364] First, the user selects the confirmed data (text data after handwriting recognition) already displayed on the display 220 using overlapping or enclosing lines. This causes the display unit 22 to display the operation command "Set as page name". When the user presses this operation command, the handwriting input unit 21 accepts the selection of the operation command (S401). Since the coordinates of the operation guide 500 are known, the selected operation command is identified by the coordinates where the tip of the pen 2500 makes contact.

[0365] The handwriting input display control unit 23 obtains the operation command definition data for the selected operation command Name="Set as page name", and executes Command=SetPageName (S402).

[0366] The handwriting input display control unit 23 sets the selected data specified by the crossing line or border to the PageName of the page currently displayed on the display 220 (the page where CurrentPage=True in the page navigation data) (S403). In this way, the user can assign a page name to the page with a simple operation of selecting confirmed data.

[0367] Until the user assigns a page name, a default value (e.g., page number X, blank page, etc.) will be set as the page name when the user begins writing on the page.

[0368] <Display page navigation> Figure 43 is an example of a flowchart illustrating how to display page navigation.

[0369] The display unit 22 determines whether or not the page navigation window button (button 2115) has been opened (S501). Since the coordinates are set in the operation button definition data 2439 for the page navigation window button, it is possible to determine whether it has been pressed from the coordinates touched by the pen 2500.

[0370] When an operation to open the page navigation window button is input, the display unit 22 opens the page navigation window areas 2005 and 2012 (S502). Until the page navigation window area is opened, the page navigation window area is closed (there is no page navigation window area), so the display unit 22 horizontally shrinks the entire display 220 to free up the right side of the display 220. Note that the page navigation window area opened is either 2005 or 2012, or both, as requested by the user.

[0371] The number of page navigation window areas is at most the same as the number of operation menus. The number of operation menus can be the number of "Angle" types included in the operation button definition data. Alternatively, the number of operation menus may be directly set as a numerical value in display device 2. Page navigation window area 2012 is also displayed in a position rotated by "Angle". Therefore, the page navigation window area will be, for example, on the user's right side.

[0372] Next, the display unit 22 acquires page navigation data 2501 to 2511 (S503). The display unit 22 processes the page navigation data 2501 to 2511 one row at a time.

[0373] First, the display unit 22 identifies the file name (S504). The display unit 22 determines whether it is a file name by checking whether "Page=0" and "Level=0". Since the file name is always displayed, the display unit 22 displays FileName (S505).

[0374] Next, the display unit 22 determines whether the "Tree" of the file under consideration is "Open" or not (S506). If there is no "Tree" setting, it can be determined to be No.

[0375] If the "Tree" of the file is open, the page navigation displays the page names within the file, so the display unit 22 displays all the page names one level lower (S507).

[0376] Next, the display unit 22 determines whether or not the page whose page name is displayed has a Tree setting (S508).

[0377] For pages with a Tree configuration, the display unit 22 displays the names of all pages one level lower (S509). This process is performed for all nested pages.

[0378] Next, the display unit 22 determines whether the processing of one file has been completed (S510).

[0379] If processing of one file is not yet complete, the process returns to step S508, and the display unit 22 repeats the process.

[0380] If processing of one file is complete, the display unit 22 determines whether all page navigation data has been processed (S511). If processing is not complete, it returns to step S504 and processes the next file.

[0381] If the user inputs an operation to close the page navigation window button (Yes in S512), the display unit 22 closes the page navigation window area (S513). The display unit 22 then clears the page navigation window area and expands the entire display 220 horizontally to display handwritten data etc. across the entire display 220.

[0382] By displaying the page navigation in this way, text and other elements that were not visible in the traditional thumbnails become easier to see, making it easier for users to decide which page to open. Furthermore, the page navigation window area can be opened and closed, allowing for wider use of display 220. Users can display the page on display 220 simply by selecting the file name or page name in the page navigation.

[0383] <Example of pen holder placement> Next, the pen holder for accommodating the pen 2500 will be explained using Figure 44. Figure 44 is a diagram illustrating an example of the arrangement of the pen holder. Figure 44 shows an example of the arrangement of the pen holder when the display device 2 is placed flat on the desk (horizontally facing upwards). Note that the display 220 shows handwritten data, etc., similar to that in Figure 30.

[0384] Display device 2 has pen holders 2603 and 2604 for the lower user. Pen holder 2603 is in arrangement example 1, and pen holder 2604 is in arrangement example 2. In other words, either one of the pen holders is sufficient, but both pen holders may also be present. Display device 2 also has pen holders 2612 and 2613 for the upper user. Pen holder 2612 is in arrangement example 1, and pen holder 2613 is in arrangement example 2.

[0385] Each pen holder 2603, 2612 allows the pen to be removed from the edge of the display 220 along which the operation menu is displayed, while each pen holder 2604, 2613 stores the pen 2500 parallel to the said edge.

[0386] Both Layout Example 1 and Layout Example 2 are positioned in a location that is easily accessible to the user below or above, and are also close to the user's operation menus 301 and 302. Therefore, there is a high probability that users will unconsciously use operation menus 301 and 302 to automatically set information related to data processing (rotation angle) corresponding to the display position into the pen ID control data.

[0387] Once the information (angle) for data processing based on the display position is set in the pen ID control data, it remains valid even after the next startup. If the user takes the pen 2500 out of the pen holder and starts writing without using operation menus 301 and 302, the information for data processing based on the display position is already set in the pen ID control data. The user can then use the pen 2500 as if it were a dedicated pen for either the lower or upper user.

[0388] <Determining the placement of the operation menu> The layout of the operation menu may be defined in advance using operation button definition data, or it may be set by an administrator or other authorized person.

[0389] Figure 45 shows an example of a layout determination screen used by the administrator to decide the placement of operation menus. The administrator displays the layout determination screen on the display device 2 using a predetermined operation. The layout determination screen displays the operation menus in their default or current settings. In Figure 45, the operation menus for the lower user (pen operation panel 2001, page operation panel 2002, and page navigation operation panel 2003) are displayed from the start. The administrator inputs an operation to duplicate these using the pen 2500, etc., and moves them to the desired location. In Figure 45, pen operation panels 2008 and 2015 are placed for the left and upper users, respectively. Page operation panel 2002 and page navigation operation panel 2003 can be placed in the same way.

[0390] Once the administrator finishes the configuration, the display device 2 creates operation button definition data based on the configuration of the operation menu set by the administrator. As a result, the operation menu will be displayed in the configuration set by the administrator the next time the system is started.

[0391] <Main effects> As described above, the display device 2 of this embodiment displays an operation menu containing information related to data processing according to the display position, enabling data processing according to the user's operation position when operating the operation menu. In addition, since a page navigation is displayed, the user can understand what content is contained on each page. Furthermore, by selecting the page name displayed in the page navigation, the user can easily switch the page displayed on the display 220. [Examples]

[0392] In Example 1, the display device 2 is described as having a large touch panel, but the display device is not limited to having a touch panel. In this example, a projector-type display device will be described.

[0393] <<Another example of a display device configuration 1>> Figure 46 shows another example of a display device configuration. In Figure 46, a projector 411 is installed on the top edge of a regular whiteboard 413. This projector 411 corresponds to the display device. A regular whiteboard 413 is not a flat panel display integrated with a touch panel, but a whiteboard on which the user directly writes with a marker. Note that the whiteboard can also be a blackboard; any flat surface large enough to project an image will suffice.

[0394] The projector 411 has an ultra-short-throw optical system and can project a low-distortion image onto the whiteboard 413 from a distance of about 10 cm. This image may be transmitted from a PC 400-1 connected wirelessly or via a wired connection, or it may be stored in the projector 411's memory.

[0395] The user writes on the whiteboard 413 using a dedicated electronic pen 2700. The electronic pen 2700 has a light-emitting part, for example at its tip, which switches on and emits light when the user presses it against the whiteboard 413 to write. The wavelength of the light is near-infrared or infrared, so it is invisible to the user's eye. The projector 411 has a camera that captures an image of the light-emitting part and analyzes the image to determine the direction of the electronic pen 2700. In addition, the electronic pen 2700 emits sound waves along with the light, and the projector 411 calculates the distance based on the arrival time of the sound waves. The projector 411 can determine the position of the electronic pen 2700 based on its direction and distance. A stroke is drawn (projected) at the position of the electronic pen 2700.

[0396] The projector 411 projects the menu 430, so when a user presses a button with the electronic pen 2700, the projector 411 identifies the pressed button based on the position of the electronic pen 2700 and the ON signal of the switch. For example, when the save button 431 is pressed, the stroke (set of coordinates) handwritten by the user is saved by the projector 411. The projector 411 saves the handwritten information to a predetermined server 412 or USB memory 2600, etc. The handwritten information is saved page by page. Since it is saved as coordinates and not as image data, the user can re-edit it. However, in this embodiment, since operation commands can be called up by handwriting, the menu 430 does not need to be displayed.

[0397] <<Another example of a display device configuration 2>> Figure 47 shows another example of the display device 2 configuration. In the example in Figure 47, the display device 2 includes a terminal device 600, an image projection device 700A, and a pen motion detection device 810.

[0398] The terminal device 600 is connected by wires to the image projection device 700A and the pen motion detection device 810. The image projection device 700A projects the image data input by the terminal device 600 onto the screen 800.

[0399] The pen motion detection device 810 communicates with the electronic pen 820 and detects the movement of the electronic pen 820 in the vicinity of the screen 800. Specifically, the electronic pen 820 detects coordinate information indicating the point it is pointing to on the screen 800 and transmits it to the terminal device 600.

[0400] The terminal device 600 generates image data of the stroke image input by the electronic pen 820 based on coordinate information received from the pen motion detection device 810, and causes the image projection device 700A to draw the stroke image on the screen 800.

[0401] Furthermore, the terminal device 600 generates superimposed image data, which is a composite image obtained by combining the background image projected onto the image projection device 700A with the stroke image input by the electronic pen 820.

[0402] <<Another example of a display device configuration 3>> Figure 48 shows an example of the configuration of a display device. In the example in Figure 48, the display device 2 includes a terminal device 600, a display 800A, and a pen motion detection device 810.

[0403] The pen motion detection device 810 is positioned near the display 800A and detects coordinate information indicating the point pointed to by the electronic pen 820A on the display 800A, and transmits it to the terminal device 600. In the example shown in Figure 48, the electronic pen 820A may be charged by the terminal device 600 via a USB connector.

[0404] The terminal device 600 generates image data of the stroke image input by the electronic pen 820A based on the coordinate information received from the pen motion detection device 810, and displays it on the display 800A.

[0405] <<Another example of a display device configuration 4>> Figure 49 shows an example of the configuration of a display device. In the example in Figure 49, the display device 2 includes a terminal device 600 and an image projection device 700A.

[0406] The terminal device 600 communicates wirelessly with the electronic pen 820B (using Bluetooth®, etc.) to receive coordinate information of the point indicated by the electronic pen 820B on the screen 800. Based on the received coordinate information, the terminal device 600 generates image data of the stroke image input by the electronic pen 820B and projects the stroke image onto the image projection device 700A.

[0407] Furthermore, the terminal device 600 generates superimposed image data, which is a composite image obtained by combining the background image projected onto the image projection device 700A with the stroke image input by the electronic pen 820.

[0408] As described above, each of the embodiments described can be applied to various system configurations.

[0409] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0410] For example, the display method of this embodiment can be suitably applied to any information processing device having a touch panel. Furthermore, devices having similar functions to display devices are also referred to as electronic blackboards, electronic whiteboards, electronic information boards, interactive boards, etc. Examples of information processing devices equipped with a touch panel include projectors (PJ), output devices such as digital signage, head-up display (HUD) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, networked home appliances, personal computers (notebook PCs), mobile phones, smartphones, tablet devices, game consoles, personal digital assistants (PDAs), digital cameras, wearable PCs, or desktop PCs.

[0411] Furthermore, in this embodiment, the server may perform some of the processing that the display device 2 would normally do. For example, the display device sends stroke information to the server, and the server retrieves and displays information to be displayed on the operation guide 500.

[0412] In this embodiment, the display device 2 detects the pen's coordinates by detecting the pen tip's coordinates via a touch panel, but the pen tip's coordinates may also be detected by ultrasound. The pen emits ultrasound along with light emission, and the display device 2 calculates the distance based on the arrival time of the ultrasound. The display device 2 can determine the pen's position based on its direction and distance. The projector draws (projects) the pen's trajectory as a stroke.

[0413] Furthermore, in this embodiment, when there is selected data, candidate operation commands for editing, modification, and "add to page name" are displayed, and when there is no selected data, candidate operation commands for input / output are displayed. However, the display device 2 may display candidate operation commands for editing, modification, and "add to page name" and candidate operation commands for input / output simultaneously.

[0414] Furthermore, the user's handwritten signature data does not necessarily need to be stored in display device 2. It may be stored in the cloud or by an internal information processing device.

[0415] Furthermore, the configuration examples shown in Figure 6 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the display device 2. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the display device 2 can be further divided into many more processing units depending on the processing content. It can also be divided so that one processing unit includes even more processing.

[0416] Furthermore, each function of the embodiments described above can be realized by one or more processing circuits. Here, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, and devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0417] Furthermore, even if a threshold is given as an example for comparison in this embodiment, the threshold is not limited to the example value. For this reason, in this embodiment, for all thresholds, the descriptions "less than the threshold" and "less than or equal to the threshold" have the same meaning, and the descriptions "greater than the threshold" and "greater than or equal to the threshold" have the same meaning. For example, if the threshold is 11, the description "less than the threshold" has the same meaning as if the threshold is 10 and therefore less than or equal to the threshold. Also, if the threshold is 10, the description "greater than the threshold" has the same meaning as if the threshold is 11 and therefore greater than or equal to the threshold. [Explanation of Symbols]

[0418] Display device 2 [Prior art documents] [Patent Documents]

[0419] [Patent Document 1] Japanese Patent Publication No. 2018-026185

Claims

1. An information processing device that displays a first operation menu including a first operation button and a second operation menu including a second operation button on a display, A display control unit that displays the first operation menu along the edge of the first of the four sides of the display, and displays the second operation menu along the edge of the second of the four sides of the display that is opposite to the first side, A memory unit that stores information about the pen used by the user and angle information used for character recognition in association with each other, A first input receiving unit that receives input from the user by pressing the pen to the first operation button included in the first operation menu or the second operation button included in the second operation menu displayed on the display, When input is received by pressing the pen on the first operation button, the angle information is set to the first rotation angle. A storage unit that, upon receiving input from pressing the pen on the second operation button, sets the angle information to a second rotation angle different from the first rotation angle, A second input receiving unit that receives handwritten input from the user using the pen that was pressed, When the first rotation angle is set in association with the information of the pressed pen, the handwritten data generated based on the handwriting input using the pen is recognized as a character using the first rotation angle set in association with the information of the pressed pen. When the second rotation angle is set in association with the information of the pressed pen, a character recognition unit recognizes the handwritten data generated based on the handwritten input using the pen using the second rotation angle set in association with the information of the pressed pen, An information processing device characterized by having the following features.

2. The display control unit, The first operation menu, including the first operation button, is displayed in the direction of the user who is performing handwriting input from the first edge, according to the first rotation angle. The second operation menu, including the second operation button, is displayed in the direction of the user who is performing handwriting input from the second edge, according to the second rotation angle. The information processing apparatus according to claim 1.

3. The information processing apparatus according to claim 1 or 2, characterized in that the first rotation angle is 0 degrees and the second rotation angle is 180 degrees.

4. The pen used to input handwritten data receives the pen's identification information. When input is received by pressing the first operation button included in the first operation menu with the pen, the pen's identification information and the first rotation angle are set in association, The handwritten data generated by the pen is processed and displayed using the first rotation angle associated with the pen's identification information. When input is received by pressing the second operation button included in the second operation menu, the pen's identification information and the second rotation angle are set in association, The handwritten data generated by the aforementioned pen is processed and displayed using the second rotation angle associated with the pen's identification information. The information processing apparatus according to any one of claims 1 to 3.

5. Receiving identification information of a pen from a pen for inputting handwritten data, When input is received by pressing the first operation button included in the first operation menu with the pen, the pen's identification information and the first rotation angle are set in association, The string obtained by recognizing the handwritten data is displayed according to the first rotation angle. When input is received by pressing the second operation button included in the second operation menu, the pen's identification information and the second rotation angle are set in association, The string obtained by recognizing the aforementioned handwritten data is displayed according to the second rotation angle. The information processing apparatus according to any one of claims 1 to 3.

6. The information processing device according to any one of claims 1 to 5, characterized in that it has a pen holder from which a pen can be removed from the edge along which the first operation menu and the second operation menu are displayed, or a pen holder that stores the pen parallel to the edge.

7. If you retrieve one or more files, The information processing apparatus according to any one of claims 1 to 6, characterized in that it displays the file name of the aforementioned file and the page names of each page contained in the aforementioned file.

8. If the selection of the aforementioned page name is accepted, The information processing device according to claim 7, characterized in that it displays a selected page using information in which the page name is recorded for each page of the aforementioned file.

9. The information processing device according to claim 7 or 8, characterized in that it displays the string obtained by recognizing the handwritten data as the page name of the page on which the string is displayed.

10. The information processing device according to claim 7 or 8, characterized in that, when it receives a selection of a string obtained by recognizing the aforementioned handwritten data, it displays the string as the page name of the page on which the string is displayed.

11. The information processing apparatus according to any one of claims 8 to 10, characterized in that it displays the same number of file names and page names as the first operation menu and the second operation menu.

12. The display control unit displays the file name and page name according to the first rotation angle, and further displays them according to the second rotation angle. The information processing apparatus according to any one of claims 7 to 11.

13. The system receives the pen's identification information from the pen used to accept the user's handwritten input. When the file name or page name included in the first operation menu is pressed, the pen identification information and the first rotation angle are stored in association with each other. The display control unit displays the string obtained by recognizing the handwritten data according to the first rotation angle. When the file name or page name included in the second operation menu is pressed, the pen identification information and the second rotation angle are stored in association with each other. The display control unit displays the string obtained by recognizing the handwritten data according to the second rotation angle. The information processing apparatus according to feature 12.

14. The information processing apparatus according to any one of claims 1 to 13, characterized in that the display control unit displays a third button that accepts a setting of whether or not to display candidate strings obtained by recognizing the handwritten data.

15. When the third button described above is pressed, The information processing apparatus according to claim 14, characterized in that even when the aforementioned handwritten data is input, it does not display candidate strings obtained by recognizing the aforementioned handwritten data, but displays candidate operation commands to be executed by the display device.

16. The information processing apparatus according to any one of claims 1 to 15, characterized in that it displays a configuration determination screen that accepts the configuration of the first operation menu and the second operation menu.

17. An information processing method performed by an information processing device that displays a first operation menu including a first operation button and a second operation menu including a second operation button on a display, The user's pen information and the angle information used for character recognition are stored in the memory unit in a corresponding manner. The process involves displaying the first operation menu along the edge of the first of the four sides of the display, and displaying the second operation menu along the edge of the second of the four sides of the display that is opposite to the first side. A process to receive input from the user by pressing the pen to the first operation button included in the first operation menu or the second operation button included in the second operation menu displayed on the display, When input is received by pressing the pen on the first operation button, the angle information is set to the first rotation angle. When input is received by pressing the pen on the second operation button, the process of setting the angle information to a second rotation angle different from the first rotation angle, A process for receiving handwritten input from the user using the pen that was pressed, When the first rotation angle is set in association with the information of the pressed pen, the handwritten data generated based on the handwriting input using the pen is recognized as a character using the first rotation angle set in association with the information of the pressed pen. When the second rotation angle is set in association with the information of the pressed pen, the process of character recognition is performed on the handwritten data generated based on the handwriting input using the pen, using the second rotation angle set in association with the information of the pressed pen. An information processing method characterized by performing the following.

18. In an information processing device, A process that associates information about the pen used by the user with angle information used for character recognition and stores it in a memory unit, A process to display a first operation menu including a first operation button along the edge of a first side of the four sides of the display, and a second operation menu including a second operation button along the edge of a second side of the four sides of the display that is opposite to the first side, A process to receive input from the user by pressing the pen to the first operation button included in the first operation menu or the second operation button included in the second operation menu displayed on the display, When input is received by pressing the pen on the first operation button, the angle information is set to the first rotation angle. When input is received by pressing the pen on the second operation button, the process of setting the angle information to a second rotation angle different from the first rotation angle, A process for receiving handwritten input from the user using the pen that was pressed, When the first rotation angle is set in association with the information of the pressed pen, the handwritten data generated based on the handwriting input using the pen is recognized as a character using the first rotation angle set in association with the information of the pressed pen. When the second rotation angle is set in association with the information of the pressed pen, the process of character recognition is performed on the handwritten data generated based on the handwriting input using the pen, using the second rotation angle set in association with the information of the pressed pen. A program to execute.

19. The information processing device, The first operation menu, including the first operation button, is displayed in the direction of the user who is performing handwriting input from the first edge, according to the first rotation angle. The second operation menu, including the second operation button, is displayed in the direction of the user who is performing handwriting input from the second edge, according to the second rotation angle. The program according to feature 18.

20. The program according to claim 18 or 19, characterized in that the first rotation angle is 0 degrees and the second rotation angle is 180 degrees.

21. The information processing device, The pen used to input handwritten data receives the pen's identification information. When input is received by pressing the first operation button included in the first operation menu, the identification information of the pen and the first rotation angle are set in association. The handwritten data generated by the pen is processed and displayed using the first rotation angle associated with the pen's identification information. When input is received by pressing the second operation button included in the second operation menu, the pen's identification information and the second rotation angle are set in association. The handwritten data generated by the aforementioned pen is processed and displayed using the second rotation angle associated with the pen's identification information. The program according to any one of claims 18 to 20.

22. The information processing device, The pen used to input handwritten data receives the pen's identification information. When input is received by pressing the first operation button included in the first operation menu, the identification information of the pen and the first rotation angle are set in association. The string obtained by recognizing the handwritten data is displayed according to the first rotation angle. When input is received by pressing the second operation button included in the second operation menu, the pen's identification information and the second rotation angle are set in association. The string obtained by recognizing the aforementioned handwritten data is displayed according to the second rotation angle. The program according to any one of claims 18 to 20.

23. The information processing device is The program according to any one of claims 18 to 22, characterized in that it has a pen holder from which a pen can be removed from the edge along which the first operation menu and the second operation menu are displayed, or a pen holder that stores the pen parallel to the edge.

24. When the program obtains one or more files, The aforementioned information processing device, The program according to any one of claims 18 to 23, characterized in that it displays the file name of the aforementioned file and the page names of each page contained in the aforementioned file.

25. When the program receives the selection of the page name, The aforementioned information processing device, The program according to claim 24, characterized in that it displays a selected page using information in which the page name is recorded for each page of the aforementioned file.

26. The information processing device, The program according to claim 24 or 25, characterized in that it displays the string obtained by recognizing the handwritten data as the page name of the page on which the string is displayed.

27. ​​When the program receives a selection of a string obtained by recognizing the handwritten data, The aforementioned information processing device, The program according to claim 24 or 25, characterized in that it displays the aforementioned string as the page name of the page displaying the string.

28. The information processing device, The program according to any one of claims 25 to 27, characterized in that it displays the same number of file names and page names as the first operation menu and the second operation menu.

29. The information processing device, The file name and page name are displayed according to the first rotation angle, and further displayed according to the second rotation angle. The program according to any one of claims 24 to 28.

30. The information processing device, The system receives the pen's identification information from the pen used to accept the user's handwritten input. When the file name or page name included in the first operation menu is pressed, the pen identification information and the first rotation angle are stored in association with each other. The string obtained by recognizing the handwritten data is displayed according to the first rotation angle. When the file name or page name included in the second operation menu is pressed, the pen identification information and the second rotation angle are stored in association with each other. The string obtained by recognizing the aforementioned handwritten data is displayed according to the second rotation angle. The program according to feature 29.

31. The information processing device, The program according to any one of claims 18 to 30, characterized in that it displays a third button that accepts a setting for whether or not to display candidate strings obtained by recognizing the aforementioned handwritten data.

32. When the program receives confirmation that the third button has been pressed, The aforementioned information processing device, The program according to claim 31, characterized in that, even when the aforementioned handwritten data is input, it does not display candidate strings obtained by recognizing the aforementioned handwritten data, but displays candidate operation commands to be executed on the display device.

33. The information processing device, The program according to any one of claims 18 to 32, characterized in that it displays a configuration determination screen that accepts the configuration of the first operation menu and the second operation menu.

Images