Display device, display method, program, and display system

Abstract

To allow am operation command associated with a selection gesture to be executed on a character string input in an input field where a handwritten input is converted into a predetermined character type.SOLUTION: A display device configured to allow an operation command associated with a selection gesture to be executed on a character string input in an input field, includes: a handwritten input unit which receives a handwritten input; an operation command recognition control unit which determines whether the handwritten input received by the handwritten input unit is a selection gesture or not; an input field defining unit which determines, when the handwritten input is determined to be a selection gesture, whether a character string has been registered in an input field where a handwritten input is converted into a predetermined character type; and an input display control unit which displays, when the character string has been registered in the input field, an operation command associated with the selection gesture so as to be selected, and converts, when the character string has not been registered in the input field, the handwritten input into the predetermined character type.SELECTED DRAWING: Figure 25

Description

[Technical Field]

[0001] The present invention relates to a display device, a display method, a program, and a display system. [Background technology]

[0002] Among display devices that display an input field on a display screen, there is known a display device that recognizes characters written by a user in the input field, converts the characters into a string, and displays the converted string in the input field. In the display device, when the user writes more handwriting on a string entered in the input field, the entered string is selected and the selected string is displayed so that it can be edited. For example, when the user draws a handwritten line over a string to be entered in the input field, the string to be entered in the input field is selected and displayed on the display screen. The user can edit the selected string, such as by enlarging or reducing the display size of the string.

[0003] There is also known a technology for translating content handwritten by a user (see, for example, Patent Document 1). Patent Document 1 discloses a technology for character recognition of content handwritten by a user (recognizing it as Japanese) and displaying it on a screen, as well as translating the recognized character string into another language (English) and displaying it on the screen. Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, there are cases where a user cannot select a character string entered in an input field. For example, a function for forcibly converting handwritten input may be set in an input field. In an input field with forcible conversion, handwritten input is automatically forcibly converted to a character type previously set in the input field. In this case, when a user writes handwriting in an input field in which a character string has already been entered, the previously entered character string is not selected in the input field where forcible conversion of handwritten input is set, and the handwritten content is forcibly converted to a character type previously set and displayed on the screen. A selection gesture for selecting a character string already entered in this input field is also converted to the corresponding character type and entered into the input field.

[0005] In view of the above-mentioned problems, the present invention provides a technique that enables an operation command associated with a selection gesture to be executed for a character string entered in an input field that converts handwritten input into a predetermined character type. [Means for solving the problem]

[0006] The present invention is a display device capable of executing an operation command associated with a selection gesture for a character string entered in an input field, comprising: a handwriting input unit that accepts handwritten input; an operation command recognition control unit that determines whether the handwritten input accepted by the handwriting input unit is a selection gesture; an input field definition unit that, if it is determined that the handwritten input is a selection gesture, determines whether a character string is registered in an input field in which the handwritten input is converted into a predetermined character type; and a handwriting input display control unit that, if it is determined that the character string is registered in the input field, selectably displays an operation command associated with the selection gesture, and, if it is determined that the character string is not registered in the input field, converts the handwritten input into a predetermined character type. [Effects of the Invention]

[0007] It is possible to make it possible to execute an operation command associated with a selection gesture for a character string input in an input field that converts handwritten input into a predetermined character type. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 10 is a diagram illustrating an operation guide for displaying conversion candidates. [Figure 2] 10A and 10B are diagrams illustrating handwritten data in a forced conversion input field and the conversion result. [Figure 3] 10A and 10B are diagrams illustrating the inconvenience of a selection gesture in a forced conversion input field being forcibly converted; [Figure 4] 10A and 10B are diagrams illustrating a method by which a user can select a character string displayed in a forced conversion input field on a display device. [Figure 5] FIG. 1 is a diagram showing an example of the arrangement of display devices. [Figure 6] FIG. 2 is a diagram illustrating a hardware configuration of an example of a display device. [Figure 7] FIG. 2 is a functional block diagram illustrating an example of functions of the display device in blocks. [Figure 8] FIG. 10 is a diagram illustrating an example of predefined control data. [Figure 9] FIG. 4 is a diagram showing an example of dictionary data in a handwriting recognition dictionary unit. [Figure 10] FIG. 10 is a diagram illustrating an example of dictionary data in a character string conversion dictionary unit. [Figure 11] FIG. 10 is a diagram illustrating an example of dictionary data in a predictive conversion dictionary unit. [Figure 12] 10A and 10B are diagrams illustrating an example of operation command definition data and system definition data held by an operation command definition unit. [Figure 13] 10A and 10B are diagrams showing an example of operation command definition data in the case where a selection object is selected by handwritten data; [Figure 14] 10 is a diagram showing an example of input field definition data stored in an input field definition section; FIG. [Figure 15] FIG. 10 is a diagram showing an example of a display of an operation guide. [Figure 16] 10A and 10B are diagrams illustrating an example of a method for selecting a character string. [Figure 17] 10A and 10B are diagrams illustrating examples of display of operation command candidates based on operation command definition data. [Figure 18] FIG. 10 is a sequence diagram (part 1) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 19] FIG. 10 is a sequence diagram (part 2) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 20] FIG. 10 is a sequence diagram (part 3) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 21] FIG. 10 is a sequence diagram (part 4) illustrating an example of a process in which the display device displays character string candidates and operation command candidates. [Figure 22] FIG. 5 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 5). [Figure 23] FIG. 6 is a sequence diagram of an example illustrating a process in which the display device displays character string candidates and operation command candidates (part 6). [Figure 24] 10A and 10B are diagrams illustrating an example of conversion when a selection gesture is converted into a character string. [Figure 25] 10 is a flowchart illustrating an example of a process for determining whether handwriting in an input field is a character string selection or forced conversion. [Figure 26] 10A and 10B are diagrams illustrating an example of handwriting overwriting a character string displayed in a forced conversion input field. [Figure 27] FIG. 10 is an example of a flowchart illustrating a process for determining whether handwriting in an input field is a character string selection or a forced conversion, depending on whether the selection gesture crosses a ruled line. [Figure 28] 10A and 10B are diagrams illustrating a method for determining whether a selection gesture crosses a ruled line that forms a forced conversion input field. [Figure 29]FIG. 10 is an example of a flowchart illustrating a process for determining whether handwriting in an input field is a character string selection or a forced conversion, depending on whether the selection gesture surrounds a ruled line. [Figure 30] 10A and 10B are diagrams illustrating a method for determining whether a selection gesture surrounds a ruled line that forms a forced conversion input field. [Figure 31] 10A and 10B are diagrams illustrating an operation guide that is displayed when it is determined that the selection gesture has selected a character string displayed in a forced conversion input field. [Figure 32] FIG. 2 is an example of a functional block diagram of a display device. [Figure 33] FIG. 10 is a flowchart illustrating an example of changing the conditions based on the determination result of whether the handwritten data in the forced conversion input field is a selection gesture or a forced conversion. [Figure 34] FIG. 10 is a diagram showing an example of a determination result screen. [Figure 35] FIG. 10 is a diagram illustrating the relationship between the intersections and the four sides. [Figure 36] FIG. 1 is a diagram illustrating an example of a system configuration of a display system. [Figure 37] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing system. [Figure 38] FIG. 2 is an example of a functional block diagram illustrating functions of the display system in blocks. [Figure 39] 10 is an example of a sequence diagram illustrating a process in which a display device and an information processing system communicate with each other to display a recognition result of handwritten data. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A display device and a display method performed by the display device will be described below as an example of an embodiment of the present invention with reference to the accompanying drawings.

[0010] <Input into forced conversion form> Computer-controlled whiteboard devices or applications that allow handwriting input (hereinafter referred to as display devices) have touch panels that allow users to write by hand. The display devices can recognize handwritten data and convert it into a string of characters.

[0011] First, the forced conversion function of the display device will be described with reference to Figs. 1 and 2. Fig. 1 is a diagram illustrating an operation guide that displays conversion candidates. The display device 2 generates conversion candidates from handwritten data and displays them in the operation guide. For handwritten data 302 of "18" in an input field 301 as shown in Fig. 1(a), the conversion candidates "18," "18 years old," "January 8th," and "1:08" are displayed in the operation guide 500 as shown in Fig. 1(b). When the user selects a conversion candidate, the selected character string 303 is displayed in the input field 301 as shown in Fig. 1(c).

[0012] In contrast, when handwriting is done in a forced conversion input field where the operation guide 500 is not displayed, the character string is forcibly converted to a character string for which the character types to be converted are limited in advance. In other words, when the user types in a forced conversion input field, the operation of selecting a conversion candidate can be omitted.

[0013] FIG. 2 is a diagram illustrating handwritten data entered into a forced conversion input field and the conversion result. The input form 300 in FIG. 2 has three input fields, and the input field 301 to the right of "age" is the forced conversion input field. As an example, handwritten data 302 entered into the forced conversion input field is converted into a number. In FIG. 2(a), handwritten data 302 "18" is entered into input field 301. In the forced conversion input field, the display device 2 automatically and forcibly converts the handwritten data 302 into the character string that most closely matches the character type pre-set in the forced conversion input field. In other words, the operation guide 500 is not displayed. Note that the character types pre-set in the forced conversion input field include numbers, alphabets, hiragana, katakana, symbols, etc.

[0014] Selection of a character string when input into a forced conversion input field will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining the inconvenience of forced conversion occurring when a selection gesture into a forced conversion input field is made. FIG. 3(a) shows an input form 300 before handwriting. FIG. 3(b) shows an input form 300 in which a user has handwritten handwritten data 302 of "18" in an input field 301, which is a forced conversion input field. FIG. 3(c) shows the input form 300 in which the display device has forcibly converted "18" into a number (character string 303). 3(d) shows a selection gesture 304 for selecting "18" displayed in the forced conversion input field. The selection gesture 304 is not a stroke for input, but a stroke for selecting a character string. If a stroke is not determined to be a selection gesture 304, it is determined to be simple handwritten data, even if it has the same shape. As shown in Figure 3(e), when the user handwrites a selection gesture 304 in the input field 301, which is a forced conversion input field, the display device 2 recognizes the selection gesture 304 as "0," which best matches the shape of the numeral. The display device 2 overwrites the character string 303 in the forced conversion input field with the character string "0" 305, deleting "18" and leaving only "0."

[0015] Thus, the display device 2 has the disadvantage that the user cannot select the character string displayed in the forced conversion input field with a selection gesture. Furthermore, if the selection is not possible, the operation guide 500 is not displayed, and the user cannot perform operations using the character string, such as deleting or moving the character string "18," changing the font, or setting the character string as a page name.

[0016] <How to select characters entered in the forced conversion form> Therefore, the display device 2 of this embodiment allows the user to select a character string displayed in the forced conversion input field in the following manner, which will be described with reference to FIG.

[0017] 4A and 4B are diagrams illustrating a method by which a user selects a character string displayed in a forced conversion input field on the display device of this embodiment. In Fig. 4A, a character string 303 ("18") displayed in the forced conversion input field is surrounded by a selection gesture 304. When the selection gesture 304 is handwritten for a character string 303 already displayed in the forced conversion input field, the display device of this embodiment does not forcibly convert the selection gesture 304.

[0018] 4(b), the display device displays an operation guide 500 including operation commands for operating the character string 303 selected by the selection gesture 304. The user can select an operation for the character string 303 from the operation guide 500.

[0019] In this way, when a selection gesture is handwritten in a forced conversion input field in which a string of characters is already displayed, the display device of this embodiment does not forcibly convert the selection gesture into a string of characters, so that the user can select the string of characters displayed in the forced conversion input field with the selection gesture.

[0020] <Terminology> A selection gesture is a stroke that allows a user to select a character string that is already displayed. A selection gesture includes a stroke that encircles a character string. Note that selection gestures are not limited to encircling strokes, as it depends on the definition of what strokes are considered selection gestures and what strokes the display device considers to be selection gestures. For example, a selection gesture may be a straight stroke handwritten on top of a character string.

[0021] A stroke is a series of operations in which a user presses a pen against a touch panel, moves it continuously, and then lifts it from the touch panel. Stroke data is information displayed on the touch panel based on the trajectory of coordinates input by an input device. Stroke data may be interpolated as appropriate.

[0022] An object is an object that is displayed on a display based on stroke data. Object means a target, but in this embodiment it means a display target.

[0023] Handwritten data is an object consisting of one or more strokes, and refers to the display object written by the user. A selected object is an object that can be selected by drawing a box, etc. A character string object is an object in which handwritten data has been converted into a character string using character recognition.

[0024] A character string is an object converted by character recognition of handwritten data. A character string can be represented by a character code. A character string and handwritten data that has been determined not to be converted to a character code can be selected using a selection gesture. A character string consists of one or more characters, and characters include hiragana, katakana, kanji, alphabets, numbers, and symbols.

[0025] An operation command is a command that instructs the execution of a specific process and is prepared for operating a display device. In this embodiment, an operation command for editing, modification, or input / output is taken as an example, but all commands that operate a display device, such as screen inversion, page switching, and setting of an operation mode, are subject to the operation command.

[0026] Forced conversion means that handwritten data is converted into a string of characters without displaying an operation guide. When no operation guide is displayed, it does not matter what character type is converted into. Furthermore, forced conversion may convert handwritten data into a different character type (for example, numbers and letters). Character type refers to the type of character. Character types include numbers, letters, hiragana, katakana, symbols, kana, full-width characters, half-width characters, etc.

[0027] <Device configuration example> The overall configuration of the display device 2 according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the arrangement of the display device 2. Fig. 5(a) shows, as an example of the display device 2, a display device 2 used as a horizontally long electronic whiteboard hung on a wall.

[0028] 5(a), a display 220 as an example of the display device 2 is provided on top of the display device 2. A user U can input characters and the like by handwriting on the display 220 using a pen 2500.

[0029] FIG. 5(b) shows a display device 2 used as a vertically long electronic whiteboard hung on a wall.

[0030] 5(c) shows the display device 2 placed flat on a desk 230. The display device 2 is about 1 cm thick, so there is no need to adjust the height of the desk even when it is placed flat on a regular desk. In addition, the user can easily move the display device 2.

[0031] The pen 2500 can input coordinates using electromagnetic induction, active electrostatic coupling, etc. The pen 2500 may also have functions such as pressure detection, tilt detection, and a hover function (displaying the cursor before the pen touches the screen).

[0032] <Device hardware configuration> Next, the hardware configuration of the display device 2 will be described with reference to Fig. 6. As shown in the figure, the display device 2 has the configuration of an information processing device or a computer. Fig. 6 is an example of a hardware configuration diagram of the display device 2. As shown in Fig. 6, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an SSD (Solid State Drive) 204.

[0033] Of these, the CPU 201 controls the overall operation of the display device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201.

[0034] The SSD 204 stores various data such as an OS and programs for the display device 2. This program may be an application program that runs on an information processing device equipped with a general-purpose OS (Windows (registered trademark), Mac OS (registered trademark), Android (registered trademark), iOS (registered trademark), etc.). In this case, the device is normally used as a general-purpose information processing device, but when a user runs the application program, the user can input handwritten text in the same way as on a dedicated device for the display device 2.

[0035] The display device 2 also includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, a power switch 227, a tilt sensor 217, a serial interface 218, a speaker 219, a microphone 221, a wireless communication device 222, an infrared I / F 223, a power control circuit 224, an AC adapter 225, and a battery 226.

[0036] The display controller 213 controls and manages the screen display in order to output an output image to the display 220, etc. The touch sensor 216 detects that the pen 2500, the user's hand, etc. (the pen and the user's hand are input means) has come into contact with the display 220. The touch sensor 216 also receives a pen ID.

[0037] The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 inputs and detects coordinates. The method for inputting and detecting coordinates will be described below. For example, in the case of an optical type, two light receiving and emitting devices installed at both ends of the upper side of the display 220 emit a plurality of infrared rays parallel to the display 220. The two light receiving and emitting devices receive light that is reflected by a reflecting member installed around the periphery of the display 220 and returns along the same optical path as the light emitted by the light receiving element.

[0038] The touch sensor 216 outputs position information of the infrared rays emitted by the two light receiving and emitting devices that are blocked by the object to the touch sensor controller 215, and the touch sensor controller 215 identifies the coordinate position of the contact position of the object. The touch sensor controller 215 also has a communication unit 215a and can communicate wirelessly with the pen 2500. For example, when communication is based on a standard such as Bluetooth (registered trademark), a commercially available pen can be used. If one or more pens 2500 are registered in the communication unit 215a in advance, the user can communicate with the display device 2 without having to perform connection settings to connect the pen 2500 to the display device 2.

[0039] The power switch 227 is a switch for switching ON / OFF the power of the display device 2. The tilt sensor 217 is a sensor for detecting the tilt angle of the display device 2. It is mainly used to detect whether the display device 2 is being used in the installation state shown in FIG. 5(a), FIG. 5(b), or FIG. 5(c), and can automatically change the thickness of characters, etc. depending on the installation state.

[0040] The serial interface 218 is a communication interface with the outside, such as a USB. The serial interface 218 is used for inputting information from the outside, etc. The speaker 219 is used for outputting audio, and the microphone 221 is used for inputting audio. The wireless communication device 222 communicates with a terminal carried by the user, and relays a connection to the Internet, for example. The wireless communication device 222 communicates using Wi-Fi, Bluetooth (registered trademark), or the like, but any communication standard is acceptable. The wireless communication device 222 forms an access point, and when the user sets the SSID (Service Set Identifier) ​​and password they have obtained in the terminal they carry, they can connect to the access point.

[0041] Preferably, the wireless communication device 222 is provided with two access points. a. Access point → Internet b. Access point → Internal network → Internet Access point a is for users outside the company, and users cannot access the company network but can use the Internet. Access point b is for users inside the company, and users can use the company network and the Internet.

[0042] The infrared I / F 223 detects the display device 2 arranged adjacently. The infrared I / F 223 can detect only the display device 2 arranged adjacently by taking advantage of the rectilinearity of infrared rays. It is preferable to provide one infrared I / F 223 on each side, and it can detect in which direction another display device 2 is arranged relative to the display device 2. This expands the screen, making it possible to display handwritten information that was previously handwritten on the adjacent display device 2 (handwritten information on another page, with the area of ​​one display 220 being one page), etc.

[0043] The power supply control circuit 224 controls an AC adapter 225 and a battery 226, which are power sources for the display device 2. The AC adapter 225 converts AC common to commercial power sources into DC.

[0044] If the display 220 is so-called electronic paper, it consumes little or no power to maintain the image display, and therefore can be driven by a battery 226. This makes it possible to use the display device 2 for purposes such as digital signage even in places where it is difficult to connect to a power source, such as outdoors.

[0045] The display device 2 further includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201 shown in FIG.

[0046] The touch sensor 216 is not limited to an optical type, and may be a capacitive touch panel that identifies the contact position by detecting changes in capacitance. The touch sensor 216 may also be a resistive film type touch panel that identifies the contact position by detecting voltage changes across two opposing resistive films. The touch sensor 216 may also be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction generated when a contacting object touches the display unit, or various other detection means may be used. The touch sensor 216 may be of a type that does not require an electronic pen to detect whether or not the pen tip is touching. In this case, a fingertip or a pen-shaped stick can be used to perform a touch operation. The pen 2500 does not have to be a long, thin pen-shaped panel.

[0047] [First embodiment] <About the device functions> Next, functions of the display device 2 will be described with reference to FIG. 7. FIG. 7 is an example of a functional block diagram showing functions of the display device 2 in blocks. The display device 2 includes a handwriting input unit 21, a display unit 22, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a character string conversion control unit 28, a character string conversion dictionary unit 29, a predictive conversion control unit 30, a predictive conversion dictionary unit 31, an operation command recognition control unit 32, an operation command definition unit 33, an input field definition unit 34, and a ruled line determination unit 35. Each function of the display device 2 is realized by any of the components shown in FIG. 6 operating in response to an instruction from the CPU 201 in accordance with a program loaded from the SSD 204 onto the RAM 203.

[0048] The handwriting input unit 21 is realized by a touch sensor 216 or the like, and accepts handwriting input by the user. The handwriting input unit 21 converts the user's pen input d1 into pen operation data d2 (pen up, pen down, or pen coordinate data) and transmits it to the handwriting input display control unit 23. The pen coordinate data is periodically transmitted as discrete values, and coordinates between the discrete values ​​are interpolated.

[0049] The display unit 22 is realized by a display 220 or the like, and displays handwritten objects and operation menus. The display unit 22 converts the drawing data d3 written to the video memory by the handwriting input display control unit 23 into data according to the characteristics of the display 220, and transmits the data to the display 220.

[0050] The handwriting input display control unit 23 performs overall control over handwriting input and display. The handwriting input display control unit 23 processes the pen operation data d2 from the handwriting input unit 21 and transmits it to the display unit 22 for display. Details of the processing of the pen operation data d2 and the display of strokes will be described later with reference to Figs. 18 to 23.

[0051] The candidate display timer control unit 24 is a timer for controlling the display of selectable candidates. The candidate display timer control unit 24 starts or stops the timer to generate the timing for starting the display of selectable candidates and the timing for erasing the display. The candidate display timer control unit 24 receives a timer start request d4 (which may also be a timer stop request) from the handwriting input display control unit 23, and transmits a timeout event d5 to the handwriting input display control unit 23.

[0052] The handwriting input saving unit 25 has a storage function for saving user data (handwriting data / character string objects). The handwriting input saving unit 25 receives user data d6-1 from the handwriting input display control unit 23 and saves it in the handwriting input saving unit 25, and receives an acquisition request d6-2 from the handwriting input display control unit 23 and transmits the user data d7 saved in the handwriting input saving unit 25. The handwriting input saving unit 25 transmits character string position information d36 to the operation command recognition control unit 32.

[0053] The handwriting recognition control unit 26 is a recognition engine that performs online handwriting recognition. Unlike a typical OCR (Optical Character Reader), it recognizes characters (not only Japanese but also multiple languages ​​such as English), numbers, symbols (%, $, &, etc.), shapes (lines, circles, triangles, etc.) in parallel with the user's pen operation. Various algorithms have been devised for recognition methods, but this embodiment uses known techniques and will not be described in detail.

[0054] The handwriting recognition control unit 26 receives the pen operation data d8-1 from the handwriting input display control unit 23, performs handwriting recognition, and stores handwriting-recognized character string candidates. The handwriting recognition control unit 26 also stores language character string candidates converted from the handwriting-recognized character string candidates d12 using the handwriting recognition dictionary unit 27. Separately, when an acquisition request d8-2 is received from the handwriting input display control unit 23, the handwriting recognition control unit 26 transmits the stored handwriting-recognized character string candidates and language character string candidates d9 to the handwriting input display control unit 23.

[0055] The handwriting recognition dictionary unit 27 stores dictionary data for language conversion in handwriting recognition. The handwriting recognition dictionary unit 27 receives handwriting recognition character string candidates d12 from the handwriting recognition control unit 26, converts them into linguistically probable language character string candidates d13, and transmits them to the handwriting recognition control unit 26. For example, in the case of Japanese, hiragana characters are converted into kanji characters or katakana characters.

[0056] The character string conversion control unit 28 controls the conversion of conversion character string candidates into character strings. A conversion character string is a character string that is likely to be generated by including a handwriting recognition character string or a language character string. The character string conversion control unit 28 receives the handwriting recognition character string and language character string candidates d11 from the handwriting recognition control unit 26, converts them into conversion character string candidates using the character string conversion dictionary unit 29, and stores them. When the character string conversion control unit 28 separately receives an acquisition request d14 from the handwriting input display control unit 23, it transmits the stored conversion character string candidates d15 to the handwriting input display control unit 23.

[0057] The character string conversion dictionary unit 29 stores dictionary data for character string conversion. The character string conversion dictionary unit 29 receives handwriting-recognized character strings and language character string candidates d17 from the character string conversion control unit 28, and transmits converted character string candidates d18 to the character string conversion control unit 28.

[0058] The predictive conversion control unit 30 receives the handwriting-recognized character string and language character string candidate d10 from the handwriting recognition control unit 26 and receives the converted character string candidate d16 from the character string conversion control unit 28, and converts each of them into a predicted character string candidate using the predictive conversion dictionary unit 31. A predicted character string candidate is a character string that is likely to be generated by including the handwriting-recognized character string, language string, or converted character string. When the predictive conversion control unit 30 separately receives an acquisition request d19 from the handwriting input display control unit 23, it transmits the predicted character string candidate d20 to the handwriting input display control unit 23.

[0059] The predictive conversion dictionary unit 31 stores dictionary data for predictive conversion. The predictive conversion dictionary unit 31 receives handwritten recognition character strings, language character string candidates, and conversion character string candidates d21 from the predictive conversion control unit 30, and transmits predicted character string candidates d22 to the predictive conversion control unit 30.

[0060] The operation command recognition control unit 32 receives the handwriting recognition character string and language character string candidates d30 from the handwriting recognition control unit 26, receives the converted character string candidates d28 from the character string conversion control unit 28, and receives the predicted character string candidates d29 from the predictive conversion control unit 30. Then, the operation command recognition control unit 32 transmits operation command conversion requests d26 for each of the characters to the operation command definition unit 33, and receives operation command candidates d27 from the operation command definition unit 33. The operation command recognition control unit 32 holds the operation command candidates d27.

[0061] If the operation command conversion request d26 partially matches the operation command definition, the operation command definition unit 33 sends operation command candidates d27 to the operation command recognition control unit 32.

[0062] The operation command recognition control unit 32 also receives pen operation data d24-1 from the handwriting input display control unit 23, transmits a request d23 to acquire position information of a character string that was previously input and confirmed to the handwriting input storage unit 25, and stores the character string specified by the pen operation data as a selected object (including position information). The operation command recognition control unit 32 identifies a selected object that satisfies the position of the pen operation data d24-1 and a predetermined criterion. When the operation command recognition control unit 32 separately receives an acquisition request d24-2 from the handwriting input display control unit 23, it transmits a selected object d25, identified as a candidate for the stored operation command, to the handwriting input display control unit 23.

[0063] The input field definition unit 34 defines, for each input field of the input form 300, etc., whether the input field is a forced conversion input field. The input field definition unit 34 returns whether the handwritten input field is a forced conversion input field. In other words, the input field definition unit 34 determines whether the handwritten input accepted by the handwriting input unit 21 is handwritten input into an input field that is to be forcibly converted into a predetermined character type. The input field definition unit 34 also defines information related to the input field. The ruled line determination unit 35 determines the relationship between the handwritten data and the ruled lines that form the input field. For example, the ruled line determination unit 35 determines whether the handwritten data crosses or surrounds a ruled line. The determination of the ruled line determination unit 35 is used to determine whether a selection gesture has occurred. The determination result may be used to determine whether a character string displayed in the forced conversion input field has been selected.

[0064] <About predefined control data> Next, the predefined control data used by the display device 2 for various processes will be described with reference to Fig. 8. Fig. 8 shows an example of the predefined control data. In the example of Fig. 8, control data is shown for each control item.

[0065] The selectable candidate display timer 401 defines the time until the selectable candidates are displayed (an example of a first time). This is because the display unit 22 does not display the selectable candidates while handwriting is in progress. In FIG. 8, this means that the selectable candidates are displayed if the pen does not come down within TimerValue=500 [ms] after the pen is up. The selectable candidate display timer 401 is held by the candidate display timer control unit 24. The selectable candidate display timer 401 is used when the selectable candidate display timer starts in step S4.8 of FIG. 20, which will be described later.

[0066] The selectable candidate deletion timer 402 defines the time until the displayed selectable candidates are deleted (an example of a second time). This is to delete the selectable candidates if the user does not select a selectable candidate. In FIG. 8, this means that if a selectable candidate is not selected within TimerValue=5000 [ms] from the display of the selectable candidates, the selectable candidate display will be deleted. The selectable candidate deletion timer 402 is held by the candidate display timer control unit 24. The selectable candidate deletion timer 402 is used when the selectable candidate display deletion timer starts in step S4.8.1.8 of FIG. 21.

[0067] The handwritten data neighborhood rectangular area 403 defines a rectangular area that is considered to be near the handwritten data. In the example of FIG. 8, the handwritten data neighborhood rectangular area 403 is a rectangular area obtained by enlarging the handwritten data rectangular area by 50% (Horizontal) of the estimated character size in the horizontal direction and 80% (Vertical) of the estimated character size in the vertical direction. In the example of FIG. 8, the rectangular area is a percentage of the estimated character size (specified as a percentage), but it can also be a fixed length by using units such as "mm". The handwritten data neighborhood rectangular area 403 is stored in the handwritten input storage unit 25. The estimated character size 405 is used to determine the overlap state between the handwritten data neighborhood rectangular area and the stroke rectangular area in step S4.2 of FIG. 19.

[0068] The estimated writing direction / character size determination condition 404 defines constants for determining the writing direction and the measurement direction of the character size. In the example of FIG. 8, if the difference between the time when the first stroke was added to the rectangular area of ​​handwritten data and the time when the last stroke was added is MinTime=1000 [ms] or more, the difference between the horizontal distance (width) and the vertical distance (height) of the rectangular area of ​​handwritten data is MinDiff=10 [mm] or more, and the horizontal distance is longer than the vertical distance, the estimated writing direction is determined to be "horizontal writing" and the estimated character size is determined to be the vertical distance. If the horizontal distance is shorter than the vertical distance, the estimated writing direction is determined to be "vertical writing" and the estimated character size is determined to be the horizontal distance. If these conditions are not met, the estimated writing direction is determined to be "horizontal writing" (DefaultDir="Horizontal") and the estimated character size is determined to be the longer of the horizontal distance and the vertical distance. The estimated writing direction / character size determination condition 404 is held in the handwriting input storage unit 25. The estimated writing direction / character size judgment condition 404 is used to obtain the estimated writing direction in step S4.8.1.5 of FIG. 21 and to obtain the character string object font in step S7.8.

[0069] The estimated character size 405 defines data for estimating the size of characters, etc. In the example of FIG. 8, this means that the estimated character size determined by the estimated writing direction / character size determination condition 404 is compared with the small character 405a (hereinafter referred to as the minimum font size) and the large character 405c (hereinafter referred to as the maximum font size) of the estimated character size 405. If the estimated character size is smaller than the minimum font size, the estimated character size is determined to be the minimum font size. If the estimated character size is larger than the maximum font size, the estimated character size is determined to be the maximum font size. Otherwise, the estimated character size is determined to be the character size of the medium character 405b. The estimated character size 405 is stored in the handwriting input storage unit 25. The estimated character size 405 is used to obtain the character string object font in step S7.8 of FIG. 23.

[0070] Specifically, the handwriting input storage unit 25 compares the estimated character size determined by the estimated writing direction / character size determination condition 404 with the FontSize of the estimated character size 405, and uses the font with the closest size. For example, if the estimated character size is 25 mm (FontSize for small characters) or less, it is determined to be a "small character," if the estimated character size is more than 25 mm but less than 50 mm (FontSize for medium characters), it is determined to be a "medium character," and if the estimated character size is more than 100 mm (FontSize for large characters), it is determined to be a "large character." "Small characters" 405a use a 25 mm Mincho font (FontStyle="Mincho" FontSize="25mm"), "medium characters" 405b use a 50 mm Mincho font (FontStyle="Mincho" FontSize="50mm"), and "large characters" 405c use a 100 mm Gothic font (FontStyle="Gothic" FontSize="100mm"). If you want more font sizes or styles, you can increase the number of estimated font sizes 405.

[0071] The crossing line determination condition 406 defines data used to determine whether multiple objects have been selected. In the example of FIG. 8, if the handwritten data is a single stroke and the length of the long side of the handwritten data is 100 mm or more (MinLenLongSide="100mm"), the length of the short side is 50 mm or less (MaxLenShortSide="50mm"), and there is an object whose overlap rate with the handwritten data in the long and short side directions is 80% or more (MinOverLapRate="80%") (a predetermined overlap rate or more), it is determined that multiple objects have been selected (selected objects). Instead of "more than or equal to," the overlap rate may be determined to be "greater than." The crossing line determination condition 406 is held by the operation command recognition control unit 32. The crossing line determination condition 406 is used in the crossing line determination for determining the selected object in step S4.7.6.2 of FIG. 20.

[0072] The encircling line determination condition 407 defines data used to determine whether an object is an encircling line. In the example of FIG. 8, the operation command recognition control unit 32 determines that a character string in which the overlap rate in the long side direction and short side direction of the handwritten data is 100% or more (MinOverLapRate="100%") is a selected object (a predetermined overlap rate or more). Instead of "above or above," the overlap rate may be determined to be "greater than." The encircling line determination condition 407 is held by the operation command recognition control unit 32. The encircling line determination condition 407 is used in the encircling line determination for determining the selected object in step S4.6.7.2 of FIG. 20.

[0073] Note that either the cross-line determination condition 406 or the enclosing line determination condition 407 may be given priority in the determination. For example, if the cross-line determination condition 406 is set to a relaxed value (to make it easier to select a cross-line) and the enclosing line determination condition 407 is set to a strict value (to make it possible to select only an enclosing line), the operation command recognition control unit 32 may give priority to the enclosing line determination condition 407 in the determination.

[0074] <Example of dictionary data> The dictionary data will be described with reference to Fig. 9 to Fig. 11. Fig. 9 is an example of dictionary data in the handwriting recognition dictionary unit 27, Fig. 10 is an example of dictionary data in the character string conversion dictionary unit 29, and Fig. 11 is an example of dictionary data in the predictive conversion dictionary unit 31. These dictionary data are used in steps S4.7.2 to S4.7.4 in Fig. 20, respectively.

[0075] In this embodiment, the conversion results using the dictionary data of the handwriting recognition dictionary unit 27 in FIG. 9 are called language string candidates, the conversion results using the dictionary data of the string conversion dictionary unit 29 in FIG. 10 are called converted string candidates, and the conversion results using the dictionary data of the predictive conversion dictionary unit 31 in FIG. 11 are called predicted string candidates. For each dictionary data, "before conversion" represents the string to be searched for in the dictionary data, "after conversion" represents the converted string corresponding to the string to be searched for, and "probability" represents the probability of selection by the user. The probability is calculated from the results of users selecting each string in the past. Therefore, the probability may be calculated for each user. Various algorithms have been devised as a method of calculating probability, but the calculation may be performed using an appropriate method as appropriate, and details will be omitted.

[0076] The dictionary data of the handwriting recognition dictionary unit 27 in Figure 9 shows that handwritten "gi" is converted to "gi" with a probability of 0.55 and to "gi" with a probability of 0.4, and handwritten "gishi" is converted to "gishi" with a probability of 0.5 and to "gishi" with a probability of 0.45. The same applies to other "before conversion" character strings. In Figure 9, the "before conversion" character string is handwritten hiragana, but characters other than hiragana may also be registered as "before conversion."

[0077] 10 indicates that the character string "" is converted to "" with a probability of 0.95, and the character string "" is converted to "" with a probability of 0.85. The same applies to other "before conversion" character strings.

[0078] The dictionary data of the predictive conversion dictionary unit 31 in Fig. 11 indicates that the character string "minutes" is converted to "addressee of the minutes" with a probability of 0.65, and the character string "skill test" is converted to "approval of skill test" with a probability of 0.75. In the example of Fig. 11, all character strings before conversion are kanji, but characters other than kanji may also be registered.

[0079] The dictionary data is not language-dependent, and any character strings may be registered before and after conversion.

[0080] <Operation command definition data held by the operation command definition section> Next, the operation command definition data used by the operation command recognition control unit 32 will be described with reference to Fig. 12 and Fig. 13. Fig. 12 shows an example of the operation command definition data and system definition data held by the operation command definition unit 33.

[0081] Fig. 12(a) shows an example of operation command definition data. The operation command definition data in Fig. 12(a) is an example of operation command definition data when there is no selected object selected by handwritten data, and covers all operation commands that operate the display device 2. The operation command in Fig. 12(a) has an operation command name (Name), a string (String) that partially matches a string candidate, and an operation command string (Command) to be executed. "%~%" in the operation command string is a variable, and is associated with system definition data as shown in Fig. 12(b). In other words, "%~%" is replaced with the system definition data shown in Fig. 12(b).

[0082] First, the operation command definition data 701 indicates that the operation command name is "Read minutes template," the string that partially matches the string candidate is "minutes" or "template," and the operation command string to be executed is "ReadFile https: / / %username%:%password%@server.com / template / minutes.pdf." In this example, the operation command string to be executed contains the system definition data "%~%," and "%username%" and "%password%" are replaced by the system definition data 704 and 705, respectively. Therefore, the operation command string to be ultimately executed is the string "ReadFile https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf," which indicates that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / template / minutes.pdf" is to be read (ReadFile).

[0083] The operation command definition data 702 indicates that the operation command name is "Save to minutes folder," the strings that partially match the string candidate are "minutes" or "save," and the operation command string to be executed is "WriteFile https: / / %username%:%password%@server.com / minutes / %machinename%_%yyyy-mm-dd%.pdf." As with the operation command definition data 701, "%username%," "%password%," and "%machinename%" in the operation command string are replaced with system definition data 704 to 706, respectively. Note that "%yyyy-mm-dd%" indicates that it should be replaced with the current date. For example, if the current date is September 26, 2018, it should be replaced with "2018-09-26." The final operation command to be executed is "WriteFile https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf", which indicates that the minutes should be saved (WriteFile) in the file "https: / / taro.tokkyo:x2PDHTyS@server.com / minutes / %My-Machine_2018-09-26.pdf".

[0084] The operation command definition data 703 indicates that the operation command name is "Print", the string that partially matches the string candidate is "Print" or "Print", and the operation command string to be executed is "PrintFile https: / / %username%:%password%@server.com / print / %machinename%-"%yyyy-mm-dd%.pdf". When the operation command string is replaced in the same way as in the operation command definition data 702, the operation command to be executed finally becomes "PrintFile https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf", which indicates that the file "https: / / taro.tokkyo:x2PDHTyS@server.com / print / %My-Machine_2018-09-26.pdf" will be printed (PrintFile). In other words, the file is sent to the server. When the user makes the printer communicate with the server and specifies a file, the printer prints the contents of the file on paper.

[0085] In this way, the operation command definition data 701 to 703 can be identified from the character string candidates, and the operation command can be displayed by the user's handwriting. Also, if the user authentication is successful, "%username%", "%password%", etc. in the operation command definition data are replaced with the user information, making it possible to input and output files associated with the user.

[0086] If user authentication is not performed (including the case where authentication fails but the user can still use the display device 2), the display device 2 replaces the username and password with the pre-set username and password of the display device 2. Therefore, file input and output can be performed in association with the display device 2 without user authentication.

[0087] Next, we will explain the operation command definition data when handwritten data is present, that is, the operation command definition data for editing and modification. Fig. 13 shows an example of the operation command definition data when a selected object is selected by handwritten data. The operation command definition data in Fig. 13 has the operation command name (Name), the group name (Group) of operation command candidates, and the operation command string to be executed (Command).

[0088] The operation command definition data 707 defines operation commands for editing (Group="Edit"), and is an example of definition data for the names of operation commands for editing: "erase," "move," "rotate," and "select." In other words, these operation commands are displayed for the selected object, allowing the user to select the desired operation command.

[0089] The operation command definition data 708 defines operation commands for the decoration system (Group="Decorate"), and is an example of definition data for the names of operation commands for the decoration system: "bold", "thin", "larger", "smaller", and "underline". These operation commands are displayed for the selected object, allowing the user to select the desired operation command. In addition, color operation commands may also be displayed.

[0090] The operation command definition data 709 defines an operation command for the input field (Group="InputArea"), and is an example of definition data for "setting the character string selected in the input field as the page name." This operation command is displayed for the selected object, and the user can select the desired operation command. When this operation command is selected, the character string selected by the selection gesture 304 is set as the page name when one screen of the display device 2 is considered to be one page.

[0091] Therefore, when the user selects a selection object with handwritten data, the operation command definition data 707, 708, and 709 are specified, and the user can display the operation command by handwriting.

[0092] <Input field definition data held by the input field definition unit 34> Fig. 14 is an example of input field definition data stored in the input field definition unit 34. The input field definition data is information about each input field in the input form 300. The input field definition data in Fig. 14 is stored for each input field. The input field ID is the identification information for the input field. The coordinates item indicates the coordinates of the input field on the display of the display device 2 (for example, the x and y coordinates of the upper left vertex). The size item is the length and width of the input field. The font item specifies the font (an example of an attribute) of the text displayed in the input field. The font size item specifies the size of the text displayed in the input field (an example of an attribute). The font color item specifies the color of the text displayed in the input field (an example of an attribute). The font alignment item specifies the alignment of the text displayed in the input field horizontally (left, center, right) and vertically (top, center, bottom) (an example of an attribute). The input field margin item specifies the amount of space between the text and the input field frame in large, medium, or small. The "Specified Dictionary" item specifies the conversion dictionary used to convert handwritten data entered in the input field. The display device 2 can improve recognition accuracy by registering a dictionary for each input field. The "Number of Displayable Characters" item is the maximum number of characters that can be displayed in the input field. The string item is the string entered in the input field. The operation guide item defines whether to display an operation guide before entering a string into the input field. An input field without this item hidden is a forced conversion input field.

[0093] <Example of displaying operation command candidates> FIG. 15 shows an example of an operation guide 500 and selectable candidates 530 displayed by the operation guide 500. The operation guide 500 is displayed when the user writes handwritten data 504 (when the selectable candidate display timer times out). The operation guide 500 has an operation header 520, operation command candidates 510, handwriting-recognized character string candidates 506, conversion character string candidates 507, predicted character string candidates 508, and a handwriting data rectangular area display 503. The selectable candidates 530 are the operation command candidates 510, handwriting-recognized character string candidates 506 (including language string candidates 571, if any), conversion character string candidates 507, and predicted character string candidates 508. The selectable candidates 530 excluding the operation command candidates 510 are referred to as character string candidates 539.

[0094] The operation header 520 has buttons 501, 509, 502, and 505. Button 501 accepts an operation to switch predictive conversion on and off. In the example of FIG. 15, when the user presses button 501 displaying "Predict," handwriting input unit 21 accepts the press and notifies handwriting input display control unit 23 of this, and display unit 22 changes the display to button 501 displaying "Kana." After the change, predictive conversion is not performed (converted character string candidate 507 and predicted character string candidate 508 are not displayed), and operation command candidate 510, handwritten recognition character string candidate 506, and language character string candidate (kanji converted from kana to kanji, such as 'gi') are displayed.

[0095] Button 502 operates the page of the candidate display. In the example of FIG. 15, there are three candidate display pages, and the first page is currently displayed. Button 505 accepts erasure of the operation guide 500. When the user presses button 505, the handwriting input unit 21 accepts it and notifies the handwriting input display control unit 23 of that fact, and the display unit 22 erases the display except for the handwritten data. Button 509 accepts erasure of all displays. When the user presses button 509, the handwriting input unit 21 accepts it and notifies the handwriting input display control unit 23 of that fact, and the display unit 22 erases all displays shown in FIG. 15, including the handwritten data, allowing the user to start handwriting again from the beginning.

[0096] In Fig. 15, handwritten data 504 is the character "gi" handwritten by the user. Handwritten data rectangular area display 503 is displayed surrounding handwritten data 504. The display procedure will be explained using the sequence diagrams of Figs. 18 to 23. In the example of Fig. 15, handwritten data rectangular area display 503 is displayed in a dotted line frame.

[0097] The character string candidates are arranged in descending order of probability in the handwriting recognition character string candidate 506, the converted character string candidate 507, and the predicted character string candidate 508. The character "gi" in the handwriting recognition character string candidate 506 is a candidate for the recognition result. In this example, the character "gi" is correctly recognized.

[0098] The converted character string candidate 507 is a character string candidate converted from the language character string candidate. In this example, "skill test" is an abbreviation for "technical mass production trial." The predicted character string candidate 508 is a predicted character string candidate converted from a handwritten recognition character string, a language character string candidate, or a converted character string candidate. In this example, "Approve the skill test" and "Address to send the minutes" are displayed.

[0099] The operation command candidate 510 is a candidate operation command selected based on the operation command definition data 701 to 703 in Fig. 13(a). In the example in Fig. 15, the first character of the line "》" 511 is shown to be a candidate operation command. In Fig. 15, there is no selection data to be selected by "gi" which is the handwritten data 504, and "meiji" which is a candidate character string for "gi" partially matches the operation command definition data 701 and 702 in Fig. 13(a), and therefore it is displayed as the operation command candidate 510.

[0100] When the user selects "Load minutes template," the operation command defined in operation command definition data 701 is executed, and when the user selects "Save in minutes folder," the operation command defined in operation command definition data 702 is executed. In this way, operation command candidates are displayed only when operation command definition data containing the converted character string is found, so they may not always be displayed.

[0101] As shown in FIG. 15, character string candidates and operation command candidates are displayed simultaneously (together), so the user can freely select either the character string candidate or the operation command that he or she intended to input.

[0102] <Example of selecting a character string> The display device 2 of this embodiment allows the user to select a character string by handwriting. The character string selected by a selection gesture is called a "selected object." The selected object can be moved, edited, modified, or the like.

[0103] 16 is an example diagram illustrating a method for selecting a character string. In FIG. 16, handwritten data 11b is displayed with a solid black line, handwritten data rectangular area 12c is displayed with gray shading, character string 13c is displayed with a black line, and selected object rectangular area 14c is displayed with a dotted line. The determination condition for determining whether a character string is a selected object (whether or not it has a predetermined relationship) is the enclosing line determination condition 407 of the predefined control data shown in FIG. 8.

[0104] 16 shows an example in which a horizontally written character string 13c is specified by an encircling line (handwritten data 11b). In this example, the character string 13c, "18," whose overlap rate with the character string 13c and the handwritten data rectangular area 12c satisfies the encircling line determination condition 407, is specified as the selected object.

[0105] <Example of operation command candidates displayed by selection gesture> 17A and 17B show examples of displaying operation command candidates based on operation command definition data, where Fig. 17A shows edit-related operation command candidates, and Fig. 17B shows modification-related operation command candidates.

[0106] As shown in Figures 17(a) and 17(b), the operation command candidates displayed following the first character "»" 511 are a main menu 550. The name of the operation command last executed or the name of the first operation command in the operation command definition data is displayed in the main menu 550. The first character "»" 511a on the first line is a candidate for an editing-related operation command, and the first character "»" 511b on the second line is a candidate for a modification-related operation command.

[0107] The ">" 512 at the end of the line indicates that there is a submenu (an example of a submenu button). The ">" 512a on the first line displays a submenu of candidate edit-related operation commands (the last selected one), and the ">" 512b on the second line displays the remaining submenus of candidate modifier-related operation commands. When the user presses the ">" 512, a submenu 560 is displayed to the right of it. All operation commands defined in the operation command definition data are displayed in the submenu 560. In the display example of Figure 17(a), the submenu 560 corresponding to the ">" 512a on the first line is also displayed from the time the main menu is displayed. The submenu 560 may also be displayed by pressing the ">" 512a on the first line.

[0108] When the user presses one of the operation command names with the pen, the handwriting input display control unit 23 executes the command in the operation command definition data associated with the operation command name on the selected object. That is, if "Erase" 521 is selected, "Delete" is executed, if "Move" 522 is selected, "Move" is executed, if "Rotate" 523 is selected, "Rotate" is executed, and if "Select" 524 is selected, "Select" is executed.

[0109] For example, if the user presses "Erase" 521 with the pen, "18" will be erased; if the user presses "Move" 522, "Rotate" 523, or "Select" 524, a bounding box (the bounding rectangle of the selected object) will be displayed; "Move" 522 and "Rotate" 523 can be used to move or rotate the object by dragging the pen; and "Select" 524 can be used to perform other bounding box operations.

[0110] The character string candidates other than the operation command candidates, "○" 551, "∞" 552, "0" 553, "00" 554, and "ロ" 555, are the recognition results of the enclosing line (handwritten data 11b), and if the user intended to input a character string instead of an operation command, the character string candidate can be selected.

[0111] In Fig. 17(b), pressing ">" 512b on the second line displays the submenu of Fig. 17(b). Similar to Fig. 17(a), the display example of Fig. 17(b) also displays a main menu 550 and a submenu 560. Based on the operation command definition data of Fig. 13, when "thicken" 531 is selected, the handwriting input display control unit 23 executes "Thick," when "thinner" 532 is selected, "Thin," when "bigger" 533 is selected, "Large," when "smaller" 534 is selected, "Small," and when underline 535 is selected, "Underline," respectively, for the selected object.

[0112] Note that fixed values ​​are separately defined for how thick to make the line when "Thick" 531 is selected, how thin to make the line when "Thin" 532 is selected, how large to make the line when "Larger" 533 is selected, how small to make the line when "Smaller" 534 is selected, and the type of line when underline 535 is selected. Alternatively, it would be better if a separate selection menu is opened when the submenu in Figure 17(b) is selected, allowing the user to make adjustments.

[0113] When the user presses "thicken" 531 with the pen, the handwriting input display control unit 23 thickens the lines that make up the character string 13c "18." When the user presses "thinner" 532 with the pen, the handwriting input display control unit 23 thins the lines that make up "18." When the user presses "larger" 533 with the pen, the handwriting input display control unit 23 enlarges the lines. When the user presses "smaller" 534 with the pen, the handwriting input display control unit 23 shrinks the lines. When the user presses "underline" 535 with the pen, the handwriting input display control unit 23 draws an underline.

[0114] 4, an operation command to be set in the page name may be displayed. For example, the operation command displayed in the operation guide 500 may be changed depending on whether a character string is selected by the selection gesture 304 in the input field or in another area. For example, when a character string in the input field is selected, "Set as page name" is displayed with priority, and in an area other than the input field, operation commands for editing and decoration are displayed with priority. This is because a font has already been defined in the input field, so operation commands for editing and decoration have a low priority.

[0115] <Operation procedure> The operation of the display device 2 will be described using the above configuration and Figs. 18 to 23. Figs. 18 to 23 are sequence diagrams illustrating an example of the process in which the display device 2 displays character string candidates and operation command candidates. The process in Fig. 18 starts when the display device 2 is started (when an application is started). Note that, due to space limitations, Figs. 18 to 23 use symbols to indicate the functions of Fig. 7.

[0116] S1: First, handwriting input display control unit 23 transmits a handwriting data start signal to handwriting input storage unit 25. Handwriting input storage unit 25 allocates a handwriting data area (a memory area for storing handwritten data). The handwriting data area may be allocated after the user brings the pen into contact with handwriting input unit 21.

[0117] S2: Next, the user touches the pen to the handwriting input unit 21. The handwriting input unit 21 detects the pen down and transmits it to the handwriting input display control unit 23.

[0118] S2.1: The handwriting input display control unit 23 transmits a stroke start signal to the handwriting input storage unit 25, and the handwriting input storage unit 25 reserves a stroke area.

[0119] S3: When the user moves the pen while keeping it in contact with the handwriting input unit 21, the handwriting input unit 21 transmits the pen coordinates to the handwriting input display control unit 23.

[0120] S3.1: The handwriting input display control unit 23 transmits pen coordinate complement display data (data for interpolating discrete pen coordinates) to the display unit 22. The display unit 22 uses the pen coordinate complement display data to interpolate the pen coordinates and display a line.

[0121] S3.2: The handwriting input display control unit 23 transmits the pen coordinates and the time of reception thereof to the handwriting input storage unit 25. The handwriting input storage unit 25 adds the pen coordinates to the stroke. While the user is moving the pen, the handwriting input unit 21 periodically repeats transmitting the pen coordinates to the handwriting input display control unit 23, and therefore the processes of steps S3 to S3.2 are repeated until the pen is lifted.

[0122] S4: When the user removes the pen from the handwriting input unit 21, the handwriting input unit 21 transmits a pen-up signal to the handwriting input display control unit 23.

[0123] S4.1: The handwriting input display control unit 23 sends a stroke end signal to the handwriting input storage unit 25, and the handwriting input storage unit 25 confirms the pen coordinates of the stroke. Once the pen coordinates of the stroke have been confirmed, it becomes impossible to add pen coordinates to the stroke.

[0124] S4.2: Next, the handwriting input display control unit 23 transmits an overlap status acquisition of the handwritten data vicinity rectangular area and the stroke rectangular area to the handwriting input storage unit 25 based on the handwritten data vicinity rectangular area 403. The handwriting input storage unit 25 calculates the overlap status and transmits the overlap status to the handwriting input display control unit 23.

[0125] The subsequent steps S4.3 to S4.5 are executed if the handwritten data vicinity rectangular area and the stroke rectangular area do not overlap.

[0126] S4.3: If the rectangular area near the handwritten data and the rectangular area of ​​the stroke do not overlap, one piece of handwritten data is confirmed, and the handwritten input / display control unit 23 transmits a command to clear the stored data to the handwriting recognition control unit 26.

[0127] S4.3.1 to S4.3.3: The handwriting recognition control unit 26 sends a retained data clear command to the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32. The data relating to character string candidates and operation command candidates retained by the handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 are cleared. Note that at the time of clearing, the last handwritten stroke has not been added to the handwritten data.

[0128] S4.4: The handwriting input display control unit 23 sends a handwritten data end signal to the handwriting input storage unit 25. The handwriting input storage unit 25 confirms the handwritten data. Confirming the handwritten data means that one piece of handwritten data has been completed (no more strokes can be added).

[0129] S4.5: The handwriting input display control unit 23 transmits a handwriting data start signal to the handwriting input storage unit 25. In preparation for the start of handwriting of the next handwritten data (pen down), the handwriting input storage unit 25 allocates a new handwritten data area.

[0130] S4.6: Next, the handwriting input display control unit 23 sends a stroke addition to the handwriting input storage unit 25 for the stroke completed in step S4.1. If steps S4.3 to S4.5 have been executed, the added stroke is the first stroke of the handwritten data, and the handwriting input storage unit 25 adds the stroke data to the starting handwritten data. If steps S4.3 to S4.5 have not been executed, the added stroke is added to the handwritten data already being handwritten.

[0131] S4.7: Next, the handwriting input display control unit 23 transmits a stroke addition to the handwriting recognition control unit 26. The handwriting recognition control unit 26 adds the stroke data to the stroke data storage area where the character string candidate is stored.

[0132] S4.7.1: The handwriting recognition control unit 26 executes handwriting recognition on this stroke data storage area.

[0133] S4.7.2: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidates, which are the execution results, to the handwriting recognition dictionary unit 27. The handwriting recognition dictionary unit 27 transmits the linguistically likely language character string candidates to the handwriting recognition control unit 26.

[0134] S4.7.3: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidate and the received language character string candidate to the character string conversion control unit 28.

[0135] S4.7.3.1: The character string conversion control unit 28 transmits the handwriting recognition character string candidates and the language character string candidates to the character string conversion dictionary unit 29. The character string conversion dictionary unit 29 transmits the conversion character string candidates to the character string conversion control unit 28.

[0136] S4.7.3.2: The character string conversion control unit 28 transmits the received converted character string candidates to the predictive conversion control unit 30.

[0137] S4.7.3.2.1: The predictive conversion control unit 30 transmits the received converted character string candidates to the predictive conversion dictionary unit 31. The predictive conversion dictionary unit 31 transmits the predicted character string candidates to the predictive conversion control unit 30.

[0138] S4.7.3.2.2: The predictive conversion control unit 30 transmits the received predicted character string candidate to the operation command recognition control unit 32.

[0139] S4.7.3.2.2.1: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) matching the predicted character string candidate.

[0140] The display device 2 performs the same processes from step S4.7.3.3 to step S4.7.5.1 up to the transmission of operation command candidates. S4.7.3.3: The character string conversion control unit 28 transmits the received conversion character string candidate to the operation command recognition control unit 32.

[0141] S4.7.3.3.1: The operation command recognition control unit 32 transmits the received conversion character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) that matches the conversion character string candidate.

[0142] S4.7.4: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidates and the language character string candidates to the predictive conversion control unit 30.

[0143] S4.7.4.1: The predictive conversion control unit 30 transmits the handwritten recognition character string candidate and the received language character string candidate to the predictive conversion dictionary unit 31. The predictive conversion dictionary unit 31 transmits the predicted character string candidate to the predictive conversion control unit 30.

[0144] S4.7.4.2: The predictive conversion control unit 30 transmits the received predicted character string candidate to the operation command recognition control unit 32.

[0145] S4.7.4.2.1: The operation command recognition control unit 32 transmits the received predicted character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) matching the predicted character string candidate.

[0146] S4.7.5: The handwriting recognition control unit 26 transmits the handwriting recognition character string candidates and the received language character string candidates to the operation command recognition control unit 32.

[0147] S4.7.5.1: The operation command recognition control unit 32 transmits the handwriting recognition character string candidate and the received language character string candidate to the operation command definition unit 33. The operation command definition unit 33 transmits the operation command candidate to the operation command recognition control unit 32. As a result, the operation command recognition control unit 32 can acquire the operation command candidate corresponding to the operation command definition data having a character string (String) that matches the language character string candidate.

[0148] S4.7.6: Next, the handwriting recognition control unit 26 sends the stroke addition to the operation command recognition control unit 32.

[0149] S4.7.6.1: The operation command recognition control unit 32 transmits the character string position information acquisition to the handwriting input storage unit 25. The handwriting input storage unit 25 transmits the character string position information to the operation command recognition control unit 32.

[0150] S4.7.6.2: To determine the selected object, the operation command recognition control unit 32 determines whether the position information of the stroke received from the handwriting recognition control unit 26 in the stroke addition in step S4.7.6 and the position information of the character string received from the handwriting input storage unit 25 have a predetermined relationship based on the crossing line determination condition 406 and the encircling line determination condition 407, and stores the character string that can be determined to be selected as the selected object. In this case, since the selected object is specified, candidates for input / output operation commands are obtained from the operation command definition unit 33.

[0151] In addition, the handwriting recognition control unit 26, the string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 each store handwriting recognition string candidates, language string candidates, conversion string candidates, predicted string candidates, operation command candidates, and data related to the selected object so that they can be obtained in subsequent steps S4.8.1.2 to 4.8.1.4.

[0152] S4.8: Immediately after transmitting the stroke addition command to the handwriting recognition control unit 26 in step S4.7, the handwriting input display control unit 23 transmits a selectable candidate display timer start command to the candidate display timer control unit 24. The candidate display timer control unit 24 starts this timer.

[0153] The following steps S5 to S5.2 are executed if a pen-down occurs before a certain time has elapsed (before the timer times out).

[0154] S5: If the user touches the pen to the handwriting input unit 21 before the timer times out, the handwriting input unit 21 transmits a pen-down signal (the same event as in step S2) to the handwriting input display control unit 23.

[0155] S5.1: The handwriting input display control unit 23 transmits a stroke start signal (same as step S2.1) to the handwriting input storage unit 25. The subsequent sequence is the same as that from step S2.1 onwards.

[0156] S5.2: Furthermore, the handwriting input display control unit 23 transmits a selectable candidate display timer stop command to the candidate display timer control unit 24. The candidate display timer control unit 24 stops the timer because the timer is no longer necessary since pen-down has been detected.

[0157] Steps S4.8.1 to S7.14 are executed if the pen does not go down before a certain time has elapsed (before the timer times out), and therefore the character string candidates and operation command candidates shown in FIG.

[0158] 4.8.1: If the user does not bring the pen into contact with the handwriting input unit 21 while the selectable candidate display timer is in progress, the candidate display timer control unit 24 transmits a timeout to the handwriting input display control unit 23 .

[0159] S4.8.1.1: The handwriting input display control unit 23 transmits a request to obtain handwriting recognition / language character string candidates to the handwriting recognition control unit 26. The handwriting recognition control unit 26 transmits the handwriting recognition / language character string candidates currently held to the handwriting input display control unit 23.

[0160] S4.8.1.2: The handwriting input display control unit 23 transmits a request to obtain a conversion character string candidate to the character string conversion control unit 28. The character string conversion control unit 28 transmits the conversion character string candidate currently held to the handwriting input display control unit 23.

[0161] S4.8.1.3: The handwriting input display control unit 23 transmits a predicted character string candidate acquisition request to the predictive conversion control unit 30. The predictive conversion control unit 30 transmits the predicted character string candidates currently held to the handwriting input display control unit 23.

[0162] S4.8.1.4: The handwriting input display control unit 23 transmits the acquired operation command candidates to the operation command recognition control unit 32. The operation command recognition control unit 32 transmits the currently held operation command candidates and the selected object to the handwriting input display control unit 23.

[0163] S4.8.1.5: Furthermore, the handwriting input display control unit 23 transmits the estimated writing direction to the handwriting input storage unit 25. The handwriting input storage unit 25 determines the estimated writing direction from the stroke addition time, horizontal distance, and vertical distance of the rectangular area of ​​the handwritten data, and transmits the estimated writing direction to the handwriting input display control unit 23.

[0164] S10.1: The handwriting input display control unit 23 determines whether or not to display the operation guide 500. To this end, the handwriting input display control unit 23 inquires of the input field definition unit 34 whether or not the location where the stroke is handwritten is a forced conversion input field.

[0165] S10.2: The input field definition unit 34 determines whether the stroke has been handwritten in the input field based on the stroke coordinates and the input field definition data, and whether the input field is a forced conversion input field. In making this determination, the input field definition unit 34 queries the ruled line determination unit 35 about the relationship between the stroke and the ruled line. This query is not essential.

[0166] S10.3: The ruled line determination unit 35 returns to the input field definition unit 34, as the relationship between the stroke and the ruled line, whether the stroke surrounds or straddles the forced conversion input field.

[0167] S10.4: The input field definition unit 34 returns to the handwriting input display control unit 23 whether or not the input is handwritten in a forced conversion input field.

[0168] The following steps S4.8.1.6 and S4.8.1.7 (displaying the operation guide 500) are executed when either of the following conditions (1) or (2) is met. (1) Handwriting in an input field that is not a forced conversion input field. (2) The stroke is a selection gesture that selects a string in the forced conversion input field.

[0169] In other words, S4.8.1.6 and S4.8.1.7 are not executed if the following conditions are met: (3) The stroke is handwritten in a forced conversion field, and there is no text in the forced conversion field. Even if there is text in the forced conversion field, the stroke is not a selection gesture.

[0170] If (3) is satisfied, the user does not select a selectable candidate, so the handwriting recognition control unit 26 determines (forces conversion) the character string that is most similar (has the highest probability) to the stroke, and displays it on the display unit 22. The process proceeds to step S7.9, where the forcibly converted character string is displayed.

[0171] S4.8.1.6: When the operation guide 500 is displayed, the handwriting input display control unit 23 creates selectable candidate display data as shown in Figure 15 from the handwriting recognition string candidate ("gi" in Figure 15), language string candidate (not shown in Figure 15, for example, "gi"), converted string candidate ("minutes" and "skill test" in Figure 15), predicted string candidate ("settle the skill test" and "address to send the minutes to" in Figure 15), operation command candidate ("load minutes template" and "save in minutes folder" in Figure 15), each selection probability, and estimated writing direction, and displays the selectable candidate display data consisting of the string candidate and operation command candidate by sending it to the display unit 22.

[0172] S4.8.1.7: The handwriting input display control unit 23 also transmits the handwritten data and rectangular area display data (rectangular frame) of the selected object (handwritten data rectangular area display 503 in FIG. 15) to the display unit 22.

[0173] S4.8.1.8: The handwriting input display control unit 23 transmits a selectable candidate display deletion timer start signal to the candidate display timer control unit 24 to delete the selectable candidate display data after a certain time has elapsed since its display. The candidate display timer control unit 24 starts this timer.

[0174] Steps S6 to S6.3 are executed when the user erases the selectable candidate display displayed on the display unit 22 while the selectable candidate deletion timer is running, or when a change occurs in the handwritten data (i.e., when a stroke in the handwritten data is added, deleted, moved, deformed, or divided), or when no candidate is selected before the timer times out.

[0175] Furthermore, steps S6 to S6.1 are executed when the candidate display is erased or when a change occurs in the handwritten data.

[0176] S6: The handwriting input unit 21 transmits to the handwriting input display control unit 23 a notice that the selectable candidate display has been deleted or that a change has occurred in the handwritten data.

[0177] S6.1: The handwriting input display control unit 23 sends a command to stop the selectable candidate deletion timer. The candidate display timer control unit 24 stops the timer. This is because an operation on the handwritten data has occurred within a certain period of time, making the timer unnecessary.

[0178] S6.2: The handwriting input display control unit 23 sends a command to erase the selectable candidate display data to the display unit 22, thereby erasing the display.

[0179] S6.3: The handwriting input display control unit 23 erases the display by transmitting a command to erase the handwritten data and the rectangular area display data of the selected object to the display unit 22. Therefore, when the display of the operation command candidate is erased under conditions other than the selection of the operation command candidate, the handwritten data remains displayed.

[0180] 4.8.1.8.1: On the other hand, if the selectable candidate display is not erased or the handwritten data is not changed while the selectable candidate deletion timer is started (if the user does not perform any pen operation), the candidate display timer control unit 24 sends 4.8.1.8.1: timeout to the handwriting input display control unit 23.

[0181] Similarly, after the selectable candidate display deletion timer times out, the handwriting input display control unit 23 executes steps S6.2 and S6.3. This is because the selectable candidate display data, handwritten data, and rectangular area display data of the selected object may be deleted after a certain period of time has elapsed.

[0182] If the user selects a selectable candidate while the selectable candidate deletion timer is running, steps S7 to S7.14 are executed.

[0183] S7: If the user selects a selectable candidate while the selectable candidate deletion timer is running, the handwriting input unit 21 transmits the character string candidate or operation command candidate selection to the handwriting input display control unit 23.

[0184] S7.1: The handwriting input display control unit 23 transmits a selectable candidate display deletion timer stop command to the candidate display timer control unit 24. The candidate display timer control unit 24 stops this timer.

[0185] S7.2: Next, the handwriting input / display control unit 23 sends a command to clear the stored data to the handwriting recognition control unit 26.

[0186] S7.2.1: The handwriting recognition control unit 26 sends a retained data clear command to the character string conversion control unit 28.

[0187] S7.2.2: The handwriting recognition control unit 26 sends a command to clear the stored data to the predictive conversion control unit 30.

[0188] S7.2.3: The handwriting recognition control unit 26 sends a command to clear the stored data to the operation command recognition control unit 32.

[0189] The handwriting recognition control unit 26, the character string conversion control unit 28, the predictive conversion control unit 30, and the operation command recognition control unit 32 clear the data relating to character string candidates and operation command candidates that have been held up to that point.

[0190] S7.3: Next, the handwriting input display control unit 23 transmits a command to erase the selectable candidate display data to the display unit 22, thereby erasing the display.

[0191] S7.4: The handwriting input display control unit 23 sends a command to erase the handwritten data and the rectangular area display data of the selected object to the display unit 22, thereby erasing the display.

[0192] S7.5: The handwriting input display control unit 23 erases the display by transmitting a command to erase the handwritten data display data and the pen coordinate complement display data transmitted in step S3.1 to the display unit 22. This is because the handwritten data, etc., are no longer needed since a character string candidate or an operation command candidate has been selected.

[0193] S7.6: The handwritten input display control unit 23 transmits a request to delete the handwritten data to the handwritten input storage unit 25.

[0194] If a character string candidate is selected, steps S7.7 to S7.9 are executed.

[0195] S7.7: If a character string candidate is selected, the handwriting input display control unit 23 transmits a character string object addition to the handwriting input saving unit 25.

[0196] S7.8: Furthermore, the handwriting input display control unit 23 transmits a character string object font acquisition command to the handwriting input storage unit 25. The handwriting input storage unit 25 selects a predefined font based on the estimated character size of the handwritten data and transmits the selected font to the handwriting input display control unit 23.

[0197] S7.9: Next, the handwriting input display control unit 23 uses the defined font received from the handwriting input storage unit 25 to send character string object display data to the display unit 22 to be displayed in the same position as the handwritten data, thereby displaying the character string object.

[0198] If a candidate operation command is selected, steps S7.10 to S7.13 are executed, and if there is a selected object, steps S7.10 to S7.12 are executed.

[0199] S7.10: When a candidate for an operation command for a selected object is selected (when a selected object exists), the handwriting input display control unit 23 erases the display by sending a command to erase the display data of the selected object to the display unit 22. This is to erase the original selected object.

[0200] S7.11: Next, the handwriting input display control unit 23 transmits an operation command for executing the selected object to the handwriting input saving unit 25. The handwriting input saving unit 25 transmits display data of the newly selected object (display data after editing or modification) to the handwriting input display control unit 23.

[0201] S7.12: Next, the handwriting input display control unit 23 transmits the selected object display data to the display unit 22, thereby re-displaying the selected object after the operation command is executed.

[0202] If there is no selected object (if an input / output operation command is selected), step S7.13 is executed.

[0203] S7.13: When an input / output operation command is selected, the handwriting input display control unit 23 executes the operation command character string (Command) of the operation command definition data corresponding to the operation command selected by the user. If the authentication unit successfully authenticates the user, the handwriting input display control unit 23 sets information about the successfully authenticated user in % to % of the operation command and executes it.

[0204] S7.14: For the next handwritten data, the handwritten input display control unit 23 sends a handwritten data start signal to the handwritten input saving unit 25. The handwritten input saving unit 25 allocates a handwritten data area. After this, the processes of steps S2 to S7.14 are repeated.

[0205] <Selection gesture to the forced conversion input field where the string is displayed> The following describes in detail how to determine whether the gesture is for selecting the forced conversion input field where the character string is displayed (steps S10.1 to S10.4).

[0206] The display device 2 of this embodiment has the following features: The display device 2 (handwriting input display control unit 23, input field definition unit 34) determines whether the following conditions (1) to (3) are met. (1) Is the input field into which handwritten data is entered a forced conversion input field? (2) Is the string registered in the input field definition data of the forced conversion input field? (3) Is the handwritten data a selection gesture for a string? If the display device 2 determines that all of the conditions (1) to (3) are satisfied, it accepts the selection of the character string already displayed in the forced conversion input field, and displays on the display screen an operation guide 500 for editing the character string already registered in the forced conversion input field (e.g., delete, enlarge, reduce).

[0207] If the display device 2 determines that at least one of the conditions (1) to (3) is not satisfied, it displays the operation guide 500 if the input field is not a forced conversion input field, and if the input field is a forced conversion input field, it forcibly converts the input field into a predetermined character type and displays it on the display screen.

[0208] <How to convert a selection gesture into a string> FIG. 24 is a diagram illustrating an example of conversion when a selection gesture is converted into a character string. FIG. 24 is a diagram for explaining a supplement to FIG. 3, so please refer to FIGS. 3(d) and (e) as appropriate. As shown in FIG. 24(a), the user handwrites a "circle" (corresponding to a selection gesture 304) that surrounds an input field 301 with a stroke. The display device 2 determines which input field 301 the handwriting was in. One method for determining this is to use the input field 301 in which the circumscribing rectangle 310 of the handwritten data overlaps the most. In FIG. 24(a), the circumscribing rectangle 310 overlaps the input field 301 to the right of age the most, so it is determined that the handwriting was in the input field 301 to the right of age.

[0209] Next, the display device 2 recognizes what has been handwritten. When the display device 2 performs character recognition (forced conversion) on the sequence of coordinate points of the handwritten data (corresponding to the selection gesture 304), it recognizes it as "0." The display device 2 displays the character string 311 "0" in the input field 301. If "18" is already displayed in the input field 301, the display device 2 deletes all of the "18" and displays only the newly handwritten character string 311 (FIG. 24(b)). In this way, the selection gesture 304 may be converted to "0."

[0210] <Determining whether to select or coerce a string> 25 is a flowchart illustrating the process of determining whether handwriting in the input field is a character string selection or forced conversion. The process in FIG. 25 starts when the user writes handwriting in the forced conversion input field.

[0211] As explained in the sequence diagram (step S4.7.6), when the operation command recognition control unit 32 receives handwritten input from the user, it determines whether the input handwritten data is a selection gesture (S101). Specifically, the operation command recognition control unit 32 determines whether the input handwritten data straddles the ruled line that forms the forced conversion input field or whether the input handwritten data surrounds the ruled line that forms the forced conversion input field.

[0212] If the determination in step S101 is Yes, the process proceeds to step S102, and if the determination is No, the process proceeds to step S104.

[0213] In step S102, the input field definition unit 34 determines whether a character string is registered in the input field definition data for the handwritten forced conversion input field (S102). If a character string has been registered in the input field definition data (it has also been defined as a forced conversion input field), and this character string has been selected with the selection gesture 304, step S102 is determined as Yes.

[0214] If the determination in step S102 is Yes, the process proceeds to step S103, and if the determination is No, the process proceeds to step S104.

[0215] In step S103, the handwriting input display control unit 23 selects a character string registered in the input field definition data. The handwriting input display control unit 23 determines to display an operation guide 500 including candidate operation commands such as editing and modification commands (S103). When a candidate operation command is selected from the operation guide 500, the selected operation command is executed.

[0216] In step S104, since it is determined that the input is a character, the handwriting recognition control unit 26 forcibly converts the handwritten data, and the handwriting input display control unit 23 displays the character string in the forcible conversion input field (S104). If a character string is already displayed in the forcible conversion input field, the display unit 22 deletes the character string.

[0217] This allows the user to select a string even if it is already displayed in the coercion input field.

[0218] If a user wants to rewrite a string that has already been entered, he or she should write it by hand so that conditions (1) to (3) are not met.

[0219] 26A and 26B are diagrams illustrating an example of handwriting that overwrites a character string displayed in a forced conversion input field. As shown in FIG. 26A, a user handwrites handwritten data 312, "0," in the forced conversion input field. The display device 2 determines in which input field the handwriting was made. The criteria are the same as those in FIG. 24. In FIG. 26A, the bounding rectangle 315 of the handwritten data 312 overlaps most with the input field 301 to the right of age, so it is determined that the handwriting was made in the input field 301 to the right of age.

[0220] Furthermore, since the bounding rectangle 315 of the handwritten data 312 and the character string 303 do not overlap at all, the encircling line determination condition 407 is not met. The display device 2 determines that the handwritten data 312 is not a selection gesture. The display device 2 recognizes what has been handwritten. When the display device 2 performs character recognition (forced conversion) on the coordinate point sequence of the handwritten data 312, it recognizes it as "0." Since the character string 303 "18" is already displayed, the display device 2 deletes all of the "18" and displays the newly handwritten character string 313, "0," in the input field 301 (FIG. 26(b)). In this way, the display device 2 can also accept rewriting of the character string 303 that has already been input by the user.

[0221] <<Judgment using the relationship with ruled lines 1>> By determining whether the selection gesture crosses the ruled line that forms the forced conversion input field, it can be determined with certainty that the selection gesture has selected the character string in the forced conversion input field.

[0222] Fig. 27 is a flowchart illustrating the process of determining whether handwriting in the input field 301 is a character string selection or forced conversion, depending on whether the selection gesture crosses a ruled line. The explanation of Fig. 27 may mainly focus on the differences from Fig. 25. In Fig. 27, the determination in step S202 differs from that in Fig. 25. In step S202, ruled line determination unit 35 determines whether the selection gesture crosses a ruled line of the forced conversion input field in which the selected character string is registered (S202). This will be explained with reference to Fig. 28.

[0223] FIG. 28 is a diagram illustrating a method for determining whether a selection gesture crosses a ruled line that forms a forced conversion input field. The ruled line determination unit 35 can determine whether a selection gesture crosses a ruled line, for example, by determining whether intersections 325-328 between the selection gesture 304 and four straight lines 321-324 of the ruled line that forms the forced conversion input field are on the four straight lines 321-324 that form the input field 301. That is, the determination is made based on whether intersections 325-328 between the ruled line and the stroke are on the four sides of the input field 301. In the drawing, all four intersections 325-328 are on the four sides of the input field 301, so step S202 is determined to be Yes. Note that the number of intersections that must be on the four sides for step S202 to be determined to be Yes may be set in advance. In the embodiment described below, this number can be adjusted by the display device 2.

[0224] When the selection gesture 304 crosses the ruled line forming the forced conversion input field, it can be determined with higher accuracy that the selection gesture 304 has selected the character string 303 in the forced conversion input field, and therefore it can be determined with certainty that the selection gesture 304 has selected the character string in the forced conversion input field.

[0225] <<Judgment using the relationship with ruled lines 2>> Similarly, by determining whether the selection gesture surrounds the ruled lines that form the forced conversion field, it can be determined with certainty that the selection gesture has selected the character string in the forced conversion field.

[0226] Fig. 29 is a flowchart illustrating the process of determining whether handwriting in input field 301 is a character string selection or forced conversion, depending on whether the selection gesture surrounds a ruled line. The explanation of Fig. 29 may mainly focus on the differences from Fig. 25. In Fig. 29, the determination in step S302 differs from that in Fig. 25. In step S302, ruled line determination unit 35 determines whether the selection gesture surrounds the ruled line of the forced conversion input field in which the selected character string is registered (S302). The explanation will be made with reference to Fig. 30.

[0227] FIG. 30 is a diagram illustrating a method for determining whether a selection gesture surrounds a ruled line forming a forced conversion input field. The ruled line determination unit 35 can determine this by, for example, determining whether intersections 331-334 between the selection gesture 304 and four straight lines 321-324 of the ruled line forming the forced conversion input field are on extensions of the four straight lines 321-324 forming the input field 301. That is, the determination is made by determining whether the intersections 331-334 of the ruled line and the selection gesture 304 are not on the four sides of the input field 301 but on extensions of the four sides. In the drawing, all four intersections 331-334 are on extensions of the four straight lines 321-324 forming the input field 301, so step S302 is determined to be Yes. Note that the number of intersections required to be on extensions of the four sides for step S302 to be determined to be Yes may be preset. In the embodiment described below, this number can be adjusted by the display device 2.

[0228] When the selection gesture surrounds the ruled line that forms the forced conversion input field, it can be determined with higher accuracy that the selection gesture has selected the character string in the forced conversion input field, so it can be determined with certainty that the selection gesture has selected the character string in the forced conversion input field.

[0229] <Example of operation guide display> FIG. 31 is a diagram illustrating an operation guide 500 that is displayed when it is determined that the selection gesture has selected the character string displayed in the forced conversion input field. FIG. 31(a) shows an example in which the selection gesture 304 does not interfere with the ruled line, FIG. 31(b) shows an example in which the selection gesture 304 straddles the ruled line, and FIG. 31(c) shows an example in which the selection gesture 304 surrounds the ruled line. In either case, the display device 2 determines that the selection gesture has selected the character string displayed in the forced conversion input field, and can display operation command candidates in the operation guide 500. When the operation command candidate "Delete" is selected, the character string "18" is deleted, and when "Set as page name" is selected, the title of the current page becomes "18." Note that while only operation command candidates are displayed in FIG. 31, character string candidates may also be displayed.

[0230] <Major Effects> In the display device of this embodiment, when a selection gesture is handwritten in a forced conversion input field in which a character string is already displayed, the selection gesture is not forcibly converted into a character string, so that the user can select the character string displayed in the forced conversion input field with a gesture. When a character that is not a selection gesture is handwritten in the forced conversion input field, the character string displayed in the forced conversion input field can be forcibly replaced.

[0231] [Second embodiment] In this embodiment, an example using machine learning will be described. The display device 2 analyzes the user's evaluation results and proposes changes to the setting conditions for selecting or editing the content already input by the user. The display device 2 notifies the user of the past judgment results and corrects the conditions for determining whether the gesture is a selection gesture or a forced conversion.

[0232] Fig. 32 is a functional block diagram of a display device 2 of this embodiment. The display device 2 of Fig. 32 newly includes an analysis unit 36 ​​and a display reception unit 37. The analysis unit 36 ​​accumulates a history of whether handwritten data in a forced conversion input field is a selection gesture or forced conversion, and corrects the conditions for determining whether it is a selection gesture or forced conversion in response to an input from the user. The display reception unit 37 displays a determination result screen 350 of Fig. 34 and receives user operations on the determination result screen 350.

[0233] FIG. 33 is a flowchart for changing the conditions based on the result of determining whether the handwritten data in the forced conversion input field is a selection gesture or a forced conversion.

[0234] First, the analysis unit 36 ​​checks the number of times it has determined whether the gesture is a selection gesture or a forced conversion (S501). Next, the analysis unit 36 ​​determines whether the number of times it has determined whether the gesture is a selection gesture or a forced conversion is 10 or more (S502). 10 times is an example, and it may be 1 or more. If the determination in step S502 is No, the process returns to step S501.

[0235] If the determination in step S502 is Yes, the display receiving unit 37 displays the determination result screen 350 (S503). An example of the determination result screen 350 is shown in FIG.

[0236] The display receiving unit 37 determines whether or not it has received a "No" operation from the user (S504). That is, the display receiving unit 37 displays the determination result screen 350 and receives input from the user as to whether or not the determination result is what the user desires. If the determination in step S504 is No, the process returns to step S501.

[0237] If the "No" operation is received (Yes in step S504), the analysis unit 36 ​​changes the conditions for determining whether the gesture is a selection gesture or a forced conversion from the next time onwards (S505).

[0238] Specifically, the conditions in steps S202 and S302 shown in Figures 27 and 29 are changed to determine how many intersections must be on the four sides, or how many intersections must be on the extensions of the four sides. The effectiveness of this change will be explained with reference to Figure 35.

[0239] When the user writes by hand so as to surround the age input field 301, part of the selection gesture 304 may end up inside the input field 301, as shown in Fig. 35. The selection gesture 364 has intersections 361 and 362 with two of the four sides that form the forced conversion input field. In addition, there are only three intersections (332a to 334a) between the selection gesture 364 and the extensions of the four sides that form the forced conversion input field.

[0240] In this case, if the judgment condition "all four intersections must be on the extensions of the four sides" is applied, the user will not be able to select "18." Therefore, by having the analysis unit 36 ​​change the condition to "three or more intersections must be on the extensions of the four sides," the user can select "18" even if the user's handwriting is as shown in Figure 35, and the operator can perform the operation he or she desires.

[0241] The judgment result screen 350 in Fig. 34 displays a list of the judgment results of handwriting in the input field 301 for which the forced conversion function has been set for the past 10 times. The judgment result screen 350 has a judgment table 351. The judgment table 351 has the number, the date and time of handwriting, the target input field, and the judgment result. Of these, the judgment result indicates whether it was determined that the character string displayed in the forced conversion input field should be selected or that the handwritten data in the forced conversion input field should be forcibly converted.

[0242] Furthermore, the judgment result screen 350 displays a note 352 saying "The past 10 judgment results for handwriting in input fields for which the forced conversion function is set are displayed. Are the judgment results what the operator intended?", a Yes button 353, and a No button 354. If necessary, the display receiving unit 37 may also display the judgment results (contents displayed on the display screen after the judgment) on the judgment result screen 350 as history information.

[0243] When the user presses the Yes button 353 on the judgment result screen 350, the judgment result is what the user intended, so the analysis unit 36 ​​does not correct the conditions for determining whether the handwritten data in the input field 301, for which the forced conversion function is set, is a selected object or forced conversion.

[0244] On the other hand, if the user presses the No button 354, the result of the judgment is not what the user intended, so the analysis unit 36 ​​corrects the conditions for determining whether the handwritten data in the input field 301, for which the forced conversion function is set, is a selected object or a forced conversion. The correction is performed so that it becomes easier to determine that the data is a selected object.

[0245] The method for correcting the conditions is not limited to this. Alternatively, when the user clicks the "No" button on the screen, the display receiving unit 37 may display on the screen the number of intersections between the ruled lines forming the forced conversion input field and the stroke (corresponding to the selection gesture 304), and the user may change the number by directly inputting the correction parameter (number of intersections).

[0246] <Major Effects> According to this embodiment, in addition to the effects of the first embodiment, the threshold value for the display device 2 to determine a selection gesture can be adjusted by input from the user.

[0247] [Third embodiment] In this embodiment, a display system in a system configuration in which an information processing system on a network performs processing such as handwriting recognition and returns the processing results to the display device 2 will be described.

[0248] In the description of this embodiment, components or contents of the figures that are given the same symbols as in the first embodiment perform the same functions, so the description of components that have already been described may be omitted or only the differences may be described.

[0249] 36 is an example of a system configuration diagram of a display system 100. The display system 100 has a display device 2 and an information processing system 10 that can communicate with each other via a network N.

[0250] The display device 2 is placed in a facility such as a company, and is connected to a LAN or Wi-Fi installed within the facility. The information processing system 10 is placed, for example, in a data center. The display device 2 is connected to the Internet i via a firewall 8, and the information processing system 10 is also connected to the Internet i via a high-speed LAN or the like within the data center.

[0251] The display device 2 may be connected to the Internet i using wireless communication such as a telephone line network. In this case, the wireless communication may be 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), or the like.

[0252] The information processing system 10 has one or more information processing devices, and the one or more information processing devices act as servers to provide services to the display device 2. A server is a computer or software that performs the function of providing information and processing results in response to requests from clients. As will be described later, the information processing system 10 receives pen coordinates from the display device 2 and transmits information required to display the operation guide 500 shown in FIG. 15 to the display device 2.

[0253] The configuration of the display device 2 may be the same as that of the first embodiment, but in this embodiment, it is sufficient to have a touch panel, a display, and a communication function. The display device 2 may include multiple computing devices configured to communicate with each other.

[0254] In this embodiment, a general information processing device such as a PC or tablet can execute a web browser or a dedicated application. The web browser or the dedicated application communicates with the information processing system 10. When the web browser is running, the user inputs or selects the URL of the information processing system 10 to connect the display device to the information processing system 10. The display device 2 executes a web application provided by the information processing system 10 in the web browser. A web application refers to software or a mechanism thereof that operates through cooperation between a program written in a programming language (e.g., JavaScript (registered trademark)) that runs on the web browser and a program on the web server side.

[0255] When the dedicated application runs, it connects to a pre-registered URL of the information processing system 10. The dedicated application has a program and a user interface, so the program sends and receives necessary information to and from the information processing system 10 and displays it on the user interface.

[0256] The communication method may be a general-purpose communication protocol such as HTTP, HTTPs, or WebSocket, or a dedicated communication protocol.

[0257] <Hardware configuration example> The hardware configuration of the display device 2 may be the same as that shown in Fig. 5. In this embodiment, an example of the hardware configuration of the information processing system 10 will be described.

[0258] Fig. 37 is a hardware configuration diagram of the information processing system 10. As shown in Fig. 37, the information processing system 10 is constructed by a computer, and includes a CPU 601, a ROM 602, a RAM 603, an HD 604, an HDD (Hard Disk Drive) controller 605, a display 606, an external device connection I / F (Interface) 608, a network I / F 609, a bus line 610, a keyboard 611, a pointing device 612, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 616.

[0259] Of these, the CPU 601 controls the overall operation of the information processing system 10. The ROM 602 stores programs used to drive the CPU 601, such as the IPL. The RAM 603 is used as a work area for the CPU 601. The HD 604 stores various data, such as programs. The HDD controller 605 controls the reading and writing of various data from and to the HD 604 under the control of the CPU 601. The display 606 displays various information, such as a cursor, menu, window, text, or image. The external device connection I / F 608 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 609 is an interface for data communication using a communication network. The bus line 610 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 601, shown in FIG. 37.

[0260] The keyboard 611 has a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 612 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 614 controls reading and writing of various data from and to a DVD-RW 613, which is an example of a removable recording medium. Note that this is not limited to DVD-RW, and may be DVD-R, etc. The media I / F 616 controls reading and writing (storing) of data from and to a recording medium 615, such as a flash memory.

[0261] <About the device functions> Next, functions of the display system 100 will be described with reference to Fig. 38. Fig. 38 is an example of a functional block diagram showing functions of the display system 100 in blocks. Note that the description of Fig. 38 will mainly focus on the differences from Fig. 7.

[0262] In this embodiment, the display device 2 has a display unit 22, a display control unit 41, a handwriting input unit 21, and a communication unit 42. Each function of the display device 2 is a function or means realized when any of the components shown in Fig. 6 operates in response to an instruction from the CPU 201 in accordance with a program loaded from the SSD 204 onto the RAM 203.

[0263] The function of the handwriting input unit 21 in this embodiment may be the same as that in the first embodiment. The handwriting input unit 21 converts the user's pen input d1 into pen operation data da (pen up, pen down, or pen coordinate data) and transmits it to the display control unit 41.

[0264] The display control unit 41 controls the display of the display device 2. First, the display control unit 41 performs interpolation calculations on the coordinates between the discrete values ​​of the pen coordinate data, which are discrete values, and converts the pen coordinate data from pen-down to pen-up into one stroke db, and transmits it to the display unit 22.

[0265] Furthermore, the display control unit 41 transmits the pen operation data dc to the communication unit 42 and acquires various display data dd from the communication unit 42. The display data includes information for displaying the operation guide 500 in FIG. 15. The display control unit 41 transmits the display data de to the display unit 22.

[0266] The communication unit 42 (an example of a first communication unit) transmits pen operation data dc to the information processing system 10, receives various display data dd from the information processing system 10, and transmits the data to the display control unit 41.

[0267] The function of the display unit 22 may be the same as in the first embodiment. The display unit 22 displays the stroke db and display data de. The display unit 22 converts the stroke db and display data de written to the video memory by the display control unit 41 into data according to the characteristics of the display 220 and transmits them to the display 220.

[0268] <<Functions of the information processing system>> The information processing system 10 includes a communication unit 43, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a character string conversion control unit 28, a character string conversion dictionary unit 29, a predictive conversion control unit 30, a predictive conversion dictionary unit 31, an operation command recognition control unit 32, an operation command definition unit 33, an input field definition unit 34, and a ruled line determination unit 35. Each function of the information processing system 10 is a function or means realized when any of the components shown in Fig. 37 operates in response to an instruction from the CPU 601 in accordance with a program loaded from the HD 604 onto the RAM 603.

[0269] The communication unit 43 (an example of a second communication unit) receives pen operation data dc from the display device 2, transmits pen operation data df to the handwriting input display control unit 23, receives display data dd from the handwriting input display control unit 23, and transmits it to the display device 2.

[0270] Other functions are the same as those of the first embodiment, or even if they are different, they will not hinder the explanation of this embodiment.

[0271] <<Action or Processing>> FIG. 39 is a sequence diagram illustrating a process in which the display device 2 and the information processing system 10 communicate with each other to display the recognition result of handwritten data.

[0272] S11: The display device 2 repeats steps S11 to S13 until the pen is lifted. The communication unit 42 of the display device 2 transmits the pen operation data (stroke data) to the information processing system 10.

[0273] S12: The communication unit 43 of the information processing system 10 receives the pen operation data (stroke data), and performs the processes described with reference to FIGS. 18 to 23, and performs character recognition every time one piece of stroke data is added.

[0274] S13: The display control unit 41 of the display device 2 causes the display 220 to display the stroke data in parallel with the transmission of the pen operation data.

[0275] S14: When the user lifts up the pen, the communication unit 43 of the information processing system 10 detects the pen-up by not receiving any pen operation data for a certain period of time. The communication unit 42 may transmit a message to the information processing system 10 that the pen has been lifted.

[0276] S15: If it is determined to be a selection gesture, when the communication unit 43 of the information processing system 10 detects that the pen has been lifted, the handwriting input display control unit 23 transmits the operation command candidates stored therein to the display device 2. Character string candidates may also be transmitted.

[0277] S16: The communication unit 42 of the display device 2 receives the operation command candidates, and the display control unit 41 causes the operation guide 500 to display them.

[0278] S17: The display device 2 executes the selected operation command. If a character string candidate is selected, the display control unit 41 displays the selected character string candidate as a character string in the forced conversion input field.

[0279] S18: If it is determined that the gesture is not a selection gesture but a character input, when the communication unit 43 of the information processing system 10 detects that the pen has been lifted, the handwriting input display control unit 23 transmits the character string candidates it has stored to the display device 2.

[0280] S19: The communication unit 42 of the display device 2 receives the character string candidate, and the display control unit 41 displays the character string in the forced conversion input field.

[0281] <Major Effects> According to this embodiment, in addition to the effects of the first embodiment, handwritten input is possible on any display device 2 connected to a network.

[0282] [Other application examples] The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0283] For example, in this embodiment, a character string is selected with a selection gesture, but the selection gesture can also select handwritten data, graphics, symbols, etc. Also, the user can select multiple objects with one selection gesture.

[0284] Although the present embodiment has been described using an electronic whiteboard as an example, the present invention can be suitably applied to any information processing device having a touch panel. Devices having similar functions to electronic whiteboards are also called electronic whiteboards, electronic information boards, interactive boards, etc. An information processing device equipped with a touch panel may also be, for example, a display device using a projector.

[0285] In this embodiment, the coordinates of the pen tip are detected by detecting them on a touch panel, but the coordinates of the pen tip may also be detected by ultrasonic waves. The pen emits light and also emits ultrasonic waves, and the display device 2 calculates the distance based on the time it takes for the ultrasonic waves to reach the screen. The position of the pen can be identified based on the direction and distance. The projector draws (projects) the trajectory of the pen as a stroke.

[0286] In addition, the configuration examples in Figure 7 and the like are divided according to main functions to make it easier to understand the processing by the display device 2. The method of dividing the processing units and their names do not limit the present invention. The processing by the display device 2 can be divided into even more processing units depending on the processing content. Also, it can be divided so that one processing unit includes even more processes.

[0287] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions. Embodiments of the present invention provide significant improvements in computer power and functionality. These improvements allow users to utilize computers that provide more efficient and robust interaction with tables, the methods by which information is stored and presented in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of more efficient, powerful, and robust user interfaces. Such user interfaces provide better interaction between humans and machines.

[0288] <Additional Notes> [Appendix 1] A display device capable of executing an operation command associated with a selection gesture for a character string input in an input field, a handwriting input unit that accepts handwriting input; an operation command recognition control unit that determines whether the handwriting input accepted by the handwriting input unit is a selection gesture; an input field definition unit that, when it is determined that the handwritten input is a selection gesture, determines whether a character string is registered in an input field in which the handwritten input is converted into a predetermined character type; When it is determined that the character string is registered in the input field, an operation command associated with the selection gesture is displayed in a selectable manner; a handwriting input display control unit that converts the handwritten input into a predetermined character type when it is determined that the character string is not registered in the input field; A display device comprising: [Appendix 2] If it is determined that the handwritten input is not the selection gesture for the character string registered in the input field, The display device according to claim 1, wherein the handwriting input display control unit replaces the character string registered in the input field with a character string obtained by character recognition of a handwritten input into the input field. [Appendix 3] The handwriting input display control unit 3. The display device according to claim 1, wherein when it is determined that the handwritten input is handwritten input into an input field to be converted into the predetermined character type, the handwritten input is the selection gesture for the character string registered in the input field, and the handwritten input spans all four sides of the input field, the display device displays a command associated with the selection gesture. [Appendix 4] The handwriting input display control unit 3. The display device according to claim 1, wherein, when it is determined that the handwritten input is handwritten input into an input field to be converted into the predetermined character type, the handwritten input is the selection gesture for the character string registered in the input field, and the handwritten input surrounds four sides that form the input field, the display device displays a command associated with the selection gesture. [Appendix 5] The handwriting input display control unit The handwritten input is input into an input field where the handwritten input is converted into the predetermined character type, and The character string is registered in the input field, and 3. The display device according to claim 1, wherein, when the handwriting input is determined to be the selection gesture for the character string, a command associated with the selection gesture is displayed. [Appendix 6] The display device according to any one of appendices 1 to 5, wherein the command associated with the selection gesture is a command to erase, move, rotate, select, thicken, thin, enlarge, shrink, or underline the character string. [Appendix 7] The display device according to claim 6, wherein the command associated with the selection gesture is a command that accepts setting the character string selected by the selection gesture as a page name when one screen corresponds to one page. [Appendix 8] The display device according to claim 6 or 7, wherein the handwriting input display control unit changes the command associated with the selection gesture to be displayed depending on whether the selection gesture is handwritten in an input field in which a character string attribute is set or in an area that is not an input field. [Appendix 9] an analysis unit that accumulates a determination result as to whether the handwritten input is the selection gesture for the character string registered in the input field, and analyzes the determination result; a display accepting unit that displays a determination result by the analyzing unit and accepts, based on the determination result, a change of a condition as to whether the handwritten input is the selection gesture for the character string registered in the input field; 5. The display device according to claim 3 or 4, further comprising: [Explanation of symbols]

[0289] 2 Display device [Prior art documents] [Patent documents]

[0290] [Patent Document 1] Patent No. 3933449

Claims

1. A display device capable of executing an operation command associated with a selection gesture for a character string input in an input field, a handwriting input unit that accepts handwriting input; an operation command recognition control unit that determines whether the handwriting input accepted by the handwriting input unit is a selection gesture; an input field definition unit that, when it is determined that the handwritten input is a selection gesture, determines whether a character string is registered in an input field in which the handwritten input is converted into a predetermined character type; When it is determined that the character string is registered in the input field, an operation command associated with the selection gesture is displayed in a selectable manner; a handwriting input display control unit that converts the handwritten input into a predetermined character type when it is determined that the character string is not registered in the input field; A display device comprising:

2. If it is determined that the handwritten input is not the selection gesture for the character string registered in the input field, The display device according to claim 1 , wherein the handwriting input display control unit replaces the character string registered in the input field with a character string obtained by character recognition of a handwritten input in the input field.

3. The handwriting input display control unit 3. The display device according to claim 1, wherein when it is determined that the handwritten input is handwritten input into an input field to be converted into the predetermined character type, the handwritten input is the selection gesture for the character string registered in the input field, and the handwritten input spans all four sides forming the input field, a command corresponding to the selection gesture is displayed.

4. The handwriting input display control unit 3. The display device according to claim 1, wherein when it is determined that the handwritten input is handwritten input into an input field to be converted into the predetermined character type, the handwritten input is the selection gesture for the character string registered in the input field, and the handwritten input surrounds four sides that form the input field, a command corresponding to the selection gesture is displayed.

5. The handwriting input display control unit The handwritten input is input into an input field where the handwritten input is converted into the predetermined character type, and The character string is registered in the input field, and The display device according to claim 1 or 2, wherein when the handwritten input is determined to be the selection gesture for the character string, a command associated with the selection gesture is displayed.

6. 2. The display device according to claim 1, wherein the command associated with the selection gesture is a command to erase, move, rotate, select, thicken, thin, enlarge, or shrink the character string, or to underline the character string.

7. 7. The display device according to claim 6, wherein the command associated with the selection gesture is a command that accepts setting the character string selected by the selection gesture as a page name when one screen corresponds to one page.

8. 8. The display device according to claim 6, wherein the handwriting input display control unit changes the command associated with the selection gesture to be displayed depending on whether the selection gesture is handwritten in an input field to which a character string attribute is set or in an area other than an input field.

9. an analysis unit that accumulates a determination result as to whether the handwritten input is the selection gesture for the character string registered in the input field, and analyzes the determination result; a display accepting unit that displays a determination result by the analyzing unit and accepts, based on the determination result, a change of a condition as to whether the handwritten input is the selection gesture for the character string registered in the input field; The display device according to claim 3 , further comprising:

10. A display method performed by a display device capable of executing an operation command associated with a selection gesture for a character string entered in an input field, A process for accepting handwritten input; a process of determining whether the received handwritten input is a selection gesture; a process of determining whether a character string is registered in an input field in which the handwritten input is converted into a predetermined character type when the handwritten input is determined to be a selection gesture; a process of displaying an operation command associated with the selection gesture in a selectable manner when it is determined that the character string is registered in the input field; a process of converting the handwritten input into a predetermined character type when it is determined that the character string is not registered in the input field; A display method characterized by performing the above.

11. a display device capable of executing an operation command associated with a selection gesture for a character string entered in an input field; a handwriting input unit that accepts handwriting input; an operation command recognition control unit that determines whether the handwriting input accepted by the handwriting input unit is a selection gesture; an input field definition unit that, when it is determined that the handwritten input is a selection gesture, determines whether a character string is registered in an input field in which the handwritten input is converted into a predetermined character type; When it is determined that the character string is registered in the input field, an operation command associated with the selection gesture is displayed in a selectable manner; a handwriting input display control unit that converts the handwritten input into a predetermined character type when it is determined that the character string is not registered in the input field; A program to function as a

12. A display system in which a display device capable of executing an operation command associated with a selection gesture for a character string input in an input field and an information processing system communicate with each other via a network, The display device includes: a handwriting input unit that accepts handwriting input; a first communication unit that transmits the handwriting input accepted by the handwriting input unit to the information processing system, The information processing system includes: an operation command recognition control unit that determines whether the handwritten input received from the display device is a selection gesture; an input field definition unit that, when it is determined that the handwritten input is a selection gesture, determines whether a character string is registered in an input field in which the handwritten input is converted into a predetermined character type; a second communication unit that, when it is determined that the character string is registered in the input field, transmits an operation command associated with the selection gesture to the display device, and, when it is determined that the character string is not registered in the input field, transmits a character string of a character type converted from the handwritten input to the display device; The display system is characterized in that the display device displays the received operation command or the character string of the character type.

Images