Display device, display method, program, and display system

Abstract

To provide a display device capable of reducing burden on a user when the user inputs in handwriting for executing operation commands.SOLUTION: A display device has a display control unit to display a display part to execute operation commands by accepting operations from a user, a storage unit to associate a second operation command assigned to the display part with information related to a second character string converted from handwriting input which is input to execute the second operation command, and to store the same, a reception unit to accept the selection of a first operation command from operation command candidates associated with a first character string, and a command assignment unit to assign the first operation command to the display part if it is determined that a prescribed condition is satisfied based on the information related to the first character string and the information related to the second character string when the reception unit accepts the selection of the first operation command. The display control unit displays the display part.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to a display device, a display method, a program, and a display system. [Background technology]

[0002] Display devices that use handwriting recognition technology to convert handwritten data into characters and display them on a display are known. Display devices with relatively large touch panels are placed in conference rooms and public facilities and are used by multiple users as electronic whiteboards, etc.

[0003] There is known a technique for improving operability when operating a display device by handwriting input (see, for example, Patent Document 1). Patent Document 1 discloses a technique for displaying an operation command corresponding to a specific character string handwritten, and for executing a corresponding function when the operation command is selected. Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technologies, the amount of effort required by a user to execute an operation command varies depending on the content of the handwritten input by the user. For example, if the number of characters or strokes required to execute an operation command by handwriting is large, the handwritten input requires a lot of effort. Also, if the number of operation commands associated with the character string converted from the handwritten input is large, the user requires a lot of effort to select a desired operation command.

[0005] An object of the present invention is to reduce the effort required by a user when inputting handwriting to execute an operation command. [Means for solving the problem]

[0006] In view of the above-mentioned problems, the present invention provides a display device that displays candidate operation commands associated with a first character string converted from handwritten input, the display device comprising: a display control unit that displays a display component that executes a pre-assigned operation command by receiving an operation from a user; a storage unit that stores a second operation command assigned to the display component and information about the second character string converted from handwritten input entered to execute the second operation command, in association with each other; a reception unit that receives a selection of a first operation command from candidate operation commands associated with the first character string, when the reception unit receives the selection of the first operation command, determines whether a predetermined condition is met based on the information about the first character string and the information about the second character string, and when it determines that the predetermined condition is met, assigns the first operation command selected by the reception unit to the display component; and the display control unit displays the display component to which the first operation command is assigned by the command assignment unit. [Effects of the Invention]

[0007] The present invention can reduce the user's effort when the user inputs handwriting to execute an operation command. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 10 is a diagram showing an example of a display of operation commands. [Figure 2] 10A and 10B are diagrams showing examples of display of icons to which operation command functions are set. [Figure 3] FIG. 1 is an example of an overall configuration diagram of a display device. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device. [Figure 5] FIG. 2 is an example of a functional block diagram illustrating functions of the display device, divided into blocks. [Figure 6] 3A and 3B are diagrams illustrating object data stored in an object data storage unit. [Figure 7]10A and 10B are diagrams illustrating command information stored in a command information storage unit. [Figure 8] 10 is a diagram illustrating use frequency information stored in a use frequency information storage unit. FIG. [Figure 9] 10 is a diagram illustrating current allocation command information stored in a current allocation command information storage unit. FIG. [Figure 10] 10A and 10B are diagrams illustrating an example of an operation guide and selectable candidates displayed on the operation guide. [Figure 11] 10 is a flowchart illustrating an example of a process performed by the display device to associate frequently executed operation commands with icons. [Figure 12] FIG. 10 is an example of a flowchart illustrating a method for assigning an operation command when no operation command is assigned to an icon. [Figure 13] FIG. 2 is an example of a top view of a display device placed flat. [Figure 14] FIG. 10 is a diagram illustrating the number of operation command candidates corresponding to a character string. [Figure 15] 10 is a diagram illustrating current allocation command information stored in a current allocation command information storage unit. FIG. [Figure 16] FIG. 10 is an example of a flowchart illustrating a process in which the display device assigns operation commands to icons based on the number of operation command candidates. [Figure 17] FIG. 10 is a diagram illustrating command information. [Figure 18] 10A and 10B are diagrams illustrating operation commands displayed based on character strings obtained by character recognition of handwritten data. [Figure 19] FIG. 10 is an example of a flowchart illustrating a process in which the display device assigns operation commands to icons based on the number of characters. [Figure 20] FIG. 10 is a diagram illustrating command information. [Figure 21] 10A and 10B are diagrams illustrating operation commands displayed based on character strings obtained by character recognition of handwritten data. [Figure 22]FIG. 10 is an example of a flowchart illustrating a process in which the display device assigns operation commands to icons based on the number of kanji characters. [Figure 23] 10A and 10B are diagrams illustrating operation commands displayed based on character strings obtained by character recognition of handwritten data. [Figure 24] 10 is an example of a flowchart illustrating a process in which the display device assigns operation commands to icons 7 based on the number of strokes. [Figure 25] FIG. 1 is a diagram illustrating an example of a system configuration of a display system. [Figure 26] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing system. [Figure 27] FIG. 2 is an example of a functional block diagram illustrating functions of the display system in blocks. [Figure 28] 10 is an example of a sequence diagram illustrating a process in which a display device and an information processing system communicate with each other to display a recognition result of handwritten data. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A display device and a display method performed by the display device will be described below as an example of an embodiment of the present invention with reference to the accompanying drawings.

[0010] <Example of operation command display> FIG. 1 is a diagram showing an example of the display of operation commands. An operation command is a command that is prepared to operate the display device and instructs the execution of a specific process. The display device displays operation command candidates 510 that are pre-associated with character strings converted from handwritten data by the user. A character string is an object converted by character recognition of handwritten data. A character string can be represented by a character code. A character string and handwritten data that has been determined not to be converted to a character code can be selected by a selection gesture. A character string consists of one or more characters, and characters include hiragana, katakana, kanji, alphabets, numbers, and symbols. When the user selects the operation command they want to execute, the operation command is executed. In FIG. 1, the "■" at the beginning of a line represents an operation command.

[0011] For example, if a user wants to execute "■ page copy," the user inputs handwritten data 301 "page." The display device converts handwritten data 301 into the character string "page," and displays operation command candidates 510 associated with the character string "page." In FIG. 1, five operation command candidates 510 are displayed. When the user selects "■ page copy" from the list, "■ page copy" is executed, and the contents of the entire page are copied.

[0012] In this way, displaying a list of operation commands by handwriting input makes it easier to execute the desired operation command than navigating through multiple levels of hierarchical menus. However, even when executing frequently used operation commands, it was necessary to handwrite a specific string of characters such as "page" each time.

[0013] Therefore, the display device of this embodiment automatically assigns operation commands that require a large number of handwritten characters or strokes to icons, so that the user can execute the operation command by selecting the icon with a pen, eliminating the need for handwritten input required to execute the operation command.

[0014] FIG. 2 shows an example of the display of an icon 7 to which an operation command function is set. In FIG. 1, assume that the user selects "■ Page Copy." Assume that the number of characters or strokes of the characters handwritten to execute "■ Page Copy" is greater than the number of characters or strokes of the characters handwritten to execute the operation command already set in icon 7. The display device 2 sets the function "■ Page Copy" in icon 7. Therefore, the user can execute "■ Page Copy" by pressing this icon 7.

[0015] In this way, the display device of this embodiment assigns the function of an operation command to the icon 7, thereby reducing the effort required for handwritten input to execute an operation command.

[0016] <Terminology> A stroke is a series of operations in which a user presses an input means against a display, moves it continuously, and then releases it from the display. Stroke data is information displayed on a display based on the trajectory of coordinates input by the input means. Stroke data may be interpolated as appropriate. Handwritten data is data that has one or more strokes. Handwritten input refers to the input of handwritten data by a user.

[0017] The character string converted from handwritten data through character recognition may include not only text data, but also data displayed based on user operations, such as stamps displayed as marks, shapes such as circles and stars, and straight lines. Characters include numbers, alphabets, and symbols. Characters are also called text data.

[0018] An operation command selected by the user from among the operation command candidates is called a "first operation command," and an operation command already set in the icon 7 is called a "second operation command."

[0019] A display component is a small picture, image, symbol, or the like that is displayed on a computer and accepts operations from a user. In this embodiment, the term "icon" is used. An icon may also be called a button, form, label, or the like.

[0020] The information about the first character string or the second character string is information extractable from the character string, metadata about the character string, etc. The information about the first character string or the second character string is, for example, the number of operation commands associated with the character string, the number of characters in the character string, the number of kanji characters included in the character string, the number of alphabets, the number of symbols, the number of strokes in the character string, etc.

[0021] The term "assign" means that one of a plurality of operation commands is associated with a display component, and the operation command is executed when the display component is pressed. A plurality of operation commands may be assigned to a display component.

[0022] In this embodiment, even if a threshold is exemplified as a comparison target, the threshold is not limited to the exemplified value. Therefore, in this embodiment, for all thresholds, the terms "less than threshold" and "below threshold" have the same meaning, and the terms "over threshold" and "above threshold" have the same meaning. For example, when the threshold is 11, "below threshold" has the same meaning as "below threshold" when the threshold is 10. Furthermore, when the threshold is 10, "over threshold" has the same meaning as "above threshold" when the threshold is 11.

[0023] [First embodiment] <Device configuration example> The overall configuration of the display device 2 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the arrangement of the display device 2. Fig. 3(a) shows, as an example of the display device 2, a display device 2 used as a horizontally long electronic whiteboard hung on a wall.

[0024] 3(a), a display 220 as an example of the display device 2 is provided on top of the display device 2. A user U can write characters or the like by hand on the display 220 using a pen 2500.

[0025] FIG. 3(b) shows a display device 2 used as a vertically long electronic whiteboard hung on a wall.

[0026] 3(c) shows the display device 2 placed flat on a desk 230. The display device 2 is about 1 cm thick, so there is no need to adjust the height of the desk even when it is placed flat on a regular desk. In addition, the user can easily move the display device 2.

[0027] The pen 2500 can input coordinates using electromagnetic induction, active electrostatic coupling, etc. The pen 2500 may also have functions such as pressure detection, tilt detection, and a hover function (displaying the cursor before the pen touches the screen).

[0028] <Device hardware configuration> Next, the hardware configuration of the display device 2 will be described with reference to Fig. 4. As shown in the figure, the display device 2 has the configuration of an information processing device or a computer. Fig. 4 is an example of a hardware configuration diagram of the display device 2. As shown in Fig. 4, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an SSD (Solid State Drive) 204.

[0029] Of these, the CPU 201 controls the overall operation of the display device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201.

[0030] The SSD 204 stores various data such as an OS and programs for the display device 2. This program may be an application program that runs on an information processing device equipped with a general-purpose OS (Windows (registered trademark), Mac OS (registered trademark), Android (registered trademark), iOS (registered trademark), etc.). In this case, the device is normally used as a general-purpose information processing device, but when a user runs the application program, the user can write by hand, just like on a dedicated device for the display device 2.

[0031] The display device 2 also includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, a power switch 227, a tilt sensor 217, a serial interface 218, a speaker 219, a microphone 221, a wireless communication device 222, an infrared I / F 223, a power control circuit 224, an AC adapter 225, and a battery 226.

[0032] The display controller 213 controls and manages the screen display in order to output an output image to the display 220, etc. The touch sensor 216 detects that the pen 2500, the user's hand, etc. (the pen and the user's hand are input means) has come into contact with the display 220. The touch sensor 216 also receives a pen ID.

[0033] The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 inputs and detects coordinates. The method for inputting and detecting coordinates will be described below. For example, in the case of an optical type, two light receiving and emitting devices installed at both ends of the upper side of the display 220 emit a plurality of infrared rays parallel to the display 220. The two light receiving and emitting devices receive light that is reflected by a reflecting member installed around the periphery of the display 220 and returns along the same optical path as the light emitted by the light receiving element.

[0034] The touch sensor 216 outputs position information of the infrared rays emitted by the two light receiving and emitting devices that are blocked by the object to the touch sensor controller 215, and the touch sensor controller 215 identifies the coordinate position of the contact position of the object. The touch sensor controller 215 also has a communication unit 215a and can communicate wirelessly with the pen 2500. For example, when communication is based on a standard such as Bluetooth (registered trademark), a commercially available pen can be used. If one or more pens 2500 are registered in the communication unit 215a in advance, the user can communicate with the display device 2 without having to perform connection settings to connect the pen 2500 to the display device 2.

[0035] The power switch 227 is a switch for switching ON / OFF the power of the display device 2. The tilt sensor 217 is a sensor for detecting the tilt angle of the display device 2. It is mainly used to detect whether the display device 2 is being used in the installation state shown in FIG. 3(a), FIG. 3(b), or FIG. 3(c), and can automatically change the orientation of characters, etc. depending on the installation state.

[0036] The serial interface 218 is a communication interface with the outside, such as a USB. The serial interface 218 is used for inputting information from the outside, etc. The speaker 219 is used for outputting audio, and the microphone 221 is used for inputting audio. The wireless communication device 222 communicates with a terminal carried by the user, and relays a connection to the Internet, for example. The wireless communication device 222 communicates using Wi-Fi, Bluetooth (registered trademark), or the like, but any communication standard is acceptable. The wireless communication device 222 forms an access point, and when the user sets the SSID (Service Set Identifier) ​​and password they have obtained in the terminal they carry, they can connect to the access point.

[0037] Preferably, the wireless communication device 222 is provided with two access points. a. Access point → Internet b. Access point → Internal network → Internet Access point a is for users outside the company, and users cannot access the company network but can use the Internet. Access point b is for users inside the company, and users can use the company network and the Internet.

[0038] The infrared I / F 223 detects the display device 2 arranged adjacently. The infrared I / F 223 can detect only the display device 2 arranged adjacently by taking advantage of the rectilinearity of infrared rays. It is preferable to provide one infrared I / F 223 on each side, and it can detect in which direction another display device 2 is arranged relative to the display device 2. This expands the screen, making it possible to display handwritten information that was previously handwritten on the adjacent display device 2 (handwritten information on another page, with the area of ​​one display 220 being one page), etc.

[0039] The power supply control circuit 224 controls an AC adapter 225 and a battery 226, which are power sources for the display device 2. The AC adapter 225 converts AC common to commercial power sources into DC.

[0040] If the display 220 is so-called electronic paper, it consumes little or no power to maintain the image display, and therefore can be driven by a battery 226. This makes it possible to use the display device 2 for purposes such as digital signage even in places where it is difficult to connect to a power source, such as outdoors.

[0041] The display device 2 further includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201 shown in FIG.

[0042] The touch sensor 216 is not limited to an optical type, and may be a capacitive touch panel that identifies the contact position by detecting changes in capacitance. The touch sensor 216 may also be a resistive film type touch panel that identifies the contact position by detecting voltage changes across two opposing resistive films. The touch sensor 216 may also be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction generated when a contacting object touches the display unit, or various other detection means may be used. The touch sensor 216 may be of a type that does not require an electronic pen to detect whether or not the pen tip is touching. In this case, a fingertip or a pen-shaped stick can be used to perform a touch operation. The pen 2500 does not have to be a long, thin pen-shaped panel.

[0043] <About the function> Next, the functions of the display device 2 will be described with reference to Fig. 5. Fig. 5 is an example of a functional block diagram that explains the functions of the display device 2 in blocks. The display device 2 has an input reception unit 21, a drawing data generation unit 22, a conversion unit 23, a display control unit 24, a data recording unit 25, a network communication unit 26, a reception unit 27, a determination unit 28, a command processing unit 29, and a command allocation unit 30. Each function of the display device 2 is a function or means that is realized when any of the components shown in Fig. 4 operates in response to an instruction from the CPU 201 in accordance with a program loaded from the SSD 204 onto the RAM 203.

[0044] The input acceptance unit 21 accepts input of a coordinate locus (coordinate point sequence, handwritten data) by detecting the coordinates of the position where an input means such as a pen 2500 touches the touch sensor 216. The drawing data generation unit 22 acquires the coordinates where the tip of the pen 2500 touches from the input acceptance unit 21. The drawing data generation unit 22 generates stroke data by connecting this coordinate point sequence through interpolation.

[0045] The conversion unit 23 performs character recognition processing on stroke data (handwritten data) of the same recognition group handwritten by the user, and converts it into text data (character code). The conversion unit 23 recognizes and converts characters (not only Japanese but also multiple languages ​​such as English), numbers, and symbols (%, $, &, etc.) in parallel with the user's pen operation. The conversion unit 23 also performs shape recognition processing on one or more stroke data (handwritten data) handwritten by the user, and converts them into shapes (lines, circles, triangles, etc.). Various algorithms have been devised for the recognition method, but details will be omitted in this embodiment as existing technology can be used.

[0046] The display control unit 24 displays handwritten data, text converted from the handwritten data, and operation menus for user operation on the display 220. The data recording unit 25 stores handwritten data written on the display device 2, converted text, PC (Personal Computer) screens, files, etc. in the storage unit 40. The network communication unit 26 connects to a network such as a LAN and transmits and receives data to and from other devices via the network. By accepting operations from the user, the display control unit 24 displays display components that execute pre-assigned operation commands.

[0047] The reception unit 27 receives a selection of an arbitrary character from a plurality of conversion candidates that have been character-recognized based on the coordinates of the contact made by the pen 2500 or the like, or receives a menu press. The reception unit 27 receives a selection of a first operation command from the operation command candidates associated with the first character string.

[0048] The determination unit 28 determines whether or not a stroke has been input into the input field based on conditions 1 and 2. That is, the determination unit 28 determines whether or not to display the character-recognized text as text to be input into a table.

[0049] The command processing unit 29 acquires operation command candidates associated with a string that partially matches the character string converted from the handwritten data from the command information storage unit 42. When an operation command is selected from the operation command candidates 510, the command processing unit 29 executes the operation command by referring to the command information storage unit 42. Furthermore, when an icon 7 is pressed, the command processing unit 29 executes the operation command assigned to the icon 7.

[0050] The command allocation unit 30 determines an operation command to be assigned to an icon 7 displayed by the display device 2. When the reception unit 27 receives a selection of a first operation command, the command allocation unit 30 determines whether a predetermined condition is met based on information about the first character string and information about the second character string, and when it determines that the predetermined condition is met, assigns the first operation command, the selection of which is received by the reception unit 27, to the display component. The method of determining an operation command will be described in detail later.

[0051] 4, the storage unit 40 includes an object data storage unit 41, a command information storage unit 42, a usage frequency information storage unit 43, and a currently assigned command information storage unit 44. The storage unit 40 may be located on a network that the display device 2 can access.

[0052] <<Object data>> FIG. 6 is a diagram illustrating the object data stored in the object data storage unit 41. As shown in FIG. The object ID item is identification information that identifies the display data. The Type item is the type of object data, and includes handwriting, text, figure, image, table, etc. Handwriting is stroke data (a sequence of coordinate points). Text is one or more characters or symbols (character code) converted from handwritten data. Figures are geometric shapes such as triangles and squares converted from handwritten data. Images are image data such as JPEG, PNG, and TIFF imported from a PC or the Internet. Tables are one-dimensional or two-dimensional table-like objects. One screen of the display device 2 is called a page. The item of the page is the page number. The coordinates field indicates the position of the object data relative to a predetermined origin on the display device 2. The position of the object data is, for example, the upper left vertex of the bounding rectangle of the object data. The coordinates are expressed, for example, in pixel units on the display. The size item is the width and height of the bounding rectangle of the object data. The input field item indicates the correspondence between the input field and the object entered in the table. For example, the text of object ID=2 is associated with the input field of input field ID=001. Also, the text of object ID=6 is associated with the input field of input field ID=002. Note that handwritten text and figures may also be entered in the table.

[0053] <<Command information>> 7 is a diagram illustrating command information stored in the command information storage unit 42. In the command information storage unit 42, attributes of operation commands are set in association with the command ID (identification information) of each operation command. The contents of each attribute are shown below. ·Name is the display name of the operation command. String is a string that is compared with the string handwritten by the user to execute this operation command. That is, when a string that partially matches the string in String is converted from handwritten data, the corresponding operation command is displayed in the operation guide 500. Command represents the content of the operation command processing. When the user selects an operation command displayed in Name from the operation guide, the command processing unit 29 executes the Command.

[0054] For example, the attribute of the operation command with command ID 001 is Name "Page Copy". Since the String is "Page" and "Copy", when the handwritten data is converted to "Page" or "Copy", "■Page Copy" is displayed in the operation guide 500. When the user selects "■Page Copy", the command "Copy Page" is executed.

[0055] <<Usage Frequency Information>> 8 is a diagram illustrating the use frequency information stored in the use frequency information storage unit 43. The use frequency information storage unit 43 stores the number of times an operation command has been executed in association with the command ID (identification information) of each operation command. The number of times an operation command has been executed is the number of times the display device 2 has executed the operation command by the user selecting one of the operation command candidates 510 displayed in the operation guide 500. The number of times an operation command has been executed may be the total number of times it has been executed in the past, or the number of times it has been executed within a certain period of time in the recent past.

[0056] The number of times an operation command has been executed may be recorded for each user in association with the user's identification information. In this case, it is assumed that the user logs in to the display device or that a user ID is associated with the pen 2500 in advance.

[0057] <<Currently assigned command information>> 9 is a diagram illustrating the currently allocated command information stored in the currently allocated command information storage unit 44. The currently allocated command information storage unit 44 stores the command ID of the operation command currently associated with the icon 7. The currently allocated command information storage unit 44 is an example of a storage unit that stores information about a second character string. The currently allocated command information storage unit 44 also stores, in association with each other, the second operation command assigned to a display component and information about the second character string converted from handwritten input that was input to execute the second operation command. The command processing unit 29 references the command ID in the currently allocated command information and identifies the operation command (more specifically, the command) to be executed from the command information storage unit 42.

[0058] <Example of operation guide display> Next, with reference to FIG. 10, an operation guide 500 that is displayed when converting handwritten data will be described. FIG. 10 shows an example of the operation guide 500 and selectable candidates 530 that are displayed on the operation guide 500. The operation guide 500 is displayed when the user writes handwritten data 504 by hand (by lifting the pen). The operation guide 500 displays operation command candidates 510, handwriting-recognized character string candidates 506, conversion character string candidates 507, and character string / predictive conversion candidates 508. The selectable candidates 530 are the operation command candidates 510, handwriting-recognized character string candidates 506, conversion character string candidates 507, and character string / predictive conversion candidates 508. The selectable candidates 530 excluding the operation command candidates 510 are referred to as character string candidates 539.

[0059] Handwritten data 504 is the character "page" handwritten by the user. A handwritten data rectangular area display 503 is displayed surrounding handwritten data 504. In FIG. 10, operation guide 500 is displayed when three characters are entered, but operation guide 500 is displayed when the user stops writing. Therefore, the number of characters in handwritten data 504 can be any number.

[0060] The handwritten recognition string candidates 506, the converted string candidates 507, and the string / predicted conversion candidates 508 each have their respective string candidates arranged in descending order of probability (in FIG. 10, only one of each is shown for the sake of drawing). The "page" in the handwritten recognition string candidates 506 is a candidate for the recognition result. In this example, the conversion unit 23 correctly recognizes "page".

[0061] The converted string candidates 507 are converted string candidates (e.g., idioms including "頁") converted from the result of kana-kanji conversion of "page" (e.g., "ページ"). The string / predicted conversion candidates 508 are predicted string candidates converted from the converted string candidates 507. In this example, "翻頁" is displayed.

[0062] The operation command candidates 510 are predefined operation command candidates 510 displayed according to the recognized characters. In the example of FIG. 10, the leading character "■" 511 indicates that it is an operation command candidate 510. In FIG. 10, since "page" which is the handwritten recognition string candidate 506 partially matches the String of the command information, five operation command candidates 510 are displayed. The operation command candidates 510 are not always displayed because they are displayed when command information including the converted string is found.

[0063] The operation header 520 has buttons 501, 509, 502, 505. The button 501 accepts the switching operation between predicted conversion and kana conversion. The button 502 performs page operations for candidate display. In the example of FIG. 10, there are 3 pages for candidate display, and currently the first page is being displayed. The button 505 accepts the deletion of the operation guide 500. When the user presses the button 505, the reception unit 27 accepts it, and the display control unit 24 deletes the display other than the handwritten data. The button 509 accepts batch display deletion. When the user presses the button 509, the reception unit 27 accepts it, and the display control unit 24 deletes all the displays shown in FIG. 10 including the handwritten data, enabling the user to rewrite the handwriting from the beginning.

[0064] <Setting operation commands to icons based on the number of times they are executed> Next, referring to Fig. 11, a process in which the display device 2 associates frequently used operation commands with icons 7 will be described. Fig. 11 is a flowchart illustrating a process in which the display device 2 associates frequently executed operation commands with icons 7. The process in Fig. 11 starts when the input receiving unit 21 receives stroke data.

[0065] First, the user inputs handwriting on the display device 2 with the pen 2500. The input accepting unit 21 accepts the handwriting input from the user (S1). Specifically, the input accepting unit 21 accepts input of a coordinate trajectory by detecting the coordinates of the position where the pen 2500 has come into contact. The drawing data generating unit 22 acquires the coordinates where the tip of the pen 2500 has come into contact from the input accepting unit 21, and generates stroke data by connecting the sequence of coordinate points by interpolating them.

[0066] The conversion unit 23 converts the stroke data into a character string candidate by performing existing character recognition. The command processing unit 29 determines whether the character string converted from the handwritten data by the conversion unit 23 partially matches the String in the command information storage unit 42 (S2).

[0067] If the determination in step S2 is Yes, the process proceeds to step S3, and if No, the process proceeds to step S4.

[0068] In step S3, since the character string converted by the conversion unit 23 partially matches the String in the command information storage unit 42, the display control unit 24 displays the operation command having the partially matching String on the operation guide 500 (S3).

[0069] In step S4, the character string converted by the conversion unit 23 does not partially match the String in the command information storage unit 42, so the display control unit 24 displays only the character string candidate 539 (S4). If step S3 has been completed, the operation command candidate 510 and the character string candidate 539 are displayed.

[0070] Next, the receiving unit 27 determines whether or not an operation command candidate 510 has been selected from the operation guide 500 (S5).

[0071] If the determination in step S5 is Yes, the process proceeds to step S6, and if No, the process proceeds to step S7.

[0072] In step S6, since the operation command candidate 510 is selected, the command processing unit 29 executes the selected operation command (S6). For example, if the operation command is "copy page," one page of handwritten data and character strings are copied to the clipboard. The command processing unit 29 also increments the number of executions of the operation command by one. The display control unit 24 also erases the handwritten data.

[0073] In step S7, since character string candidate 539 is selected, display control unit 24 displays selected character string candidate 539 as a character string (S7). Also, display control unit 24 erases the handwritten data.

[0074] Next, the command allocation unit 30 refers to the usage frequency information storage unit 43 and determines whether the number of times the operation command executed this time is executed is greater than the number of times the operation command assigned to the icon 7 is executed (S8). For example, if the number of times the operation command executed this time is executed is "10" and the number of times the operation command assigned to the icon 7 is executed is "9", the determination in step S8 is Yes. In this case, the predetermined condition determined by the command allocation unit 30 is that the number of times the operation command executed this time is executed is greater than the number of times the operation command assigned to the icon 7 is executed.

[0075] If the determination in step S8 is Yes, the process proceeds to step S9, and if the determination is No, the process proceeds to step S10.

[0076] In step S9, the command allocation unit 30 allocates the currently executed operation command to the icon 7 (S9). The command allocation unit 30 replaces the command ID of the operation command currently set in the allocation command information storage unit 44 with the command ID of the currently executed operation command.

[0077] Furthermore, if the icon 7 is not always displayed, the display control unit 24 displays the icon 7 with a shape or name corresponding to the assigned operation command, and if the icon 7 is always displayed, the display control unit 24 updates the shape or name of the icon 7 (S9-2). Note that if the operation command assigned to the icon 7 is changed, the shape or name of the icon 7 may be changed or updated, or the shape of the icon 7 may not be changed or updated. Also, if the operation command assigned to the icon is changed, the operation command may be assigned to an icon other than the icon 7, and both the icon 7 and the other icon may be displayed on the display screen.

[0078] In step S10, the command assignment unit 30 does not change the operation command assigned to the icon 7 (S10).

[0079] Therefore, since an operation command is assigned to the icon 7, the user can execute the operation command by pressing the icon 7 with the pen 2500. Since an operation command that the user frequently uses is set to the icon 7, the operation command can be executed without having to write the character string for executing the operation command by hand, thereby reducing the effort required by the user.

[0080] <<Additional information regarding cases where no operation command is assigned to an icon>> If an operation command is not assigned to the icon 7 (for example, after resetting or immediately after shipping), the first executed operation command is assigned to the icon 7. If an operation command is not assigned to the icon 7, the command assignment unit 30 may assign the executed operation command to the icon 7.

[0081] 12 is a flowchart illustrating a method for assigning an operation command when no operation command is assigned to the icon 7. The process in FIG. 12 is executed, for example, in steps S9 and S10 in FIG.

[0082] The command allocation unit 30 determines whether or not a function is assigned to the icon 7 (S101). Whether or not a function is assigned to the icon 7 can be determined by whether or not a command ID is currently set in the assigned command information storage unit 44.

[0083] If the determination in step S101 is Yes, the process proceeds to step S102, and if the determination is No, the process proceeds to step S103.

[0084] In step S102, as described in FIG. 11, the command allocation unit 30 determines whether or not to allocate the operation command based on the number of times the operation command is executed (S102).

[0085] In step S103, the command allocation unit 30 allocates the operation command selected from the operation guide 500 to the icon 7 (S103). The command allocation unit 30 sets the command ID of the operation command currently executed (selected) in the currently allocated command information storage unit 44.

[0086] <Icon display example> Fig. 13 is a top view of the display device 2 placed flat. In Fig. 13(a), it is assumed that an operation command "page copy" is assigned to the icon 7. In this case, it is preferable that the icon 7 has a shape or appearance that represents "page copy." In Fig. 13(a), the operation command "page copy" is represented by the appearance of two overlapping pages.

[0087] The icon 7 may be displayed constantly, or may be displayed in response to a user operation. The icon 7 may be displayed for at least a certain period of time immediately after the assigned operation command is changed. In this case, the display control unit 24 erases the icon 7 after a certain period of time has elapsed since it was displayed. Even if the icon 7 is displayed constantly, the shape of the icon 7 may change depending on the function of the assigned operation command.

[0088] 13(a), a hover function may be used to display the name of the operation command assigned to the icon 7 in a speech bubble 311. The user can easily determine the operation command assigned to the icon 7 without selecting the icon 7.

[0089] Immediately after the power is turned on, the icon 7 is displayed with the operation command that was assigned to the icon 7 before the power was turned off still assigned to it.

[0090] 13(b), the icon 7 may have a command name 312. In this case, for example, the text "Copy Page" is displayed near or inside the icon 7. The user can easily determine which operation command is assigned to the icon 7. Note that there may be multiple icons 7 to which an operation command is assigned. The command assignment unit 30 assigns operation commands to each icon 7 in descending order of the number of times that the operation command is executed, as many as the number of icons 7. When the command assignment unit 30 changes the assignment to an icon 7, the assignment is changed to the icon 7 to which the operation command executed the least number of times is assigned.

[0091] <Major Effects> The display device 2 of this embodiment assigns frequently used operation commands to icons 7, so that the user is less likely to write by hand to execute operation commands, thereby reducing the effort required for the user to input by hand to execute operation commands.

[0092] [Second embodiment] In this embodiment, a display device 2 will be described that determines the operation command to be assigned to the icon 7 based on the largest number of operation command candidates corresponding to the character string converted from handwritten data.

[0093] In this embodiment, the hardware configuration diagram of FIG. 4 and the functional block diagram of FIG. 5 described in the above embodiment can be used.

[0094] <Candidates for operation commands corresponding to the string> 14 is a diagram illustrating the number of operation command candidates corresponding to a character string. In FIG. 14(a), the character string converted from handwritten data 323 is "page" (an example of a first character string). Five operation command candidates 321, "■ Copy page," "■ Cut page," "■ Paste page," "■ Erase page," and "■ Erase page background," are displayed in operation guide 500 (five is an example of a first number). In other words, five operation commands are associated with the character string converted from handwritten data 323.

[0095] In Fig. 14(b), the character string converted from handwritten data 324 is "address" (an example of a second character string). In operation guide 500, one operation command candidate 322, "■ address book" (one is an example of a second number), is displayed. In other words, one operation command is associated with the character string converted from handwritten data 324.

[0096] Between the character strings "page" and "address," "page" has more associated operation command candidates 321, 322, so the command allocation unit 30 allocates the "■Page Copy" executed by the user to the icon 7 in priority over "■Address Book."

[0097] When there are many options, as in Figure 14(a), selecting a candidate operation command can be time-consuming and can lead to mistakes. When there are many candidate operation commands associated with the character string converted from handwritten data, the user can efficiently execute the operation command by assigning the selected operation command to icon 7.

[0098] The command allocation unit 30 may allocate all of the operation command candidates 321 displayed in the operation guide 500 to the icon 7. In this case, when the user presses the icon 7 with the pen 2500, the operation guide 500 shown in Fig. 14(a) is displayed, and the user can select an operation command from the operation command candidates 321. This widens the range of choices.

[0099] <Current Allocation Command Information Storage Unit of the Present Embodiment> 15 is a diagram illustrating the currently assigned command information stored in the currently assigned command information storage unit 44. The currently assigned command information storage unit 44 stores the command ID of the operation command currently assigned to the icon 7 and the character string converted from handwritten data when this command ID was set. By storing the character string converted from handwritten data, the command assignment unit 30 can identify the number of operation command candidates corresponding to the character string converted from handwritten data.

[0100] In addition, the currently assigned command information storage unit 44 may store the number of operation command candidates 322 corresponding to the character string converted from the handwritten data, rather than or separately from the character string converted from the handwritten data.

[0101] <Process or Action> Next, referring to Fig. 16, a process will be described in which the display device 2 assigns an operation command executed by the user to an icon 7 when the operation command has a large number of operation command candidates corresponding to a character string converted from handwritten data. Fig. 16 is a flowchart illustrating a process in which the display device 2 assigns an operation command to an icon 7 based on the number of operation command candidates. The explanation of Fig. 16 will mainly focus on the differences from Fig. 11. The processes of steps S11 to S17 may be the same as steps S1 to S7 in Fig. 11.

[0102] In step S18, the command allocation unit 30 determines whether the number of operation command candidates 321 associated with the currently handwritten input character string is greater than the number of operation command candidates 322 displayed together with the operation commands already assigned to the icon 7 (S18). The operation command candidates 322 displayed together with the operation commands already assigned to the icon 7 can be determined by searching the command information storage unit 42 for the character string currently input in the assigned command information storage unit 44. In this case, the predetermined condition determined by the command allocation unit 30 is whether the number of operation command candidates 321 associated with the currently handwritten input character string is greater than the number of operation command candidates 322 displayed together with the operation commands already assigned to the icon 7.

[0103] For example, if the number of operation command candidates associated with the character string converted from handwritten data this time is "5" and the number of operation command candidates displayed together with the operation command already assigned to icon 7 is "1", the judgment in step S18 will be "Yes".

[0104] If the determination in step S18 is Yes, the process proceeds to step S19, and if the determination is No, the process proceeds to step S20.

[0105] In step S19, the command allocation unit 30 allocates the currently executed operation command to the icon 7 (S19). The command allocation unit 30 replaces the command ID currently set in the allocation command information storage unit 44 with the command ID of the currently executed operation command. The command allocation unit 30 also replaces the character string input to the current allocation command information storage unit 44 with the character string converted from the currently handwritten data.

[0106] In step S20, the command assignment unit 30 does not change the operation command assigned to the icon 7 (S20).

[0107] <Major Effects> The display device of this embodiment assigns an operation command with a large number of operation command candidates corresponding to the character string converted from handwritten data to icon 7, thereby eliminating the need for the user to select from a large number of operation command candidates and allowing the user to efficiently execute operation commands.

[0108] [Third embodiment] In this embodiment, a display device 2 will be described that determines the operation command to be assigned to the icon 7 based on the number of characters handwritten by the user in order to display operation command candidates.

[0109] In this embodiment, the hardware configuration diagram of FIG. 4 and the functional block diagram of FIG. 5 described in the above embodiment can be used.

[0110] <Candidates for operation commands corresponding to the string> 17 is a diagram for explaining command information used in this embodiment. The explanation of FIG. 17 will mainly focus on the differences from FIG. The Name in FIG. 17 is "Sign Out," which is the display name of the operation command. String is the string that will be compared with the user's handwritten input to execute this command. So, the string could be "Sign" or "Sign out". Command represents the processing content of the operation command "Sign Out." For example, if the command ID is 011, when the user handwrites "Sign" or "Sign Out," "■Sign Out" will be displayed in the operation guide. When the user selects "■Sign Out," the command "Sign Out" will be executed.

[0111] 18A and 18B are diagrams illustrating operation command candidates displayed based on a character string converted from handwritten data. In FIG. 18A, the character string converted from handwritten data 333 is "sign out" (an example of a first character string). In operation guide 500, an operation command candidate 331 called "■ sign out" is displayed, which defines a string that partially matches "sign out". That is, in order to display operation command candidate 331 called "■ sign out", the user handwrites six characters (an example of a first number of characters) for "sign out".

[0112] In Fig. 18(b), the character string obtained by character recognition of handwritten data 334 is "page" (an example of a second character string). Five operation command candidates 332, namely, "■ Copy page," "■ Cut page," "■ Paste page," "■ Erase page," and "■ Erase page background," are displayed in operation guide 500. That is, in order to display these operation command candidates 332, the user handwrote the three characters "page" (an example of a second number of characters).

[0113] Between the handwritten data or character strings "sign out" and "page," the number of characters handwritten for "sign out" is greater, so the command assignment unit 30 assigns the operation command "■Sign out" to icon 7 in priority over "■Copy page."

[0114] If the number of characters required to display an operation command is large, the effort required to write it by hand will be large. Therefore, by assigning an operation command with a large number of characters to icon 7, the user can execute the operation command efficiently.

[0115] <Current Allocation Command Information Storage Unit of the Present Embodiment> The current allocation command information storage unit 44 of this embodiment may be similar to that shown in Fig. 15. However, the character string input to the current allocation command information storage unit 44 may store the number of characters instead of the character string itself or together with the character string.

[0116] <Process or Action> Next, referring to Fig. 19, a process in which the display device 2 assigns an operation command with a large number of handwritten characters to an icon 7 in order to display operation command candidates will be described. Fig. 19 is a flowchart illustrating a process in which the display device 2 assigns an operation command to an icon 7 based on the number of characters. The explanation of Fig. 19 will mainly focus on the differences from Fig. 11. The processes of steps S21 to S27 may be the same as steps S1 to S7 in Fig. 11.

[0117] In step S28, the command assignment unit 30 determines whether the number of characters in the character string input by handwriting this time is greater than the number of characters input by handwriting to execute the operation command already assigned to the icon 7 (S28). The character string input by handwriting includes handwritten data and character strings converted from the handwritten data. In this case, the predetermined condition determined by the command assignment unit 30 is whether the number of characters in the character string input by handwriting this time is greater than the number of characters input by handwriting to execute the operation command already assigned to the icon 7.

[0118] For example, if the number of characters in the character string input by hand this time is "6" and the number of characters in the character string input by hand to execute the operation command assigned to icon 7 is "3", the determination in step S28 will be "Yes".

[0119] If the determination in step S28 is Yes, the command allocation unit 30 allocates the currently executed operation command to the icon 7 (S29). The command allocation unit 30 replaces the command ID of the operation command currently set in the allocation command information storage unit 44 with the command ID of the currently executed operation command. In addition, the command allocation unit 30 replaces the input character string currently set in the allocation command information storage unit 44 with the character string converted from the currently handwritten data.

[0120] If the determination in step S28 is No, the command assignment unit 30 does not change the operation command assigned to the icon 7 (S30).

[0121] <Major Effects> The display device of this embodiment assigns operation commands that require a large number of characters in handwritten data to display the operation command or a large number of character strings converted from handwritten data to icons 7, thereby eliminating the need to handwrite many characters to execute the operation command and allowing the user to execute the operation command efficiently.

[0122] [Fourth embodiment] In this embodiment, a display device 2 will be described that determines the operation command to be assigned to the icon 7 to be the one that includes the largest number of kanji characters in the handwritten data.

[0123] In this embodiment, the hardware configuration diagram of FIG. 4 and the functional block diagram of FIG. 5 described in the above embodiment can be used.

[0124] <Candidates for operation commands corresponding to the string> 20 is a diagram for explaining command information used in this embodiment. The explanation of FIG. 20 will mainly focus on the differences from FIG. The Name in FIG. 20 is "Start a remote conference," "Host a remote conference," and "Join a remote conference," which are display names of operation commands. String is a string that is compared with the string handwritten by the user to execute this operation command. String defines the strings to call each operation command, such as "remote," "start remote conference," "conduct remote conference," and "join remote conference." Command indicates the processing content of each operation command. For example, if the command ID is 021, when the user handwrites "remote", the operation guide will display "■Start remote meeting". When the user selects "■Start remote meeting", the command "Start Meeting" will be executed.

[0125] 21A and 21B are diagrams illustrating operation command candidates displayed based on a character string obtained by character recognition of handwritten data. In FIG. 21A, the character string converted from handwritten data 343 is "remote" (a first character string). In operation guide 500, "■ Start remote conference," "■ Host remote conference," and "■ Join remote conference" are displayed as operation command candidates 341. That is, in order to display these operation command candidates 341, the user handwrote the two-character kanji phrase "remote" (a second kanji character).

[0126] In Figure 21(b), the character string converted from handwritten data 344 is "page" (an example of a second character string). Five operation command candidates 342, namely, "■ Copy page," "■ Cut page," "■ Paste page," "■ Erase page," and "■ Erase page background," are displayed in operation guide 500. That is, in order to display these operation command candidates 342, the user handwrites the three characters "page."

[0127] The handwritten data or the character string converted from the handwritten data, "remote" ("remote"), contains two kanji characters (an example of the third number), while "page" ("page") contains zero kanji characters (an example of the fourth number). Therefore, since "remote" has more kanji characters, the command assignment unit 30 assigns the "■ Start remote conference" executed by the user to the icon 7 in priority over "■ Copy page" and the like.

[0128] Since the effort required to execute an operation command increases when a large number of kanji characters are handwritten, the user can execute the operation command efficiently by assigning the operation command with a large number of kanji characters to icon 7. Note that the same processing can be achieved by replacing the number of kanji characters with the number of alphabets or symbols.

[0129] <Current Allocation Command Information Storage Unit of the Present Embodiment> The current allocation command information storage unit 44 of this embodiment may be the same as that shown in Fig. 15. However, the character string input to the current allocation command information storage unit 44 may not be the character string itself, or the number of kanji characters may be stored together with the character string.

[0130] <Process or Action> Next, referring to Fig. 22, a process in which the display device 2 assigns an operation command that includes a large number of kanji characters handwritten to execute the operation command to the icon 7 will be described. Fig. 22 is a flowchart illustrating a process in which the display device 2 assigns an operation command to the icon 7 based on the number of kanji characters. The explanation of Fig. 22 will mainly focus on the differences from Fig. 11. The processes of steps S31 to S37 may be the same as steps S1 to S7 in Fig. 11.

[0131] In step S38, the command assignment unit 30 determines whether the number of kanji characters contained in the currently handwritten input string is greater than the number of kanji characters contained in the string handwritten to execute the operation command already assigned to the icon 7 (S38). The handwritten input string includes handwritten data and strings converted from the handwritten data. In this case, the predetermined condition determined by the command assignment unit 30 is whether the number of kanji characters contained in the currently handwritten input string is greater than the number of kanji characters contained in the string handwritten to execute the operation command already assigned to the icon 7.

[0132] For example, if the number of kanji characters contained in the string converted from the current handwritten data is "2," and the number of kanji characters contained in the string converted from the handwritten data handwritten to execute the operation command assigned to icon 7 is "0," the determination in step S38 will be "Yes."

[0133] If the determination in step S38 is Yes, the command allocation unit 30 allocates the currently executed operation command to the icon 7 (S39). The command allocation unit 30 replaces the command ID of the operation command currently set in the allocation command information storage unit 44 with the command ID of the currently executed operation command. In addition, the command allocation unit 30 replaces the input character string currently set in the allocation command information storage unit 44 with the character string converted from the currently handwritten data.

[0134] If the determination in step S38 is No, the command allocation unit 30 does not change the operation command allocated to the icon 7 (S40).

[0135] <Major Effects> The display device of this embodiment assigns operation commands that contain a large number of kanji characters in the handwritten data required to display the operation command or in the string converted from the handwritten data to icon 7, thereby eliminating the need for the user to handwrite many kanji characters to execute the operation command, and allowing the user to execute the operation command efficiently.

[0136] [Fifth embodiment] In this embodiment, a display device 2 will be described that determines the operation command to be assigned to the icon 7 to be the one with the largest number of strokes in the character string converted from handwritten data.

[0137] In this embodiment, the hardware configuration diagram of FIG. 4 and the functional block diagram of FIG. 5 described in the above embodiment can be used.

[0138] <Candidates for operation commands corresponding to the string> First, the command information of this embodiment may be the same as that in FIG. 20.

[0139] FIG. 23 is a diagram for explaining an operation command displayed based on a character string converted from handwritten data. FIG. 23 shows the same operation guide 500 as FIG. 21, but in this embodiment, attention is paid to the number of strokes of the character string.

[0140] In FIG. 23(a), the character string converted from the handwritten data 343 is "remote" (an example of the first character string). In the operation guide 500, "■Start Remote Conference", "■Hold Remote Conference", and "■Participate in Remote Conference" are displayed as candidates 341 for operation commands. That is, in order to display these candidates 341 for operation commands, the user handwrote a two-character idiom "remote". The number of strokes of "remote" is 26 strokes (an example of the first number of strokes).

[0141] In FIG. 23(b), the character string converted from the handwritten data 344 is "page" (an example of the second character string). In the operation guide 500, five candidates 342 for operation commands, namely "■Page Copy", "■Page Cut", "■Page Paste", "■Page Erase", and "■Page Background Erase", are displayed. That is, in order to display these candidates 342 for operation commands, the user handwrote the three characters "page". The number of strokes of "page" is 8 strokes (an example of the second number of strokes).

[0142] Therefore, since the number of strokes of "remote" is more than that of "page", the command assignment unit 30 assigns the "■Start Remote Conference" executed by the user to the icon 7 with higher priority than "■Page Copy" or the like. Note that since the number of strokes is approximately the same as the number of strokes of the handwritten data, the display device 2 may make a similar determination regarding the number of strokes as the number of strokes. However, since the conversion unit 23 can convert the handwritten data into a character string even if it is scribbled, the number of strokes and the number of strokes may not match.

[0143] Since the number of strokes in the character string converted from handwritten data is large, and therefore the effort required to write the character string is large, the user can efficiently execute the operation command by assigning the operation command with a large number of strokes in the character string converted from handwritten data to icon 7. The number of strokes may also be used as the comparison target.

[0144] <Current Allocation Command Information Storage Unit of the Present Embodiment> The current allocation command information storage unit 44 of this embodiment may be similar to that shown in Fig. 15. However, the character strings in the current allocation command information storage unit 44 may not be character strings, or the number of strokes in the character strings may be stored together with the character strings.

[0145] <Process or Action> Next, referring to Fig. 24, a process in which the display device 2 assigns an operation command with a large number of strokes in a character string converted from handwritten data to an icon 7 will be described. Fig. 24 is a flowchart illustrating a process in which the display device 2 assigns an operation command to an icon 7 based on the number of strokes. The explanation of Fig. 24 will mainly focus on the differences from Fig. 11. The processes of steps S41 to S47 may be the same as steps S1 to S7 in Fig. 11.

[0146] In step S48, the command assignment unit 30 determines whether the number of strokes in the character string input by handwriting this time is greater than the number of strokes in the character string input by handwriting that is intended to execute the operation command already assigned to the icon 7 (S48). The number of strokes in the character string input by handwriting includes the number of stroke data that constitutes the handwritten data and the number of strokes in the character string converted from the handwritten data. In this case, the predetermined condition determined by the command assignment unit 30 is whether the number of strokes in the character string input by handwriting this time is greater than the number of strokes in the character string input by handwriting that is intended to execute the operation command already assigned to the icon 7.

[0147] For example, if the number of strokes in the character string converted from handwritten data this time is "26" and the number of strokes in the character string converted from handwritten data that was handwritten to execute the operation command assigned to icon 7 is "8", the judgment in step S48 will be "Yes".

[0148] If the determination in step S48 is Yes, the command allocation unit 30 allocates the currently executed operation command to the icon 7 (S49). The command allocation unit 30 replaces the command ID of the operation command currently set in the allocation command information storage unit 44 with the command ID of the currently executed operation command. In addition, the command allocation unit 30 replaces the input character string currently set in the allocation command information storage unit 44 with the character string converted from the currently handwritten data.

[0149] If the determination in step S48 is No, the command allocation unit 30 does not change the operation command allocated to the icon 7 (S50).

[0150] <Major Effects> The display device 2 of this embodiment assigns operation commands with a large number of strokes in the character string entered to execute the operation command to an icon 7, so that the user does not need to handwrite a large number of strokes to execute the operation command, and the user can execute the operation command efficiently.

[0151] [Sixth embodiment] In this embodiment, a display system in a system form in which an information processing system on a network performs character recognition and the like and returns the recognition result to the display device 2 will be described.

[0152] In the description of this embodiment, components or contents of the figures that are given the same symbols as in the first embodiment perform the same functions, so the description of components that have already been described may be omitted or only the differences may be described.

[0153] 25 is an example of a system configuration diagram of a display system 100. The display system 100 has a display device 2 and an information processing system 10 that can communicate with each other via a network N.

[0154] The display device 2 is placed in a facility such as a company, and is connected to a LAN or Wi-Fi installed within the facility. The information processing system 10 is placed, for example, in a data center. The display device 2 is connected to the Internet i via a firewall 8, and the information processing system 10 is also connected to the Internet i via a high-speed LAN or the like within the data center.

[0155] The display device 2 may be connected to the Internet i using wireless communication such as a telephone line network. In this case, the wireless communication may be 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), or the like.

[0156] The information processing system 10 has one or more information processing devices, and the one or more information processing devices act as servers to provide services to the display device 2. A server is a computer or software that functions to provide information and processing results in response to requests from clients. As will be described later, the information processing system 10 receives pen coordinates from the display device 2 and transmits information required to display the operation guide 500 shown in FIG. 10 etc. to the display device 2.

[0157] The configuration of the display device 2 may be the same as that of the first embodiment, but in this embodiment, it is sufficient to have a touch panel, a display, and a communication function. The display device 2 may include multiple computing devices configured to communicate with each other.

[0158] In this embodiment, a general information processing device such as a PC or tablet can execute a web browser or a dedicated application. The web browser or the dedicated application communicates with the information processing system 10. When the web browser is running, the user inputs or selects the URL of the information processing system 10 to connect the display device to the information processing system 10. The display device 2 executes a web application provided by the information processing system 10 in the web browser. A web application refers to software or a mechanism thereof that operates through cooperation between a program written in a programming language (e.g., JavaScript (registered trademark)) that runs on the web browser and a program on the web server side.

[0159] When the dedicated application runs, it connects to a pre-registered URL of the information processing system 10. Since the dedicated application has a program and a user interface, the program sends and receives necessary information to and from the information processing system 10 and displays it on the user interface.

[0160] The communication method may be a general-purpose communication protocol such as HTTP, HTTPs, or WebSocket, or a dedicated communication protocol.

[0161] <Hardware configuration example> The hardware configuration of the display device 2 may be the same as that shown in Fig. 4. In this embodiment, an example of the hardware configuration of the information processing system 10 will be described.

[0162] Fig. 26 is a hardware configuration diagram of the information processing system 10. As shown in Fig. 26, the information processing system 10 is constructed by a computer, and includes a CPU 601, a ROM 602, a RAM 603, an HD 604, an HDD (Hard Disk Drive) controller 605, a display 606, an external device connection I / F (Interface) 608, a network I / F 609, a bus line 610, a keyboard 611, a pointing device 612, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 616.

[0163] Of these, the CPU 601 controls the overall operation of the information processing system 10. The ROM 602 stores programs used to drive the CPU 601, such as an IPL. The RAM 603 is used as a work area for the CPU 601. The HD 604 stores various data, such as programs. The HDD controller 605 controls the reading and writing of various data from and to the HD 604 under the control of the CPU 601. The display 606 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 608 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memories and printers. The network I / F 609 is an interface for data communication using a communication network. The bus line 610 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 601, shown in FIG. 26.

[0164] The keyboard 611 has a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 612 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 614 controls reading and writing of various data from and to a DVD-RW 613, which is an example of a removable recording medium. The DVD-RW may be a DVD-R or the like. The media I / F 616 controls reading and writing (storing) of data from and to a recording medium 615, such as a flash memory.

[0165] <About the device functions> Next, functions of the display system 100 will be described with reference to Fig. 27. Fig. 27 is an example of a functional block diagram showing functions of the display system 100 in blocks. Note that the description of Fig. 27 will mainly focus on the differences from Fig. 5.

[0166] In this embodiment, the display device 2 has an input receiving unit 21, a drawing data generating unit 22, a display control unit 24, a data recording unit 25, a network communication unit 26, and a receiving unit 27. These functions may be the same as those in FIG.

[0167] <<Functions of the information processing system>> The information processing system 10 has a conversion unit 23, a command processing unit 29, a command allocation unit 30, and a communication unit 33. Each function of the information processing system 10 is a function or means realized when any of the components shown in Fig. 26 operates in response to an instruction from the CPU 601 in accordance with a program loaded from the HD 604 onto the RAM 603.

[0168] The communication unit 33 receives stroke data and gesture operations, etc. from the display device 2, and transmits operation commands, etc. to be assigned to the icons 7, to the display device 2. Other functions are the same as those in the first embodiment, or, even if they are different, they will not hinder the explanation of this embodiment. It is assumed that the information processing system 10 has a current allocation command information storage unit 44 that is synchronized with the current allocation command information storage unit 44 of the display device 2.

[0169] <<Action or Processing>> FIG. 28 is a sequence diagram illustrating a process in which the display device 2 and the information processing system 10 communicate with each other to display the recognition result of handwritten data.

[0170] S101: The display device 2 repeats steps S101 to S103 until a pen-up is detected. The network communication unit 26 of the display device 2 transmits stroke data to the information processing system 10.

[0171] S102: The communication unit 33 of the information processing system 10 receives the stroke data, and the conversion unit 23 performs character recognition every time one piece of stroke data is added.

[0172] S103: The display control unit 24 of the display device 2 causes the display 220 to display the stroke data in parallel with the transmission of the stroke data.

[0173] S104: When the user holds the pen up for a certain period of time or longer, the communication unit 33 of the information processing system 10 detects the pen up by not receiving stroke data. The fact that the pen has been held up may be explicitly transmitted.

[0174] S105: When the communication unit 33 of the information processing system 10 detects that the pen has been lifted, the communication unit 33 of the information processing system 10 transmits the character string candidate 539 and operation command candidate stored in the conversion unit 23 to the display device 2.

[0175] S106: The network communication unit 26 of the display device 2 receives the character string candidates and operation command candidates, and the display control unit 24 displays the operation guide 500 including these candidates.

[0176] S107: If a candidate operation command is selected, the accepting unit 27 accepts it, and the network communication unit 26 transmits the character string converted from the handwritten data and the command ID of the selected operation command to the information processing system 10. This character string is included in the character string candidates of step S105.

[0177] S108: Upon receiving the command ID, the communication unit 33 of the information processing system 10 determines that an operation command has been selected, and performs processing to determine an operation command to be assigned to the icon 7 as described in the first to fifth embodiments.

[0178] S109: The command processing unit 29 executes the selected operation command. Also, the command processing unit 29 increments the execution count of the operation command by one. Note that the execution of the operation command may be completed within the display device 2, or may be executed by the information processing system 10. The command processing unit 29 replaces the command ID of the operation command set in the currently allocated command information storage unit 44 for synchronization held by the information processing system 10 with the received command ID.

[0179] S110: The communication unit 33 of the information processing system 10 transmits the command ID of the operation command to be assigned to the icon 7 to the display device 2. The communication unit 33 may transmit the icon 7 itself and the name of the icon 7 to the display device 2.

[0180] S111: The network communication unit 26 of the display device 2 receives the command ID of the operation command to be assigned to the icon 7, and replaces the command ID of the operation command currently set in the assigned command information storage unit 44 with the received command ID. If the icon 7 is not always displayed, the display control unit 24 may display the icon 7 immediately after the assigned operation command is changed. If the icon 7 is always displayed, the display control unit 24 updates at least one of the shape or name of the icon 7 identified by the command ID.

[0181] When the icon 7 is pressed with the pen 2500, the display device 2 transmits the command ID assigned to the icon 7 to the information processing system 10, so that the information processing system 10 can execute the operation command.

[0182] S112: When the character string candidate 539 is selected from the operation guide 500, the accepting unit 27 accepts it, and the display control unit 24 deletes the handwritten data and displays the selected character string candidate 539 as a character string.

[0183] <Major Effects> According to this embodiment, in addition to the effects of the first to fifth embodiments, if the display device 2 is connected to a network, the display device 2 can display a character string converted from handwritten data.

[0184] [Other application examples] The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0185] For example, in this embodiment, the display device 2 assigns the operation command functions to the icons 7, but the operation command functions may be assigned to hard keys.

[0186] Furthermore, the display device 2 may determine the operation command to be assigned to the icon 7 by combining any two or more of the first to fifth embodiments. For example, the display device 2 calculates an evaluation value (which is calculated so that the greater the user's effort, the greater the weighting) by weighting the number of displayed operation command candidates and the number of handwritten characters, and compares it with the evaluation value of the operation command already assigned to the icon 7. The display device 2 assigns the operation command with the greater evaluation value to the icon 7.

[0187] Alternatively, the function of the most recently executed operation command may simply be assigned to the icon 7. Furthermore, if there are multiple icons 7, the display device 2 may replace the operation command to which the oldest function is assigned with the function of the executed operation command.

[0188] Furthermore, in this embodiment, the number of characters actually handwritten by the user is compared, but the string of command information may also be compared. For example, the command assignment unit 30 compares the number of characters in the string with the longest number of characters among all strings contained in the command information of one operation command between the selected operation command and the currently assigned command. The command assignment unit 30 can also compare the number of kanji characters and the number of strokes in a similar manner. The number of characters, the number of kanji characters, or the number of strokes in the string contained in the command information may be compared using the minimum, average, median, etc.

[0189] Although the present embodiment has been described using an electronic whiteboard as an example, the present invention can be suitably applied to any information processing device having a touch panel. Devices having similar functions to electronic whiteboards are also called electronic whiteboards, electronic information boards, interactive boards, etc. An information processing device equipped with a touch panel may also be, for example, a display device using a projector.

[0190] In this embodiment, the coordinates of the pen tip are detected by detecting the coordinates of the pen tip on a touch panel, but the coordinates of the pen tip may also be detected by ultrasonic waves. The pen emits light and also emits ultrasonic waves, and the display device 2 calculates the distance based on the arrival time of the ultrasonic waves. The display device 2 can identify the position of the pen based on the direction and distance. The projector draws (projects) the trajectory of the pen as a stroke.

[0191] In addition, the configuration examples in Figure 5 and the like are divided according to main functions to make it easier to understand the processing by the display device 2. The method of dividing the processing units and their names do not limit the present invention. The processing by the display device 2 can be divided into even more processing units depending on the processing content. Also, it can be divided so that one processing unit includes even more processes.

[0192] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions. Embodiments of the present invention provide significant improvements in computer power and functionality. These improvements allow users to utilize computers that provide more efficient and robust interaction with tables, the methods by which information is stored and presented in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of more efficient, powerful, and robust user interfaces. Such user interfaces provide better interaction between humans and machines.

[0193] <Additional Notes> [Appendix 1] A display device that displays operation command candidates associated with a first character string converted from a handwritten input, a display control unit that displays a display component that executes a pre-assigned operation command in response to an operation from a user; a storage unit that stores a second operation command assigned to the display component and information about a second character string converted from a handwritten input that is input to execute the second operation command, in association with each other; a receiving unit that receives a selection of a first operation command from candidates of the operation command associated with the first character string; a command assignment unit that, when the reception unit receives a selection of the first operation command, determines whether a predetermined condition is satisfied based on information about the first character string and information about the second character string, and, when it determines that the predetermined condition is satisfied, assigns the first operation command, the selection of which is received by the reception unit, to a display component; and The display control unit is a display device that displays a display component to which the first operation command is assigned by the command assignment unit. [Appendix 2] the information about the first character string is a first number of candidates for the first operation command associated with the first character string; the information about the second character string is a second number of candidates for the second operation command associated with the second character string; The display device described in Appendix 1, wherein the command assignment unit assigns the function of the first operation command, the selection of which is accepted by the acceptance unit, to the display component when the first number is greater than the second number. [Appendix 3] the information about the first character string is a first number of characters in the first character string; the information about the second character string is a second number of characters in the second character string; The display device described in Appendix 1, wherein the command assignment unit assigns the function of the first operation command, the selection of which is accepted by the acceptance unit, to the display component when the first number of characters is greater than the second number of characters. [Appendix 4] the information about the first character string is a third number of Chinese characters included in the first character string; the information about the second character string is a fourth number of Chinese characters included in the second character string; The display device described in Appendix 1, wherein the command assignment unit assigns the function of the first operation command, the selection of which is accepted by the acceptance unit, to the display component when the third number is greater than the fourth number. [Appendix 5] the information about the first character string is a first number of strokes of the first character string; the information about the second character string is a second number of strokes of the second character string; The display device described in Appendix 1, wherein the command assignment unit assigns the function of the first operation command, the selection of which is accepted by the acceptance unit, to the display component when the first number of strokes is greater than the second number of strokes. [Appendix 6] If no operation command is assigned to the display component, 6. The display device according to any one of appendices 1 to 5, wherein the command assignment unit assigns a function of the first operation command the selection of which is accepted by the acceptance unit. [Appendix 7] 7. The display device according to claim 1, wherein the display control unit causes the display component to be constantly displayed. [Appendix 8] 7. The display device according to any one of appendices 1 to 6, wherein the display control unit displays the display component at least immediately after a function of the first operation command is assigned to the display component. [Appendix 9] The display device according to any one of appendices 1 to 8, wherein the display control unit displays the display component having a shape corresponding to the function of the first operation command assigned to the display component. [Appendix 10] 9. The display device according to any one of appendices 1 to 8, wherein the display control unit displays a name of the first operation command assigned to the display component together with the display component. [Explanation of symbols]

[0194] 2 Display device [Prior art documents] [Patent documents]

[0195] [Patent Document 1] Japanese Patent Publication No. 2020-064625

Claims

1. A display device that displays operation command candidates associated with a first character string converted from a handwritten input, a display control unit that displays a display component that executes a pre-assigned operation command in response to an operation from a user; a storage unit that stores a second operation command assigned to the display component and information about a second character string converted from a handwritten input that is input to execute the second operation command, in association with each other; a receiving unit that receives a selection of a first operation command from candidates of the operation command associated with the first character string; a command assignment unit that, when the reception unit receives a selection of the first operation command, determines whether a predetermined condition is satisfied based on information about the first character string and information about the second character string, and assigns the first operation command, the selection of which is received by the reception unit, to a display component when it is determined that the predetermined condition is satisfied; and The display control unit displays a display component to which the first operation command is assigned by the command assignment unit.

2. the information about the first character string is a first number of candidates for the operation command associated with the first character string; the information about the second character string is a second number of candidates for the operation command associated with the second character string; The display device according to claim 1 , wherein the command allocation unit allocates the first operation command, the selection of which is accepted by the acceptance unit, to the display component when the first number is greater than the second number.

3. the information about the first character string is a first number of characters in the first character string; the information about the second character string is a second number of characters in the second character string; The display device according to claim 1 , wherein the command assignment unit assigns the first operation command, the selection of which is accepted by the acceptance unit, to the display component when the first number of characters is greater than the second number of characters.

4. the information about the first character string is a third number of Chinese characters included in the first character string; the information about the second character string is a fourth number of Chinese characters included in the second character string; The display device according to claim 1 , wherein the command allocation unit allocates the first operation command, the selection of which is accepted by the acceptance unit, to the display component when the third number is greater than the fourth number.

5. the information about the first character string is a first number of strokes of the first character string; the information about the second character string is a second number of strokes of the second character string; The display device according to claim 1 , wherein the command assignment unit assigns the first operation command, the selection of which is accepted by the acceptance unit, to the display component when the first number of strokes is greater than the second number of strokes.

6. If no operation command is assigned to the display component, 6. The display device according to claim 1, wherein the command assignment unit assigns the first operation command the selection of which is accepted by the acceptance unit.

7. The display device according to claim 1 , wherein the display control unit causes the display component to be constantly displayed.

8. The display device according to claim 1 , wherein the display control unit displays the display component at least immediately after a function of the first operation command is assigned to the display component.

9. The display device according to claim 1 , wherein the display control unit displays the display component having a shape corresponding to the function of the first operation command assigned to the display component.

10. The display device according to claim 1 , wherein the display control unit displays a name of the first operation command assigned to the display component together with the display component.

11. A display method performed by a display device that displays candidates for a first operation command associated with a first character string converted from a handwritten input, the method comprising: a process in which the display control unit displays a display component that executes a pre-assigned operation command in response to an operation from a user; a process in which a receiving unit receives a selection of a first operation command from candidates of operation commands associated with the first character string; When the reception unit receives the selection of the first operation command, information about the first character string; based on information about a second operation command assigned to the display component and a second character string converted from a handwritten input input for executing the second operation command, the information being stored in a storage unit; a command assignment unit determining whether a predetermined condition is satisfied, and when it is determined that the predetermined condition is satisfied, assigning the first operation command, the selection of which is accepted by the acceptance unit, to a display component; a process in which the display control unit displays a display component to which the first operation command is assigned by the command assignment unit; Display method.

12. a display device that displays candidates for a first operation command associated with a first character string converted from a handwritten input; a process in which the display control unit displays a display component that executes a pre-assigned operation command in response to an operation from a user; a process in which a receiving unit receives a selection of a first operation command from candidates of operation commands associated with the first character string; When the reception unit receives the selection of the first operation command, information about the first character string; based on information about a second operation command assigned to the display component and a second character string converted from a handwritten input input for executing the second operation command, the information being stored in a storage unit; a command assignment unit determining whether a predetermined condition is satisfied, and when it is determined that the predetermined condition is satisfied, assigning the first operation command, the selection of which is accepted by the acceptance unit, to a display component; a process in which the display control unit displays a display component to which the first operation command is assigned by the command assignment unit; A program that executes the following.

13. A display system in which a display device and an information processing system communicate with each other via a network, The display device includes: a display component that executes a pre-assigned operation command by accepting an operation from a user, and a display control unit that displays candidates for a first operation command associated with a first character string converted from a handwritten input; a receiving unit that receives a selection of a first operation command from candidates of the operation command associated with the first character string; a network communication unit that, when the reception unit receives a selection of the first operation command, transmits information about the first operation command and information about the first character string to the information processing system; The information processing system includes: a storage unit that stores a second operation command assigned to the display component and information about a second character string converted from a handwritten input that is input to execute the second operation command, in association with each other; a command assignment unit that determines whether a predetermined condition is satisfied based on information about the first character string and information about the second character string transmitted from the display device, and assigns the first operation command, the selection of which is accepted by the acceptance unit, to a display component when it is determined that the predetermined condition is satisfied; a communication unit that transmits to the display device a message indicating that the function of the first operation command is to be assigned to the display component, the display device assigns the function of the received first operation command to the display component in response to the instruction to assign the function of the first operation command to the display component; The display control unit displays a display component to which the first operation command is assigned by the command assignment unit.

Images