Display unit, display method, and program

Abstract

To reduce the time and effort to input, by handwriting, a content similar to at least part of an object that has been already displayed.SOLUTION: A display unit displays a first object based on handwriting, and has: an input receiving unit that receives handwriting of a sign that copies at least part of the first object; and a display control unit that, on the basis of the sign that copies at least part of the first object, displays a second object in which the at least part of the first object is copied.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a display device, a display method, and a program. [Background technology]

[0002] There is a known display device that uses handwriting recognition technology to convert handwritten data into characters and display them on a display. Display devices equipped with relatively large touch panels are placed in conference rooms and public facilities and are used by multiple users as electronic whiteboards.

[0003] There is known a technology for displaying objects such as text data that has been character-recognized so that they are upright when viewed from a user facing a flat display device (see, for example, Patent Document 1). Patent Document 1 discloses a communication support device that determines the user's position by operating a pen device and displays characters in an orientation that corresponds to the user's position. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, even when displaying at least a part of the same content as an object that has already been displayed, the user must again handwrite and input the same content. This is a problem.

[0005] For example, in a hospital, when a table listing patient information is displayed on a display device and hospital staff handwrites information on multiple patients into the table, they may handwrite "male" or "female" to indicate the gender of each patient. When handwriting "male" or "female" into the display device, if the hospital staff wants to add and display a new "male" object written in the patient information that is already displayed, they must perform a separate operation to copy the "male" object by pressing and holding the already displayed "male" object with a pen or the like to select it. In this case, handwriting input takes time and effort, increasing the workload.

[0006] In view of the above-mentioned problems, the present invention provides a technique for reducing the effort required to handwrite and input content similar to at least a part of an object that is already displayed. [Means for solving the problem]

[0007] In view of the above problems, the present invention is a display device that displays a first object based on handwritten input, and includes an input accepting unit that accepts handwritten input of a symbol that copies at least a portion of the first object, and a display control unit that displays a second object in which at least a portion of the first object is copied based on the symbol that copies at least a portion of the first object. Effect of the Invention

[0008] This reduces the effort required to handwrite input the same content as at least a part of an object that is already displayed. [Brief description of the drawings]

[0009] [Figure 1] 11A and 11B are diagrams illustrating assistance in inputting the same text data as already-input text data. [Diagram 2] FIG. 1 is a diagram showing an example of the arrangement of display devices. [Diagram 3] 1 is a diagram illustrating an example of a hardware configuration of a display device. [Figure 4] FIG. 2 is an example of a functional block diagram for explaining functions of the display device by dividing them into blocks. [Diagram 5] 4A and 4B are diagrams illustrating object data stored in an object data storage unit. [Figure 6] 11 is a diagram illustrating input field information stored in an input field information storage unit. FIG. [Figure 7] 11 is a diagram illustrating command information stored in a command information storage unit. FIG. [Figure 8] 11 is a diagram illustrating copy symbol information stored in a copy symbol information storage unit. FIG. [Figure 9]11A and 11B are diagrams illustrating an example of an operation guide and selectable candidates displayed on the operation guide. [Figure 10] 13 is a diagram for explaining a copy process when a user inputs a copy symbol by handwriting, which means the same as above. FIG. [Figure 11] FIG. 13 is a diagram showing an example of an operation guide that is displayed when a user inputs a copy symbol by handwriting, which means the same as above. [Figure 12] FIG. 13 is a diagram showing an example of input in a case where a user handwrites a copy symbol meaning the same as above, but there is no text data in the input field above. [Figure 13] FIG. 13 is a flowchart illustrating an example of a copying process when a copy symbol meaning the above is handwritten input. [Figure 14] 13A and 13B are diagrams illustrating a copy process when a user inputs a copy symbol by handwriting, which means a selective copy. [Figure 15] FIG. 13 is a diagram showing an example of an operation guide that is displayed when a user inputs by hand a copy symbol, which means a selective copy. [Figure 16] FIG. 13 is a diagram showing an example of input in which a user handwrites a copy symbol, which means selective copying, but no text data exists in the input field of the arrowhead. [Figure 17] FIG. 11 is a flowchart illustrating an example of a copying process when a copy symbol indicating a selected copy is input by handwriting. [Figure 18] 13A and 13B are diagrams illustrating a copy process when a user inputs a copy symbol, which means auto-fill copy (range specification), by handwriting. [Figure 19] FIG. 13 is a diagram showing an example of an operation guide that is displayed when a user inputs by handwriting a copy symbol that means auto-fill copy (range specification). [Figure 20] FIG. 13 is a diagram showing an example of input in which a user handwrites a copy symbol meaning auto-fill copy (range specification), but no text data exists in the input field at the arrowhead. [Figure 21] FIG. 13 is a flowchart illustrating an example of a copying process when a copy symbol meaning auto-fill copy (range specification) is input by handwriting. [Figure 22] 13 is a diagram illustrating auto-fill copy (range specification) when text data already entered in an arrowhead input field contains alphabets. FIG. [Figure 23] FIG. 13 is a diagram illustrating auto-fill copy (range specification) when text data already entered in the arrowhead input field includes a fraction. [Figure 24] 13A and 13B are diagrams illustrating a copy process when a user inputs a copy symbol, which means auto-fill copy (all range), by handwriting. [Diagram 25] FIG. 13 is a diagram showing an example of an operation guide that is displayed when a user inputs by handwriting a copy symbol that means auto-fill copy (all range). [Figure 26] FIG. 13 is a diagram showing an example of input in which a user handwrites a copy symbol, which means to autofill copy the entire range, but there is no text data in the input field directly above. [Figure 27] FIG. 13 is a flowchart illustrating an example of a copy process when a copy symbol meaning auto-fill copy (all range) is handwritten input. [Figure 28] 13 is a diagram illustrating auto-fill copy (all range) when text data already entered in an input field includes alphabets. FIG. [Figure 29] 13 is a diagram illustrating auto-fill copy (all range) when text data already entered in an input field includes a fraction. FIG. [Diagram 30] 13A and 13B are diagrams illustrating a copy process when a user inputs a copy symbol, which means autofill copy (continuous), by handwriting. [Diagram 31] FIG. 13 is a diagram showing an example of an operation guide that is displayed when a user inputs by handwriting a copy symbol that means autofill copy (continuous); [Diagram 32] FIG. 13 is a flowchart illustrating an example of a copy process when a copy symbol meaning autofill copy (continuous) is input by handwriting. [Diagram 33] 13 is a diagram illustrating auto-fill copy (continuous) when text data already entered in an input field includes alphabets. FIG. [Diagram 34] FIG. 13 is a diagram illustrating auto-fill copy (continuous) when text data already entered in an input field includes a fraction. [Diagram 35] FIG. 11 is an example of a functional block diagram for explaining functions of a display device by dividing them into blocks (second embodiment); [Diagram 36] 11 is a flowchart illustrating a process in which the display device accepts a setting change based on operation history information. FIG. [Figure 37] FIG. 13 is a diagram showing an example of a proposal screen displayed on a display device. [Figure 38] FIG. 13 is a diagram showing an example of an input field and a displayed operation guide. [Figure 39] 13A and 13B are diagrams showing an example of operation history information of operation commands and an example of an operation guide displayed by handwriting input of a copy symbol. [Diagram 40] FIG. 13 is a diagram illustrating copying when there is no input frame. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A display device and a display method performed by the display device will be described below as an example of an embodiment of the present invention with reference to the drawings.

[0011] [First embodiment] <Outline of input support for the same text data> An outline of text data input support will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining input support for text data that is the same as previously input text data. For example, in Fig. 1(a), two input fields are displayed in advance. The user handwrites "file" into the upper input field 301. "File" is converted into text data by character recognition.

[0012] There are cases where a user wishes to input the same text data as "File" in the upper input field 301 into the lower input field 302. For example, in a medical setting, a display device may display a list of patient information. If a user handwrites "Male" in the gender field for Patient A and has it displayed, but then later wants to display "Male" in the gender field for Patient B as well, then the user must handwrite "Male" again in the gender field for Patient B.

[0013] In such a case, the user inputs "〃", which means the same as above, into the lower input field 302. "〃" is called a copy symbol 303. As shown in FIG. 1(b), the display device 2 copies the text data in the upper input field 301 to the lower input field 302 based on the copy symbol 303 written in the lower input field 302. This copying method is preset. Note that the display device 2 erases the copy symbol 303 before copying the text data.

[0014] In this way, the display device 2 of the present embodiment can copy at least a part of the already-entered text data into the input field by handwriting the copy symbol 303. This can reduce the effort required for the user to handwrite the same content as the same text data that has already been entered.

[0015] <Terminology> The input means may be any means capable of specifying coordinates on a touch panel and writing by hand, such as a pen, a human finger or hand, or a rod-shaped member.

[0016] A stroke refers to a series of operations in which the user presses the input means against the display, moves it continuously, and then lifts it off the display. A stroke includes tracking the user's movement without touching the display. In this case, the display device 2 may initiate a stroke by a user gesture, for example using a mouse or pointing device, by the user pressing a button with the user's hand or foot, or in other ways. Furthermore, the user may terminate a stroke with the same or a different gesture, by releasing a button, or by using a mouse or pointing device.

[0017] Stroke data is information displayed on a display based on a trajectory of coordinates input by an input means. Stroke data may be appropriately interpolated. Handwritten data is data having one or more stroke data. Handwritten input refers to handwritten data being input by a user. Handwritten input may be performed using a touch interface, a tactile object such as a pen or stylus, or the user's body. Handwritten input may also be performed via other types of input, such as gesture-based input, hand tracking input, or other touch-free input by a user. Although embodiments of the present invention refer to handwritten input and handwritten input data, other forms of handwritten input may be utilized.

[0018] An object is an object that is displayed on a display based on stroke data. Object means a target, but in this embodiment it means a display target. Objects converted from stroke data through handwriting recognition may include text data, as well as stamps that are displayed as fixed characters such as "Done," shapes such as circles and stars, and straight lines. Text data is, for example, letters, symbols (!", #, $, etc.), numbers, and alphabets, and is actually converted into character codes.

[0019] An input field is an area surrounded by a frame such as a table cell or a shape, or an area on a ruled line. An input field may also be an area indicated by some kind of line head character such as "·" or ">".

[0020] Duplication means to copy an object exactly as it is. Duplication may also be called a copy. Duplication includes auto-fill copy, in which some characters are changed with a certain regularity. The symbol for copying at least a part of the first object is a symbol handwritten by the user to cause the display device 2 to copy the object, and is a symbol of a predetermined shape. In this embodiment, it is described using the term duplication symbol.

[0021] The "predetermined relationship" refers to a case where the copy symbol is handwritten in a manner that copies an object that has already been entered. The predetermined relationship is, for example, that there is an object around (e.g. directly above) the copy symbol, or that when the copy symbol is an arrow, there is an object in the input field indicated by the arrowhead, and the arrowhead and arrowhead point to a different input field, etc.

[0022] A figure refers to various shapes that are determined according to certain rules. There are many types of figures, such as triangles, rectangles, circles, and diamonds, and figures for creating electronic signatures are preset. A user can display a figure as a figure, and the figure can also be used to select table input. For this reason, in this embodiment, it is determined whether a stroke handwritten by a user is recognized as text or a figure.

[0023] "Directly below" means that it is below the displayed object and is displayed at a certain distance from the object. The input field directly above does not have to be directly above the input field directly below it, and it may be shifted to the left or right. In other words, it is sufficient that the input field directly above and the input field directly below overlap to a certain extent in the left or right direction. The same applies to the term "directly above."

[0024] <Example of device configuration> The overall configuration of the display device 2 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the arrangement of the display device 2. Fig. 2(a) shows, as an example of the display device 2, a display device 2 used as a horizontally long electronic whiteboard hung on a wall.

[0025] 2(a), a display 220 as an example of the display device 2 is provided on the upper part of the display device 2. A user U can use a pen 2500 to handwrite characters or the like on the display 220.

[0026] FIG. 2(b) shows a display device 2 used as a vertically long electronic whiteboard suspended on a wall.

[0027] 2(c) shows the display device 2 placed flat on a desk 230. Since the thickness of the display device 2 is about 1 cm, there is no need to adjust the height of the desk even if the display device 2 is placed flat on a normal desk. In addition, the user can easily move the display device 2.

[0028] The input method of the coordinates of the pen 2500 includes an electromagnetic induction method, an active electrostatic coupling method, etc. The pen 2500 may also have functions such as pen pressure detection, tilt detection, and a hover function (displaying the cursor before the pen touches the screen).

[0029] <Hardware configuration of the device> Next, the hardware configuration of the display device 2 will be described with reference to Fig. 3. As shown in the figure, the display device 2 has the configuration of an information processing device or a computer. Fig. 3 is an example of a hardware configuration diagram of the display device 2. As shown in Fig. 3, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an SSD (Solid State Drive) 204.

[0030] Of these, the CPU 201 controls the overall operation of the display device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201.

[0031] The SSD 204 stores various data such as the OS and programs for the display device 2. This program may be an application program that runs on an information processing device equipped with a general-purpose OS (Windows (registered trademark), Mac OS (registered trademark), Android (registered trademark), iOS (registered trademark), etc.). In this case, it is usually used as a general-purpose information processing device, but when a user executes an application program, the user can input handwritten text in the same way as with a dedicated device for the display device 2.

[0032] The display device 2 also includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, a power switch 227, a tilt sensor 217, a serial interface 218, a speaker 219, a microphone 221, a wireless communication device 222, an infrared I / F 223, a power control circuit 224, an AC adapter 225, and a battery 226.

[0033] The display controller 213 controls and manages the screen display in order to output an output image to the display 220, etc. The touch sensor 216 detects that the pen 2500, the user's hand, etc. (the pen and the user's hand serve as input means) have touched the display 220. The touch sensor 216 also receives a pen ID.

[0034] The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 inputs and detects coordinates. The method of inputting and detecting coordinates will be described. For example, in the case of an optical type, two light receiving and emitting devices installed at both ends of the upper side of the display 220 emit a plurality of infrared rays in parallel to the display 220. The two light receiving and emitting devices receive light that is reflected by a reflecting member installed around the display 220 and returns along the same optical path as the optical path of the light emitted by the light receiving element.

[0035] The touch sensor 216 outputs position information of the infrared rays emitted by the two light receiving and emitting devices that are blocked by the object to the touch sensor controller 215, and the touch sensor controller 215 identifies the coordinate position that is the contact position of the object. The touch sensor controller 215 also has a communication unit 215a and can wirelessly communicate with the pen 2500. For example, when communication is performed according to a standard such as Bluetooth (registered trademark), a commercially available pen can be used. If one or more pens 2500 are registered in advance in the communication unit 215a, the user can communicate with the pen 2500 without performing connection settings to communicate with the display device 2.

[0036] The power switch 227 is a switch for switching ON / OFF the power of the display device 2. The tilt sensor 217 is a sensor for detecting the tilt angle of the display device 2. It is mainly used to detect whether the display device 2 is being used in the installation state of Fig. 2(a), Fig. 2(b), or Fig. 2(c), and can automatically change the thickness of characters, etc. according to the installation state.

[0037] The serial interface 218 is an external communication interface such as a USB. The serial interface 218 is used for inputting information from the outside. The speaker 219 is used for outputting audio, and the microphone 221 is used for inputting audio. The wireless communication device 222 communicates with a terminal carried by the user, and relays a connection to the Internet, for example. The wireless communication device 222 communicates using Wi-Fi, Bluetooth (registered trademark), or the like, but any communication standard may be used. The wireless communication device 222 forms an access point, and when the user sets the SSID (Service Set Identifier) ​​and password obtained by the user in the terminal carried by the user, the user can connect to the access point.

[0038] It is preferable that the wireless communication device 222 is provided with two access points. a. Access point → Internet b. Access point → Internal network → Internet Access point a is for users outside the company, and users cannot access the company network but can use the Internet. Access point b is for users inside the company, and users can use the company network and the Internet.

[0039] The infrared I / F 223 detects the display device 2 arranged adjacently. The infrared I / F 223 can detect only the display device 2 arranged adjacently by utilizing the linearity of infrared rays. It is preferable that one infrared I / F 223 is provided on each side, and it can detect in which direction of the display device 2 another display device 2 is arranged. This expands the screen, and it is possible to display handwritten information that was previously handwritten on the adjacent display device 2 (handwritten information on another page, with the area of ​​one display 220 being one page) and the like.

[0040] The power supply control circuit 224 controls an AC adapter 225 and a battery 226 which are power sources for the display device 2. The AC adapter 225 converts AC common to commercial power sources into DC.

[0041] If the display 220 is so-called electronic paper, it consumes little or no power to maintain the image display, and therefore can be driven by a battery 226. This makes it possible to use the display device 2 for purposes such as digital signage even in places where it is difficult to connect to a power source, such as outdoors.

[0042] The display device 2 further includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201 shown in FIG.

[0043] The touch sensor 216 is not limited to an optical type, and may be a capacitive touch panel that identifies the contact position by detecting a change in capacitance. The touch sensor 216 may be a resistive film type touch panel that identifies the contact position by a voltage change in two opposing resistive films. The touch sensor 216 may be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by a contact object touching the display unit, and various detection means may be used. The touch sensor 216 may be of a type that does not require an electronic pen to detect the presence or absence of a touch with the tip of the pen. In this case, a fingertip or a pen-shaped stick can be used to perform a touch operation. The pen 2500 does not need to be a long, thin pen type.

[0044] <About the function> Next, the functions of the display device 2 will be described with reference to Fig. 4. Fig. 4 is an example of a functional block diagram for explaining the functions of the display device 2 in blocks. The display device 2 has an input reception unit 21, a drawing data generation unit 22, a conversion unit 23, a display control unit 24, a data recording unit 25, a network communication unit 26, an operation reception unit 27, a determination unit 28, and a copy processing unit 30. Each function of the display device 2 is a function or means realized by any of the components shown in Fig. 3 operating according to an instruction from the CPU 201 in accordance with a program loaded from the SSD 204 onto the RAM 203.

[0045] The input reception unit 21 receives input of a coordinate trajectory (coordinate point sequence, handwritten data) by detecting the coordinates of the position where an input means such as a pen 2500 touches the touch sensor 216. The drawing data generation unit 22 acquires the coordinates where the tip of the pen 2500 touches from the input reception unit 21. The drawing data generation unit 22 generates stroke data by connecting the coordinate point sequence by interpolating.

[0046] The conversion unit 23 performs character recognition processing on stroke data (handwritten data) of the same recognition group handwritten by the user, and converts it into text data (character code). The conversion unit 23 recognizes and converts characters (not only Japanese but also multiple languages ​​such as English), numbers, and symbols (%, $, &, etc.) in parallel with the user's pen operation. The conversion unit 23 also performs graphic recognition processing on one or more stroke data (handwritten data) handwritten by the user, and converts them into graphics (lines, circles, triangles, etc.). Various algorithms have been devised for the recognition method, but details will be omitted since existing technology can be used in this embodiment.

[0047] The display control unit 24 displays the handwritten data, text converted from the handwritten data, and an operation menu for user operation on the display 220. The data recording unit 25 stores the handwritten data handwritten on the display device 2, the converted text, and a PC (Personal Computer) screen, files, etc. in the storage unit 40. The network communication unit 26 connects to a network such as a LAN and transmits and receives data to and from other devices via the network.

[0048] The operation acceptance unit 27 accepts the selection of any character from a plurality of conversion candidates that have been character-recognized based on the coordinates touched by the pen 2500 or the like, or accepts the pressing of a menu.

[0049] The determination unit 28 determines whether or not a stroke has been input to the input field based on the conditions 1 and 2. That is, the determination unit 28 determines whether or not to display the text that has been character-recognized as text to be input into a table.

[0050] The copy processing unit 30 analyzes the copy symbol to copy text data that has already been handwritten into an input field to another input field. The copy processing unit 30 can copy using different copy methods depending on the copy symbol. The correspondence between the copy symbol and the copy method is set in the copy information storage unit 44, and the function of copying by the copy method is set in the operation command corresponding to the copy symbol.

[0051] In addition, the display device 2 has a memory unit 40 constructed in the SSD 204 or RAM 203 shown in Figure 3, and the memory unit 40 has an object data memory unit 41, an input field information memory unit 42, a command information memory unit 43, and a copy information memory unit 44.

[0052] <<Object data>> FIG. 5 is a diagram illustrating the object data stored in the object data storage unit 41. As shown in FIG. The object ID item is identification information for identifying the display data. The Type item is the type of object data, and can be handwriting, text, figure, image, table, etc. Handwriting is stroke data (a sequence of coordinate points). Text is one or more characters or symbols (character code) converted from handwritten data. Figures are geometric shapes such as triangles and squares converted from handwritten data. Images are image data such as Jpeg, Png, and Tiff imported from a PC or the Internet. Tables are one-dimensional or two-dimensional table-like objects. One screen of the display device 2 is called a page. The item on a page is its page number. The coordinates item indicates the position of the object data relative to a predetermined origin of the display device 2. The position of the object data is, for example, the upper left vertex of the circumscribing rectangle of the object data. The coordinates are expressed in pixel units of the display 220, for example. The size item is the width and height of the bounding rectangle of the object data. The item of the input field indicates the correspondence between the object entered in the input field and the input field. For example, the text of object ID=2 is associated with the input field of input field ID=001.

[0053] <<Input field information>> 6 is a diagram illustrating the input field information stored in the input field information storage unit 42. The input field information has attributes of each input field. When an object is input into an input field, it is stored as object data in association with the input field on the display 220. The Input Column ID field is the identification information for the input column. For location information items, the coordinates of the input field are indicated by the top left and bottom right corners. The cell size item is the length and width of the input field. The font item is the font of the text to be entered into the input field. The font size is the size of the text entered into the text box. If the font size is set to automatic, the font size will be automatically enlarged or reduced to fit into the text box. The font color is the color of the text entered in the input field. The font alignment item specifies the alignment of the text to be entered into the table in the input field horizontally (left, center, right) and vertically (top, center, bottom). The Margins of the Assignable Area item specifies the amount of margin between the entered text and the frame of the input field as large, medium, or small. The designated dictionary item specifies the conversion dictionary to be applied to the input field. Since the content entered into the input field may be fixed to information from a certain field, the recognition accuracy can be improved by setting a dedicated conversion dictionary for the input field. For example, if "ぎ" is handwritten in an input field to which a medical terminology dictionary is registered, medical terms such as "pseudomembranous colitis" will be displayed as the text to be entered into the table. The Operation Guide item is set to display or hide the operation guide for handwritten data entered in the input field. For example, in an input field where numbers are entered, erroneous conversion rarely occurs, so the user can omit the process of selecting the text to be entered from the operation guide. When the operation guide is not displayed and a copy symbol is handwritten, the operation guide item can also be set to prioritize whether to execute the operation command or to display the copy symbol itself converted to text data.

[0054] There are two types of input fields: those with predefined information as shown in Figure 6, and those where the user can input any information as a graphic. In the latter case, the default values ​​are set for the items in Figure 6, such as font, font size, font color, font layout, input area margin, designated dictionary, and operation guide.

[0055] <<Command information>> 7 is a diagram for explaining command information stored in the command information storage unit 43. In the command information storage unit 43, attributes of operation commands are set in association with the operation command ID (identification information) of each operation command. The contents of each attribute are shown below. ·Name is the display name of the operation command in the operation guide, etc. String is a character string that the user inputs by hand to call up this operation command. In other words, when handwritten data that is converted to text data that partially matches the character string in String is input by hand, the corresponding operation command is displayed in the operation guide. Command represents the content of the operation command processing. When the user selects an operation command displayed by Name from the operation guide, the display device 2 executes the Command. The copy symbol ID is identification information of the copy symbol corresponding to the operation command.

[0056] For example, the attribute of an operation command with an operation command ID of 001 is that the Name is "Copy". Since the String is "〃", "same as above", and "Copy", when the user inputs "〃", "same as above", or "Copy" by handwriting, "■Copy" is displayed in the operation guide. When the user selects "■Copy", an operation command called "Copy Upper Area" is executed. The content of the operation command "Copy Upper Area" is set in the copy method in the copy information storage unit 44.

[0057] <<Copy symbol information>> Fig. 8 is a diagram for explaining the copy symbol information stored in the copy symbol information storage unit. The copy symbol information associates the copy symbol with a copy method. The copy processing unit 30 judges whether the text data converted from the handwritten data by the conversion unit 23 is the same as the copy symbol in Fig. 8, and if it is the same, copies it using the corresponding copy method. The copy method will be described in detail later. The copy symbol with the copy symbol ID of 001 is a copy symbol that means the same as above, and it means that the text data in the input field directly above is copied into the input field where the copy symbol has been handwritten. The copy symbol with ID 002 means that the text data in the input field at the tip of the arrow is copied to the input field at the head of the arrow. The copy symbol with ID 003 means that if there are multiple overlapping input fields, the text data in the input field with the arrow tip will be auto-fill copied to the overlapping input fields. This auto-fill copy can select input fields, so it is called auto-fill copy (range selection). The copy symbol with ID 004 means that the text data in the input field directly above is auto-fill-copied to all input fields below. This auto-fill copy is called auto-fill copy (all range) because it is copied to all input fields below. The copy symbol with ID 005 means that the attribute values ​​of the text data in the upper multiple input fields are adapted and auto-fill copied to the lower multiple input fields. This auto-fill copy is called auto-fill copy (continuous) because it changes the attribute values ​​that have continuity and copies them to the lower input fields.

[0058] The copying method of the copy information storage unit 44 shown in FIG. 8 is described for the purpose of explanation, and the copy processing unit 30 has a function of copying by a copying method corresponding to an operation command.

[0059] <Example of selectable candidates> Next, the operation guide 500 displayed when converting handwritten data will be described with reference to Fig. 9. Fig. 9 shows an example of the operation guide 500 and selectable candidates 530 displayed by the operation guide 500. The operation guide 500 is displayed when the user inputs handwritten data 504 by handwriting (by lifting the pen). The operation guide 500 is displayed when the operation guide 500 is set to be displayed for the input field.

[0060] The operation guide 500 displays operation command candidates 510, handwriting-recognized character string candidates 506, converted character string candidates 507, and character string / predictive conversion candidates 508. Note that selectable candidates 530 are operation command candidates 510, handwriting-recognized character string candidates 506, converted character string candidates 507, and character string / predictive conversion candidates 508. Note that selectable candidates 530 other than operation command candidates 510 are referred to as character string candidates 539.

[0061] Handwritten data 504 is a character string "file" handwritten by the user. Handwritten data rectangular area display 503 is displayed surrounding handwritten data 504. In Fig. 9, operation guide 500 is displayed when four characters are input, but the timing at which operation guide 500 is displayed is the timing at which the user stops handwriting. Therefore, the number of characters in handwritten data 504 is arbitrary.

[0062] The character string candidates in handwriting recognition character string candidate 506, conversion character string candidate 507, and character string / predictive conversion candidate 508 are arranged in descending order of probability (for convenience of drawing, only one of each is shown in FIG. 9). The "file" in the handwriting recognition character string candidate 506 is a candidate for the recognition result. In this example, the conversion unit 23 correctly recognizes "file".

[0063] Conversion character string candidate 507 is a conversion character string candidate (e.g., a phrase containing "file") converted from the result of kana-kanji conversion of "file" (e.g., "file"). Character string / predictive conversion candidate 508 is a predicted character string candidate converted from handwritten recognition character string candidate 506 or conversion character string candidate 507. In this example, "file server", which is predictively converted from "file", is displayed.

[0064] Operation command candidates 510 are predefined operation command candidates that are displayed according to the recognized characters. In the example of Fig. 9, the first character of the line "■" 511 indicates that it is a candidate operation command. In Fig. 9, there are two pieces of command information that partially match the handwriting recognition character string candidate 506 "file", so two operation command candidates 510 are displayed. Operation command candidates 510 are displayed when command information containing the converted character string is found, so they are not always displayed.

[0065] The operation header 520 has buttons 501, 509, 502, and 505. Button 501 accepts an operation to switch between predictive conversion and kana conversion. Button 502 operates the page of candidate display. In the example of FIG. 9, there are three candidate display pages, and the first page is currently displayed. Button 505 accepts the deletion of the operation guide 500. When the user presses button 505, the operation acceptance unit 27 accepts it, and the display control unit 24 erases the display except for the handwritten data. Button 509 accepts the deletion of all displays. When the user presses button 509, the operation acceptance unit 27 accepts it, and the display control unit 24 erases all displays shown in FIG. 9, including the handwritten data, allowing the user to start over with handwriting.

[0066] 9 is an example, and operation command candidates 510 may be displayed below character string candidates 539, or may be displayed in descending order of the past operation history. The user may be able to set the display order.

[0067] <Copying process in the above case> Next, the copying process using some copying symbols will be described in detail with reference to Figs. 10 to 34. First, the copying symbol meaning the same will be described. Fig. 10 is a diagram for explaining the copying process when the user inputs a copying symbol meaning the same by handwriting. In Fig. 10(a), two input fields 311 and 312 are displayed in advance. In the upper input field 311, the user inputs "Create a notification form" by handwriting. This handwritten data is assumed to be converted into text data by character recognition, but it may remain as handwritten data. "Create a notification form" in the input field 311 is an example of a first object, and the input field 311 is an example of a first input field.

[0068] When the user wishes to input the same text data as in the upper input field 311 in the lower input field 312, the user inputs "〃", which is a copy symbol 303 meaning the same as above, in the lower input field 312. The conversion unit 23 converts "〃" into text data. In addition, the conversion unit 23 may convert the handwritten data "〃" into operation command candidates 510, handwritten recognition character string candidates 506, converted character string candidates 507, and character string / predictive conversion candidates 508.

[0069] 11 shows the operation guide 500 that is displayed when the user handwrites the copy symbol 303, which means the same as above. The operation guide 500 is not displayed if the input field information storage unit 42 is set to not display the operation guide 500.

[0070] In FIG. 11, two operation command candidates 510, one handwritten recognition character string candidate 506, one converted character string candidate 507, and one character string / predictive conversion candidate 508 are displayed. A method of displaying the operation command candidates 510 will be described. The copy processing unit 30 searches the command information storage unit 43 for the copy symbol 315 converted from the copy symbol 303, and if the copy symbol 315 partially matches the String, the display control unit 24 displays the operation command in the operation guide 500 (if the copy source text data exists in the input field 311 directly above as described later). In FIG. 11, the copy symbol 315 partially matches the String of the two operation commands, so the two operation command candidates 510 are displayed. Note that "excluding attributes" means that the font, size, color, etc. of the text data are not copied, and the text data is copied with default attributes.

[0071] Next, when the user presses "■ Copy" with the pen 2500, the copy processing unit 30 acquires a copy method of the copy symbol 315 (copying the text data in the input field directly above to the input field below) from the copy information storage unit 44 using the copy symbol ID set in this operation command as a key. The copy processing unit 30 acquires the text data already entered in the input field 311 directly above the input field 312 in which the copy symbol 303 is handwritten from the object data storage unit 41. The copy processing unit 30 passes the acquired text data to the display control unit 24. As shown in FIG. 10(b), the display control unit 24 erases the copy symbol 303, which means the same, and displays "Create a notification form" in the input field 312. "Create a notification form" in the input field 312 is an example of a first object, and the input field 312 is an example of a second input field. The input field 311 directly above the input field 312 is specified by the position information of the input field information storage unit 42.

[0072] Fig. 12 shows an example of input in which a user handwrites a copy symbol 303, meaning the same as above, but there is no text data in the upper input field 313. In Fig. 12(a), two input fields 313, 314 are displayed in advance, but there is no text data in the input field 313 directly above the input field 314 in which the copy symbol 303 has been handwritten.

[0073] The user inputs "〃", which is the copy symbol 303 meaning "same as above", into the input field 314. The conversion unit 23 converts "〃" into the copy symbol 315, which is text data. In this case, since there is no text data in the input field 313 directly above, it is difficult to copy the text data of the input field 313 directly above into the input field 314 in which the copy symbol 303 is handwritten. For this reason, the display control unit 24 does not display the operation command candidate 510 meaning "same as above" in the operation guide 500. When the user selects the copy symbol 315 converted from the copy symbol 303 in the operation guide 500, the copy symbol 315 is displayed in the input field 314 in which the copy symbol 303 is handwritten, as shown in FIG. 12(b). In this manner, the user can also display the copy symbol 315.

[0074] FIG. 13 is a flow chart for explaining a copy process when a copy symbol meaning the same is input by handwriting.

[0075] The user hand-writes a copy symbol 303, which means the same as above, in an input field. The input accepting unit 21 accepts the hand-written input, the drawing data generating unit 22 generates drawing data, and the display control unit 24 displays the copy symbol 303, which means the same as above. The conversion unit 23 converts this copy symbol 303 into a copy symbol 315, which is text data (S1).

[0076] The copy processing unit 30 searches the command information storage unit 43 for the copy symbol 315 converted from the copy symbol 303, and if a corresponding operation command is found, then it acquires a copy method from the copy information storage unit 44. Since the copy method of the copy symbol 303, which means the same as above, is "copy the text data in the input field directly above," the copy processing unit 30 judges whether or not text data exists in the input field 311 directly above the input field 312 in which the copy symbol 303 has been handwritten (S2). The copy processing unit 30 judges, for example, whether or not text data exists in the closest input field above the pen coordinates immediately before the user lifted the pen.

[0077] If the determination in step S2 is Yes, the process proceeds to step S3, and if the determination in step S2 is No, the process proceeds to step S5.

[0078] In step S3, the display control unit 24 displays "■ Copy" and "〃" on the operation guide 500 (S3). When the user selects the operation command candidate 510 "■ Copy" with the pen 2500, the display control unit 24 uses the above-mentioned procedure to display the text data already entered in the input field 311 directly above the input field 312 in which the copy symbol 303 is handwritten (S4).

[0079] In step S5, since there is no text data in the input field 311 directly above the input field 312 in which the copy symbol 303 is handwritten, the display control unit 24 displays the copy symbol 315 converted from the copy symbol 303 in the operation guide 500, but does not display the operation command candidates 510 (S5). In step S3 or S5, when the user selects the copy symbol 315 converted from the copy symbol 303 with the pen 2500, the display control unit 24 displays the copy symbol 315 in the input field 312 in which the copy symbol 303 was handwritten (S6), as shown in Fig. 12(b).

[0080] In this way, when the user wishes to input the same text data as the previously input text data, the user can reduce the labor of inputting the data by handwriting the copy symbol 303, which means the same as the previously input text data.

[0081] <Copy process when selecting copy> Fig. 14 is a diagram illustrating the copying process when the user handwrites copy symbol 304, which means selective copying. In Fig. 14(a), three input fields are pre-displayed. In the upper input field 321, the user handwrites "Create a notification form." "Create a notification form" in input field 321 is an example of a first object, and input field 321 is an example of a first input field. This handwritten data is assumed to be converted into text data by character recognition, but it may remain as handwritten data.

[0082] When a user wishes to input the same text data as in the upper input field 321 in the lower input field 322, the user handwrites and inputs a copy symbol 304 with an arrowhead pointing in the upper input field 321 and an arrowhead pointing in the lower input field 322. The conversion unit 23 converts this copy symbol 304 into text data. Such a copy symbol 304 with a significantly different aspect ratio may be recognized as a graphic. In addition, the conversion unit 23 may convert the handwritten data of this copy symbol 304 into an operation command candidate 510, a handwritten recognition character string candidate 506, a conversion character string candidate 507, and a character string / predictive conversion candidate 508.

[0083] The user may input the copy symbol 304 by handwriting, with the lower input field 322 (where text data has already been input) as the arrowhead and the upper input field 321 as the arrowhead.

[0084] 15 shows an operation guide 500 that is displayed when the user inputs the copy symbol 304, which means a selection copy, by handwriting. The operation guide 500 is not displayed if the input field information storage unit 42 is set to not display the operation guide 500.

[0085] In Fig. 15, two operation command candidates 510 and one handwritten recognition character string candidate 506, one converted character string candidate 507, and one character string / predictive conversion candidate 508 are displayed. A method of displaying the operation command candidates 510 will be described. The copy processing unit 30 searches the command information storage unit 43 for a copy symbol 325, which means a selection copy, and if the copy symbol 325 partially matches the String, the display control unit 24 displays the operation command in the operation guide 500 (if text data is entered in the arrowhead input field 321). In Fig. 15, the copy symbol 325 partially matches the String of the two operation commands, so the two operation command candidates 510 are displayed.

[0086] When the user presses "■ Selected Copy" with the pen 2500, the copy processing unit 30 acquires a copy method of the copy symbol 325 (copying the text data of the arrowhead input field 321 to the arrowhead input field 322) from the copy information storage unit 44 using the copy symbol ID set in this operation command as a key. The copy processing unit 30 acquires the text data already input in the arrowhead input field 321 from the object data storage unit 41. The copy processing unit 30 passes the acquired text data to the display control unit 24. As shown in FIG. 14(b), the display control unit 24 erases the copy symbol 304, which means selected copy, and displays "Create a notification form" in the input field 322. The arrowhead input field 321 is specified by the position information of the input field information storage unit 42. "Create a notification form" in the input field 322 is an example of a second object, and the input field 322 is an example of a second input field.

[0087] Fig. 16 shows an example of input in which a user handwrites a copy symbol 304, which means selective copying, but there is no text data in the arrowhead input field 323. In Fig. 16(a), three input fields are displayed in advance, but there is no text data in any of the input fields.

[0088] The user hand-writes a copy symbol 304 with an arrowhead in the upper input field 323 and an arrowhead pointing to the lower input field 324. The conversion unit 23 converts the hand-written data hand-written as the copy symbol 304 into text data. In this case, since there is no text data in the arrowhead input field 323, it is difficult to copy the text data in the arrowhead input field 323 to the arrowhead input field 324. For this reason, the display control unit 24 does not display operation command candidates 510 in the operation guide 500.

[0089] 16(b), when the user selects the copy symbol 325 on the operation guide 500, the copy symbol (the copy symbol 325 converted from the handwritten data) is displayed in the same size as the copy symbol 304. The display position of the copy symbol 325 may be the arrowhead input field 323 or the arrowhead input field 324.

[0090] On the other hand, if there is no text data in the arrowhead input field 323, the handwritten data may be recognized as a graphic. In this case, the operation guide 500 is not displayed, and instead, as shown in Fig. 16(c), a broken line arrow 326 is displayed connecting the arrowhead input field 323 to the arrowhead input field 324. The copy symbol 325 and the arrow 326 have a similar shape, but while the former can be represented by a character code, the latter is a graphic. In this way, the user can also display the copy symbol 325 or a graphic.

[0091] FIG. 17 is a flow chart for explaining the copy process when a copy symbol 304, which means a selected copy, is handwritten.

[0092] The user hand-writes a copy symbol 304, which means to select and copy, into the input field 322. The input accepting unit 21 accepts the hand-written input of the copy symbol 304, the drawing data generating unit 22 generates drawing data, and the display control unit 24 displays the copy symbol 304, which means to select and copy. The conversion unit 23 converts this copy symbol 304 into a copy symbol 325, which is text data (S11). The copy processing unit 30 may determine whether the arrowhead and arrowhead of the copy symbol 304 are each in contact with another input field or are at least located around the input field.

[0093] The copy processing unit 30 stores the copy symbol 325 in the command information storage unit 43, and if there is a corresponding operation command, it then acquires the copy method from the copy information storage unit 44. Since the copy method of the copy symbol 325, which means selective copying, is "copy the text data in the arrowhead input field to the arrowhead input field," the copy processing unit 30 determines whether or not text data exists in the arrowhead input field 321 (S12). The arrowhead may be within the input field 321, or may be within a certain distance from the input field 321.

[0094] If the determination in step S12 is Yes, the process proceeds to step S13, and if the determination in step S12 is No, the process proceeds to step S15.

[0095] In step S13, the display control unit 24 displays the copy symbol 325 converted in step S11, "■ Select and copy," on the operation guide 500 (S13). When the user selects the operation command candidate 510, "■ Select and copy," with the pen 2500, the display control unit 24 displays the text data already input in the arrowhead input field 321 in the above-described procedure in the arrowhead input field 322 (S14).

[0096] In step S15, since there is no text data in the arrowhead input field 323, the display control unit 24 displays the copy symbol 325 converted in step S11, etc. on the operation guide 500 (S15). When the user selects the copy symbol 325 converted in step S11 with the pen 2500 in step S13 or S15, the display control unit 24 displays the copy symbol 325 converted in step S11 in the arrowhead input field 324 (S16), as shown in Fig. 16(b).

[0097] In this way, when the user wishes to input the same text data as previously input text data into an arbitrary input field, the user can reduce the effort of inputting by handwriting copy symbol 304, which means to selectively copy the data.

[0098] <Copy process when using AutoFill Copy (range specification)> Fig. 18 is a diagram for explaining the copy process when the user handwrites a copy symbol meaning autofill copy (range specification). In Fig. 18(a), the user handwrites "Creating file 1" in an arbitrary input field 331. "Creating file 1" in input field 331 is an example of a first object, and input field 331 is an example of a first input field. This handwritten data is converted into text data by character recognition.

[0099] When the user wishes to input text data that is the same as this input field 331 but with characters 333 of a specified attribute changed incrementally into a series of input fields 332 below, the user handwrites a copy symbol 305 as shown in Fig. 18(a). This copy symbol 305 has an arrowhead that overlaps the input field 331 (where the original text data has already been input), passes through the series of input fields 332 to be copied, and has an arrowhead that points to the lowest input field in the series of input fields 332.

[0100] The conversion unit 23 converts this duplicated symbol 305 into text data. Such duplicated symbols 305 with greatly different aspect ratios may be recognized as figures. In addition, the conversion unit 23 may convert the handwritten data of this duplicated symbol 305 into operation command candidates 510, handwritten recognition character string candidates 506, conversion character string candidates 507, and character string / prediction conversion candidates 508.

[0101] The user may handwrite a copy symbol 305, which points to a lower input field where text data has already been input, and passes through multiple input fields above.

[0102] 19 shows an operation guide 500 that is displayed when the user handwrites a copy symbol that means autofill copy (range specification). The operation guide 500 is not displayed if the input field information storage unit 42 is set to not display the operation guide 500.

[0103] In Fig. 19, two operation command candidates 510, one handwritten recognition character string candidate 506, one converted character string candidate 507, and one character string / predictive conversion candidate 508 are displayed. A method of displaying the operation command candidates 510 will be described. The copy processing unit 30 searches the command information storage unit 43 for a copy symbol 335 converted from a copy symbol 305 meaning autofill copy (range specification), and if the copy symbol 335 partially matches the String, the display control unit 24 displays the operation command in the operation guide 500 (if there is text data in the input field of the arrowhead). In Fig. 19, the copy symbol 335 partially matches the String of the two operation commands, so the two operation command candidates 510 are displayed.

[0104] When the user presses "■ Autofill copy (range specification)" with the pen 2500, the copy processing unit 30 acquires a copy method (when the copy symbol overlaps with multiple input fields, the text data in the arrowhead input field 331 is autofill-copied to the overlapping input fields) from the copy information storage unit 44 using the copy symbol ID associated with the operation command as a key. The copy processing unit 30 acquires the text data already entered in the arrowhead input field 331 from the object data storage unit 41. Next, the copy processing unit 30 detects characters 333 with a predetermined attribute, such as numbers, from the text data. The copy processing unit 30 creates text data in which the characters are incremented by a preset increment according to the characters with the predetermined attribute, for example. For example, when the text data in the input field 331 with the arrowhead is "Creating file 1", the character 333 with the predetermined attribute is the number "1", so the increment is set to "1". The copy processing unit 30 creates "Creating file 2," "Creating file 3," ... "Creating file 10," and passes the created text data to the display control unit 24. The number of pieces of text data created is the number of input fields through which the copy symbol 305 passed, not including the input field 331 that the arrowhead overlaps.

[0105] 18(b), the display control unit 24 erases the copy symbol 305, which means auto-fill copy (range specification), and displays the text data received from the copy processing unit 30 in a series of input fields 332 from top to bottom. The text data "Creating file 2" to "Creating file 10" are an example of a second object, and the input field 332 is an example of a second input field. The series of input fields 332 into which the text data is copied are the input fields through which the copy symbol 305 passed, and are therefore specified by the position information of the input field information storage unit 42.

[0106] Also, if the character of the predetermined attribute is a number, the increment may be "-1." That is, the decrement may be determined. For example, the user sets in advance in the display device 2 whether to increase or decrease the number.

[0107] Fig. 20 shows an example of input in which a user handwrites a copy symbol 305, which means autofill copy (range specification), but there is no text data in the arrowhead input field 331. In Fig. 20(a), there is no text data in any of the input fields.

[0108] 18(a) is handwritten by the user. The conversion unit 23 converts the copy symbol 305 into a copy symbol 335, which is text data. In this case, since there is no text data in the arrowhead input field 331, it is difficult to copy the text data in the arrowhead input field 331 into the series of input fields 332. For this reason, the display control unit 24 does not display the operation command candidates 510 in the operation guide 500.

[0109] When the user selects the copy symbol 335 on the operation guide 500, as shown in Fig. 20(b), a copy symbol (copy symbol 335 converted from handwritten data) is displayed at the position where the copy symbol 305 was. The display position of the copy symbol 335 may be any one of the series of input fields 332.

[0110] On the other hand, if there is no text data in the arrowhead input field 331, the handwritten data may be recognized as a graphic. In this case, the operation guide 500 is not displayed, and a graphic arrow 336 is displayed that starts from the arrowhead input field 331 and passes through a series of input fields 332, as shown in FIG. 20(c). Since it is a graphic, the arrow 336 is not handwritten but is formed as a straight line. The copy symbol 335 and the arrow 336 have a similar shape, but the former can be expressed by character codes, while the latter is a graphic. In this way, the user can display the copy symbol 335 or a graphic.

[0111] FIG. 21 is a flow chart for explaining the copy process when a copy symbol 305, which means auto-fill copy (range specification), is handwritten.

[0112] The user hand-writes the copy symbol 305, which means auto-fill copy (range designation). The input accepting unit 21 accepts the hand-written input of the copy symbol 305, the drawing data generating unit 22 generates drawing data, and the display control unit 24 displays the copy symbol 305, which means auto-fill copy (range designation). The conversion unit 23 converts this copy symbol 305 into a copy symbol 335, which is text data (S21).

[0113] The copy processing unit 30 stores the copy symbol 335 in the command information storage unit 43, and if there is a corresponding operation command, it then acquires the copy method from the copy information storage unit 44. The copy method of the copy symbol 335, which means autofill copy, is "when overlapping with multiple input fields, autofill copy the text data of the arrowhead input field to the overlapping input field", so the copy processing unit 30 judges whether the copy symbol 305 overlaps with multiple input fields (S22). The judgment of whether there are multiple input fields does not include the input field 331 in which the text data of the copy source has already been entered.

[0114] If the determination in step S22 is Yes, the process proceeds to step S23, and if the determination in step S12 is No, the process proceeds to step S27.

[0115] In step S23, the copy processing unit 30 determines whether or not text data exists in the input field 331 overlapping with the arrowhead of the copy symbol 305 (S23). The arrowhead may be within the input field 331, or may be within a certain distance from the input field 331.

[0116] If the determination in step S23 is Yes, the process proceeds to step S24, and if the determination in step S23 is No, the process proceeds to step S28.

[0117] In step S24, the display control unit 24 displays "■ Autofill copy (range specification)" on the operation guide 500, as well as the copy symbol 335 converted from the copy symbol 305 in step S21 (S24). When the user selects the operation command candidate 510 "■ Autofill copy (range specification)" with the pen 2500, the copy processing unit 30 determines whether or not a character with a predetermined attribute exists in the text data already entered in the input field 331 at the arrowhead of the copy symbol 305 (S25).

[0118] If the determination in step S25 is Yes, the process proceeds to step S26, and if the determination in step S25 is No, the process proceeds to step S30.

[0119] In step S26, the copy processing unit 30 creates a plurality of text data items that are changed in increments that are set in advance according to characters of a predetermined attribute, and passes the created text data items to the display control unit 24. The number of text data items created is the number of input fields 332 that overlap the copy symbol 305, excluding the input field 331 that overlaps the arrowhead. The display control unit 24 displays the passed text data in the input fields, from top to bottom, in the input fields 332 that overlap the copy symbol 305 (S26).

[0120] In step S27, since the copy symbol 305 does not overlap multiple input fields, it is treated as a process of copying the text data of the input field 331 to one input field 332 (S27). Therefore, the process of step S27 is the same as the process of the copy symbol 303, which means the same as above.

[0121] In step S28, since there is no text data in the input field 331 of the arrowhead of the copy symbol 305, the display control unit 24 displays the copy symbol 335 converted from the copy symbol 305 in step S21 in the operation guide 500, and does not display the operation command candidates 510 (S28). When the user selects the copy symbol 335 with the pen 2500 in step S24 or S28, the display control unit 24 displays the copy symbol 335 converted from the copy symbol 305 (S29), as shown in Fig. 20(b).

[0122] In step S30, since there is no character with a predetermined attribute in the text data of the input field 331, the copy processing unit 30 simply creates text data (the same text data as the input fields 331) for the number of input fields 332, and passes it to the display control unit 24. The display control unit 24 causes the passed text data to be displayed in the input fields 332 overlapping with the copy symbol 305, from the top down (S30).

[0123] In this way, when the user wishes to change the attributes of already entered text data and enter the data into multiple input fields, the user can reduce the effort required for input by handwriting copy symbol 305, which means auto-fill copy (range specification).

[0124] Another example of autofill copy (range specification) will be described with reference to Figs. 22 and 23. Fig. 22 is a diagram for explaining autofill copy (range specification) in the case where the text data already entered in the arrowhead input field 331 includes alphabets. In Fig. 22, the text data already entered in the arrowhead input field 331 is "Creating file A", so the character 337 of the specified attribute is the alphabet "A". Therefore, since the character of the specified attribute is alphabets, the copy processing unit 30 determines that the increment is one character in alphabetical order. The copy processing unit 30 creates "Creating file B", "Creating file C", ... "Creating file J", and passes the created text data to the display control unit 24.

[0125] FIG. 23 is a diagram for explaining autofill copy (range specification) in the case where the text data already entered in the arrowhead input field 331 includes a fraction. In FIG. 23, the text data already entered in the arrowhead input field 331 is "Creating file_1 / 1", so the character 338 of the specified attribute is a fraction "1 / 1". Therefore, since the character of the specified attribute is a fraction, the copy processing unit 30 determines that the increment is the denominator "1" of the fraction. The copy processing unit 30 creates "Creating file_1 / 2", "Creating file_1 / 3", ... "Creating file 1 / 10", and passes the created text data to the display control unit 24. Note that if the character of the specified attribute is a fraction, the increment may be the numerator "1".

[0126] <Copy process when using AutoFill Copy (all range)> Fig. 24 is a diagram illustrating the copy process when the user handwrites copy symbol 306, which means autofill copy (all range). In Fig. 24(a), the user handwrites "Creating file 2" in the second input field 341. "Creating file 2" in input field 341 is an example of a first object, and input field 341 is an example of a first input field. This handwritten data is converted into text data by character recognition.

[0127] When the user wishes to input text data that is the same as the second input field 341 but with characters 347 of a specified attribute changed incrementally into all input fields 342, 343 below input field 341, the user handwrites a copy symbol 306 as shown in Fig. 24(a). This copy symbol 306 is three dots lined up vertically and input into input field 342 located directly below (below) input field 341, which is the source of the copy.

[0128] The conversion unit 23 converts this duplicated symbol 306 into text data, that is, duplicated symbol 345. In addition, the conversion unit 23 may convert the handwritten data of this duplicated symbol 306 into operation command candidates 510, handwritten recognition character string candidates 506, conversion character string candidates 507, and character string / prediction conversion candidates 508.

[0129] 25 shows an operation guide 500 that is displayed when the user handwrites a copy symbol that means autofill copy (all range). The operation guide 500 is not displayed if the input field information storage unit 42 is set to not display the operation guide 500.

[0130] In Fig. 25, two operation command candidates 510, one handwritten recognition character string candidate 506, one converted character string candidate 507, and one character string / predictive conversion candidate 508 are displayed. A method of displaying the operation command candidates 510 will be described. The copy processing unit 30 searches the command information storage unit 43 for a copy symbol 345 converted from a copy symbol 306 meaning autofill copy, and if the copy symbol 345 partially matches the String, the display control unit 24 displays the operation command in the operation guide 500 (if text data is input in the input field 341 directly above). In Fig. 25, the copy symbol 345 partially matches the String of the two operation commands, so the two operation command candidates 510 are displayed.

[0131] When the user presses "■ Autofill copy (range specification)" with the pen 2500, the copy processing unit 30 obtains a copy method (autofill copy the text data in the input field directly above to all input fields below) from the copy information storage unit 44 using the copy symbol ID associated with the operation command as a key. The copy processing unit 30 obtains the text data already entered in the input field 341 directly above from the object data storage unit 41. Next, the copy processing unit 30 detects characters 347 with a predetermined attribute, such as numbers, from the text data. The copy processing unit 30 creates text data in which the text data is increased by a preset increment according to the characters 338 with the predetermined attribute. For example, if the text data in the input field 341 directly above is "Creating file 2", the character with the predetermined attribute is the number "2", so the increment is set to "1". The copy processing unit 30 creates "Creating file 3", "Creating file 4", ... "Creating file 7", and passes the created text data to the display control unit 24.

[0132] As shown in FIG. 24(b), the display control unit 24 erases the copy symbol 306, which means autofill copy, and displays the text data in each of the input fields 342 and 343 in order from the top. The input fields 342 and 343, "Creating file 3" to "Creating file 7", are an example of a second object, and the input field 342 is an example of a first input field. The input fields 342 and 343 to which the text data is copied are input fields that overlap the input field 341, in which the copy symbol 306 is handwritten, by a certain percentage or more in the width direction. Such input fields are specified by the position information of the input field information storage unit 42. Alternatively, the input field 343 to which the text data is copied can be all input fields that are lined up in the same direction as the input fields 341 and 342 (the distance between the input fields is within a certain range).

[0133] In FIG. 24(a), it is determined whether the text data already entered in the input field 341 directly above the input field 342 in which the copy symbol 306 is handwritten contains characters with a predetermined attribute, but the text data already entered in the multiple input fields 349, 341 above the input field 342 in which the copy symbol 306 is handwritten may be referenced. For example, if the text data already entered in the two input fields 349, 341 are "Creating file 1" and "Creating file 2", the difference in the numbers (predetermined attributes) is 1, so the copy processing unit 30 creates "Creating file 3", "Creating file 4", ... "Creating file 7" by changing the numbers by the difference amount. If the text data already entered in the two input fields 349, 341 are "Creating file 1" and "Creating file 3", the difference in the numbers (predetermined attributes) is 2, so the copy processing unit 30 creates "Creating file 5", "Creating file 7", ... "Creating file 13". The amount of difference can be controlled by referring to the text data already entered in a plurality of input fields 349, 341 above the input field 342 in which the copy symbol 306 is handwritten.

[0134] Fig. 26 shows an example of input in which a user handwrites a copy symbol 306, which means a full range autofill copy, but there is no text data in the input field 341 directly above it. In Fig. 26(a), there is no text data in the input field 341 directly above the input field 342 in which the copy symbol 306 has been handwritten.

[0135] 24(a) in the input field 342. The conversion unit 23 converts the copy symbol 306 into a copy symbol 345, which is text data. In this case, since there is no text data in the input field 341 directly above, the copy processing unit 30 cannot copy the text data to the input field 342 in which the copy symbol 306 is handwritten and all of the input fields 343 below. For this reason, the display control unit 24 does not display the operation command candidates 510 in the operation guide 500.

[0136] When the user selects the copy symbol 345 on the operation guide 500, the copy symbol 345 converted from the copy symbol 306 is displayed in the input field 342 in which the copy symbol 306 is handwritten, as shown in Fig. 26(b). In this manner, the user can also display the copy symbol 345.

[0137] FIG. 27 is a flow chart for explaining the copy process when a copy symbol 306, which means auto-fill copy (all range), is handwritten.

[0138] The user hand-writes a copy symbol 306, which means auto-fill copy (all range). The input accepting unit 21 accepts the hand-written input, the drawing data generating unit 22 generates drawing data, and the display control unit 24 displays the copy symbol 306, which means auto-fill copy (all range). The conversion unit 23 converts this copy symbol 306 into a copy symbol 345, which is text data (S31).

[0139] The copy processing unit 30 stores the copy symbol 345 in the command information storage unit 43, and if there is a corresponding operation command, next acquires the copy method from the copy information storage unit 44. Since the copy method of the copy symbol 306, which means auto-fill copy (all range), is "auto-fill copy the text data in the input field directly above to the multiple input fields below," the copy processing unit 30 judges whether or not text data exists in the input field 341 directly above the input field 342 in which the copy symbol 306 is handwritten (S32). The copy processing unit 30 judges, for example, whether or not text data exists in the closest input field above the pen coordinates immediately before the user lifted the pen.

[0140] If the determination in step S32 is Yes, the process proceeds to step S33, and if the determination in step S32 is No, the process proceeds to step S36.

[0141] In step S33, the display control unit 24 displays "■ Autofill copy (all range)" and the copy symbol 345 converted in step S31 on the operation guide 500 (S33). When the user selects the operation command candidate 510 "■ Autofill copy (all range)" with the pen 2500, the copy processing unit 30 determines whether or not a character with a predetermined attribute exists in the text data of the input field 341 directly above the input field 342 in which the copy symbol 306 is handwritten (S34).

[0142] If the determination in step S34 is Yes, the process proceeds to step S35, and if the determination in step S34 is No, the process proceeds to step S38.

[0143] In step S35, the copy processing unit 30 creates a plurality of text data items that are changed in increments that are set in advance according to characters of a predetermined attribute, and passes the created text data items to the display control unit 24. The number of text data items created is the number of all input fields 343 below, including the input field 342 in which the copy symbol 306 is handwritten. The display control unit 24 displays the passed text data items in order from the top of all input fields 343 below, including the input field 342 in which the copy symbol 306 is handwritten (S35).

[0144] In step S36, since there is no text data in the input field 341 directly above the input field 342 in which the copy symbol 306 is handwritten, the display control unit 24 displays the copy symbol 345 converted from the copy symbol 306 in step S31 in the operation guide 500, and does not display the operation command candidates 510 (S36). In step S33 or S36, when the user selects the copy symbol 345 with the pen 2500, the display control unit 24 displays the copy symbol 345 in the input field 342 in which the copy symbol 306 is handwritten (S37).

[0145] In step S38, since there are no characters with the predetermined attribute in the input field 341, the copy processing unit 30 creates the same text data as the input field 341 for each of the input fields 342, 343 below the input field 341, and passes this data to the display control unit 24. The amount of text data created is the number of all input fields 343 below, including the input field 342 in which the copy symbol 306 is handwritten. The display control unit 24 displays the passed text data in order from the top down, for all of the input fields 343 below, including the input field 342 in which the copy symbol 306 is handwritten (S38).

[0146] In this way, when the user wishes to change the attributes of already entered text data and enter it into all the input fields below, the user can reduce the effort of inputting by handwriting the copy symbol 306, which means auto-fill copy (all range).

[0147] Another example of copying by AutoFill Copy will be described with reference to Figs. 28 and 29. Fig. 28 is a diagram for explaining AutoFill Copy (all range) when the text data already entered in the input field 341 includes alphabets. In Fig. 28, the text data already entered in the input field 341 directly above the input field 342 in which the copy symbol 306 is handwritten is "Creating file B", so the character 338A of the specified attribute is the alphabet "B". Therefore, since the character of the specified attribute is the alphabet "B", the copy processing unit 30 determines that the increment is one character in alphabetical order. The copy processing unit 30 creates "Creating file C", "Creating file D", ... "Creating file G", and passes the created text data to the display control unit 24.

[0148] FIG. 29 is a diagram for explaining auto-fill copy (all range) when the text data already entered in the input field includes a fraction. In FIG. 29, the text data already entered in the input field 341 directly above the input field 342 in which the copy symbol 306 is handwritten is "Creating file_1 / 2", so the character 338B of the specified attribute is "1 / 2", which is a fraction. Therefore, since the character of the specified attribute is a fraction, the copy processing unit 30 determines that the increment is "1", which is the denominator of the fraction. The copy processing unit 30 creates "Creating file_1 / 3", "Creating file_1 / 4", ... "Creating file 1 / 7", and passes the created text data to the display control unit 24.

[0149] <Copy process when using AutoFill Copy (continuous)> Fig. 30 is a diagram explaining the copy process when the user handwrites copy symbol 307, which means autofill copy (continuous). In Fig. 30(a), the user handwrites "Creating file 1" and "Creating file 3" in the first input field 351 and the second input field 352, respectively. "Creating file 1" and "Creating file 3" in the input fields 351 and 352 are examples of a first object, and the input fields 351 and 352 are examples of a first input field and a second input field. This handwritten data is converted into text data by character recognition.

[0150] Assume that the user wishes to input text data that is the same as the first input field 351 and the second input field 352, but with characters 357 of a specified attribute changed by a fixed amount, into all of the lower input fields 353, 354. In this case, the user handwrites a copy symbol 307 as shown in Fig. 30(a) into the input field 353. The input field 353 is an example of a third input field. This copy symbol 307 is three dots lined up vertically and input into the input field 353 that is located directly below (below) the input fields 351, 352 in which text data has already been input.

[0151] The conversion unit 23 converts this copy symbol 307 into a copy symbol 355, which is text data. In addition, the conversion unit 23 may convert this copy symbol 307 into an operation command candidate 510, a handwritten recognition character string candidate 506, a conversion character string candidate 507, and a character string / prediction conversion candidate 508.

[0152] 31 shows an operation guide 500 that is displayed when the user inputs by handwriting a copy symbol 307, which means autofill copy (continuous). The operation guide 500 is not displayed if the input field information storage unit 42 is set to not display the operation guide 500.

[0153] In Fig. 31, two operation command candidates 510, one handwritten recognition character string candidate 506, one converted character string candidate 507, and one character string / predictive conversion candidate 508 are displayed. A method of displaying the operation command candidates 510 will be described. The copy processing unit 30 searches the command information storage unit 43 for a copy symbol 355 converted from a copy symbol 307 meaning autofill copy (continuous), and if the copy symbol 355 partially matches the String, the display control unit 24 displays the operation command in the operation guide 500 (when text data is input in the input field 352 directly above and the input field 351 further above). In Fig. 31, the copy symbol 355 partially matches the String of the two operation commands, so the two operation command candidates 510 are displayed.

[0154] When the user presses "■ Autofill copy (continuous)" with the pen 2500, the copy processing unit 30 obtains a copy method (adapting the attribute values ​​of the text data in the multiple input fields directly above and autofill copying to the multiple input fields below) from the copy information storage unit 44 using the copy symbol ID associated with the operation command as a key. The copy processing unit 30 obtains the text data already entered in the input fields 351 and 352 directly above from the object data storage unit 41. Next, the copy processing unit 30 detects characters with a specified attribute, such as numbers, from the text data. The copy processing unit 30 judges whether the first input field 351 and the second input field 352 are the same text data and whether the characters 357 with a specified attribute at the same character position have the same attribute. In the case of the same attribute, the copy processing unit 30 inputs text data in which the characters 357 of the specified attribute in the input fields 351 and 352 have been changed by the difference amount into all input fields 353, 354 below, thereby creating text data in which the characters 357 of the specified attribute have been increased by the difference amount. For example, if the text data in the two input fields 351, 352 directly above are "Creating file 1" and "Creating file 3", the difference amount of the characters 357 of the specified attribute is "2". The copy processing unit 30 creates "Creating file 5", "Creating file 7", ... "Creating file 13", and passes the created text data to the display control unit 24.

[0155] As shown in FIG. 30(b), the display control unit 24 erases the copy symbol 307, which means autofill copy (continuous), and displays the created text data in the input fields 353 and 354 in order from the top. The input fields 353 and 354, "Creating file 5" to "Creating file 13", are an example of a second object, and the input fields 353 and 354 are an example of a third input field. The input fields 353 and 354 to which the text data is copied are input fields that overlap the input field 353, in which the copy symbol 307 is handwritten, by a certain percentage or more in the width direction. Such input fields are specified by the position information of the input field information storage unit 42. Alternatively, the input fields 353 and 354 to which the text data is copied can be all input fields that are lined up in the same direction as the input fields 351 and 352 (the distance between the input fields is within a certain range).

[0156] In FIG. 30(a), the copy symbol 307 is three vertical dots, but the copy symbol may be as shown in FIG. 30(c). The copy symbol 308 in FIG. 30(c) overlaps two input fields 351, 352 (examples of the first input field and the second input field) in which text data has already been input. The copy symbol 308 starts from the input field 351, passes through the input field 352, and overlaps with one or more input fields 353, 354. Therefore, the copy processing unit 30 can obtain text data from the two input fields 351, 352 in which the copy symbol 308 overlaps, and determine the difference amount ("2") of characters with a predetermined attribute. The display control unit 24 copies the text data to the input fields 353, 354 (examples of the third input field). 30(d), the display control unit 24 erases the copy symbol 308 meaning autofill copy (continuous), and displays the created text data in order from the top in the input fields 353 and 354. "Creating file 5" to "Creating file 13" in the input fields 353 and 354 are an example of a second object, and the input fields 353 and 354 are an example of a third input field.

[0157] Therefore, the user can control the amount of difference in characters with a specified attribute in the created text data by handwriting the copy symbol 308 over an input field where text data has already been entered. Note that the operation guide 500 in the case of Fig. 30(c) may be the same as that in Fig. 19 (however, the operation command becomes Autofill Copy (continuous)).

[0158] 26. In addition, when the user handwrites copy symbol 307, which means autofill copy (continuous), but there is no text data in input field 352 directly above, the input example may be the same as that shown in Fig. 26. When there is text data in input field 352 directly above, but no text data in input field 351, the process is the same as that shown in Fig. 24.

[0159] Also, the copy symbol 307 in Fig. 30 and the copy symbol 306 in Fig. 24 have the same shape. This is merely for the sake of convenience, and the designer or developer may design the copy symbol 307 and the copy symbol 306 by changing their shapes. Alternatively, the user may set which copy function the copy symbol 307 and the copy symbol 306 will be used for.

[0160] FIG. 32 is a flow chart for explaining the copy process when a copy symbol 307, which means auto-fill copy (continuous), is handwritten.

[0161] The user hand-writes a copy symbol 307, which means auto-fill copy (continuous). The input accepting unit 21 accepts the hand-written input, the drawing data generating unit 22 generates drawing data, and the display control unit 24 displays the copy symbol 307, which means auto-fill copy (continuous). The conversion unit 23 converts the copy symbol 307 into a copy symbol 355, which is text data (S41).

[0162] The copy processing unit 30 stores the copy symbol 355 in the command information storage unit 43, and if there is a corresponding operation command, next acquires the copy method from the copy information storage unit 44. Since the copy method of the copy symbol 355, which means autofill copy (continuous), is "adapt the attribute values ​​of the text data of the upper multiple input fields and autofill copy to the lower multiple input fields," the copy processing unit 30 judges whether or not text data exists in the input field 352 directly above the input field 353 in which the copy symbol 307 is handwritten, and in the input field 351 further above (S42). The copy processing unit 30 judges, for example, whether or not text data exists in the input field closest to the pen coordinates immediately before the user lifted the pen, and in the input field directly above that input field.

[0163] If the determination in step S42 is Yes, the process proceeds to step S43, and if the determination in step S42 is No, the process proceeds to step S47.

[0164] In step S43, the display control unit 24 displays "■ Autofill copy (continuous)" and the copy symbol 355 converted from the copy symbol 307 in step S41 on the operation guide 500 (S43).

[0165] The user selects operation command candidate 510 "■ Autofill copy (all range)" with pen 2500. Copy processing unit 30 acquires text data in input field 352 located immediately above input field 353 in which copy symbol 307 is handwritten, and text data in input field 351 located even higher. Copy processing unit 30 determines whether or not a character with the same attribute exists in the same character position in the two pieces of text data (S44). Characters with the same attribute include numbers, letters, and fractions.

[0166] If the determination in step S44 is Yes, the process proceeds to step S45, and if the determination in step S44 is No, the process proceeds to step S49.

[0167] In step S45, the copy processing unit 30 calculates the amount of difference between each of the characters determined to have the same attribute in step S44 (S45). For example, if the respective characters are "1" and "3", the amount of difference is "2". If the respective characters are "A" and "C", the amount of difference is one character in alphabetical order. If the respective characters are "1 / 1" and "1 / 3", the amount of difference is "denominator 2".

[0168] Next, the copy processing unit 30 creates a plurality of text data in which characters of a predetermined attribute are changed by a difference amount, and passes the data to the display control unit 24. The number of text data created is the number of all input fields 354 below, including the input field 353 in which the copy symbol 307 is handwritten. The display control unit 24 displays the passed text data in order from the top of all input fields 354 below, including the input field 353 in which the copy symbol 307 is handwritten (S46).

[0169] There are two cases in step S47. If text data exists only in the input field 352 directly above the input field 353 in which the duplicated symbol 307 is handwritten, the process is the same as in Fig. 27. If text data does not exist in the input field 352 directly above the input field 353 in which the duplicated symbol 307 is handwritten in step S47, the process is to convert the duplicated symbol 307 into text data.

[0170] In step S48, the display control unit 24 displays the copy symbol 355 converted from the copy symbol 307 in step S41 on the operation guide 500, and does not display the operation command candidates 510 (S48). When the user selects the copy symbol 355 with the pen 2500 in step S43 or S48, the display control unit 24 displays the copy symbol 355 converted from the copy symbol 307 converted in step S41 in the input field 353 where the copy symbol 307 is handwritten.

[0171] In step S49, since the two text data do not have characters with the same attribute, the copy processing unit 30 creates text data identical to the input field 352 for each of the input fields 353, 354, and passes the data to the display control unit 24. The display control unit 24 displays the passed text data in order from the top to the bottom of all of the input fields 354 below, including the input field 353 in which the copy symbol 307 is handwritten (S49).

[0172] In this way, when the user wishes to change the attributes of already entered text data and enter it into all the input fields below, the user can reduce the effort of inputting by handwriting the copy symbol 307, which means auto-fill copy (continuous).

[0173] Another example of copying by AutoFill Copy will be described with reference to Figs. 33 and 34. Fig. 33 is a diagram for explaining AutoFill Copy (continuous) in the case where the text data already entered in the input field includes alphabets. In Fig. 33, the text data already entered in the input field 352 directly above the input field 353 in which the copy symbol 307 is handwritten and the input field 351 further above are "Creating file A" and "Creating file C". The characters 338A of the specified attribute are the alphabets "A" and "C". Therefore, the copy processing unit 30 judges the difference amount to be two characters in alphabetical order. The copy processing unit 30 creates "Creating file E", "Creating file G", ... "Creating file M", and passes the created text data to the display control unit 24.

[0174] FIG. 34 is a diagram for explaining autofill copy (continuous) when the text data already entered in the input field includes a fraction. In FIG. 34, the text data already entered in the input field 352 directly above the input field 353 in which the copy symbol 307 is handwritten and the input field 351 further above are "Creating file_1 / 1" and "Creating file_1 / 3". The characters 338B of the specified attribute are the fractions "1 / 1" and "1 / 3". Therefore, the copy processing unit 30 determines that the difference is the denominator "2" of the fraction. The copy processing unit 30 creates "Creating file_1 / 5", "Creating file_1 / 7", ... "Creating file 1 / 13", and passes the created text data to the display control unit 24.

[0175] <Major Effects> The display device 2 of this embodiment can copy at least a part of already-entered text data into an input field by handwriting inputting the copy symbols 303 to 308. This can reduce the effort required for handwriting input of the same content as the same text data that has already been entered.

[0176] [Second embodiment] In this embodiment, a display device 2 that collects a user's operation history on the display device 2 and proposes to change the setting conditions based on the analysis results will be described. For example, the display device 2 analyzes the user's operation history information, and finds that the operation command candidates 510 displayed according to the copy symbol have been selected a number of times equal to or greater than a threshold value. In this case, the display device 2 suggests to the user whether or not to display the operation command candidates 510 even if the user does not handwrite the copy symbol.

[0177] Fig. 35 is an example of a functional block diagram for explaining the functions of the display device 2 of this embodiment by dividing them into blocks. The explanation of Fig. 35 will mainly focus on the differences from Fig. 4. The display device 2 newly includes an operation history recording unit 31 and an analysis unit 32. The storage unit 40 newly includes an operation history information storage unit 45.

[0178] The operation history recording unit 31 records, for each user, the number of times that an operation command selected by the user in the past has been selected in association with the operation command. The operation history recording unit 31 stores the operation history information in which the operation command is associated with the number of times that the operation command has been selected in the operation history information storage unit 45. The operation history information will be described with reference to FIG.

[0179] The analysis unit 32 analyzes the operation history information and determines whether or not the operation command candidates have been selected a number of times equal to or greater than a threshold value. If so, the display control unit 24 suggests to the user whether or not to display the operation command related to copying in the operation guide 500.

[0180] <Processing or Action> Fig. 36 is a flow chart for explaining the process in which the display device 2 accepts setting changes based on operation history information. The process in Fig. 36 is executed every time the user presses an operation command. Fig. 37 is an example of a proposal screen displayed by the display device 2.

[0181] First, the analysis unit 32 analyzes the operation history information (S51).

[0182] The analysis unit 32 judges whether the operation command displayed based on the copy symbols 303 to 308 has been selected a number of times equal to or greater than a threshold (for example, five times) (S52). The operation history information 375 is displayed on the proposal screen 370 of FIG. 37. According to the operation history information 375, for example, it is recorded that the operation command corresponding to the handwritten copy symbol 303 meaning "same as above" has been executed five times, and the operation command corresponding to the handwritten copy symbol 305 meaning "autofill copy (range specification)" has been executed once. The analysis unit 32 compares the selection number of times that these operation commands have been executed with a threshold.

[0183] If the determination in step S52 is No, the process returns to step S51, and if the determination in step S52 is Yes, the process proceeds to step S53.

[0184] In step S53, the analysis unit 32 determines that a setting change should be proposed to the user. For this reason, the display control unit 24 causes the display 220 to display a proposal screen 370 in Fig. 37. The proposal screen 370 displays a message 371 stating "There is a tendency for the operation command to be selected frequently to duplicate the characters displayed on the screen. Would you like to display an operation command to duplicate the characters displayed on the screen in the list of operation command candidates?" The user can see the message and press a Yes button 372 or a No button 373.

[0185] The operation acceptance unit determines whether or not the Yes button 372 has been accepted (S54). If the decision in step S54 is No, the process returns to step S51, and if the decision in step S54 is Yes, the process proceeds to step S55.

[0186] In step S55, the operation reception unit 27 receives a setting change to display the copy symbol 303 in the operation command candidates 510 (S55). As a result, the display control unit 24 displays, in the operation guide 500, an operation command corresponding to the copy symbol 303 whose selection count is equal to or greater than the threshold in the operation history information, even if the copy symbol 303 is not handwritten after the setting change.

[0187] <Display control of operation commands using past operation history> A display example of operation guide 500 when the user presses Yes button 372 in Fig. 37 and accepts a setting change will be described. In the display device 2, before the user handwrites copy symbol 303, the display control unit 24 can display operation guide 500 including operation command candidates 510 corresponding to copy symbol 303. The description will be made based on Fig. 38.

[0188] FIG. 38 shows input fields 381 and 382 and an operation guide 500 displayed. (1) As shown in Fig. 38(a), the user hand-writes, for example, "Preparing a notification form" in an input field 381. This hand-written data may be converted to text data or may remain as hand-written data. (2) After accepting the setting change, the display control unit 24 displays the operation commands selected a number of times equal to or greater than the threshold value in the operation command candidates 510. In FIG. 38(b), "■ Copy" is displayed in the operation guide 500. If there are other operation commands selected to be displayed on the proposal screen 370, the one selected more times in the past is displayed at the top. The timing for displaying the operation guide 500 may be when the pen 2500 is lifted from the input field 381, or after the converted text data is displayed in the input field 381. In the former case, the operation command candidates 510 can be displayed together with the character string candidates 539. In the latter case, the operation command candidates 510 are displayed independently. (3) If operation command candidates 510 are displayed on the operation guide 500 before the copy symbol is handwritten, it is assumed that the copy symbol is handwritten in the input field 382 directly below the current input field 381. In FIG. 38(c), the user selects “■Copy” with the pen 2500, so that the text data in the input field 381 is copied to the input field 382.

[0189] In this way, when the display control unit 24 inputs text data into the input field, an operation guide 500 including the operation commands most frequently selected in the past is automatically displayed, so that the user can select the operation command corresponding to the desired copy process without having to input the copy symbol by hand.

[0190] <Changing the display order of operation command candidates using the operation command operation history> Using the operation history of the user's operation commands, the display control unit 24 can change the display order of operation command candidates. This will be described with reference to FIG.

[0191] Fig. 39 shows operation history information 400 of operation commands and an operation guide 500 displayed by handwriting input of a copy symbol 303. The number of times that an operation command selected in the past has been selected for each user is recorded in the operation history information 400. The operation history information 400 in Fig. 39(c) shows the number of times that an operation command selected with a copy symbol has been selected among the operation commands.

[0192] (1) As shown in Fig. 39(a), the user hand-writes a copy symbol 303, which means "same as above," in an input field 392. Here, a case will be described in which an operation guide 500 is displayed as in step S3 of Fig. 13.

[0193] (2) When displaying the operation guide 500, the display control unit 24 refers to the operation history information 400 and judges whether the number of times the operation command corresponding to the copy symbol 303 has been selected is equal to or greater than the threshold. Since the number of times the operation command has been selected is "15", the display control unit 24 judges that the number of times is equal to or greater than the threshold. Therefore, as shown in FIG. 39(d), the display control unit 24 displays the operation command candidate 510 higher than the character string candidate 539. In FIG. 39(d), "■Copy" is displayed higher than the character string candidate 539 such as "〃". FIG. 39(e) shows the operation guide 500 shown for reference, and shows the display order of the operation command candidate 510 and the character string candidate 539 when the number of times the operation command corresponding to the copy symbol 303 has been selected is not equal to or greater than the threshold.

[0194] (3) Next, in FIG. 39(d), the user selects "■ Copy" with the pen 2500, so that the text data in the input field 391 directly above is copied into the input field 392 below, as shown in FIG. 39(b).

[0195] In this way, when there is an operation command that has been selected a number of times equal to or greater than the threshold value in the past, the display control unit 24 displays it at the top of the operation guide 500, so that the user can easily select an operation command corresponding to the copy symbol.

[0196] In FIG. 39, a case has been described in which a copy symbol meaning the same has been handwritten, but the same applies to operation commands corresponding to other copy symbols.

[0197] <Major Effects> In this embodiment, by creating operation history information of the user on the display device 2, it is possible to display operation commands before the user writes the copy symbol by hand. Also, it is possible to give priority to displaying operation commands that the user frequently uses in the operation guide 500.

[0198] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0199] For example, in this embodiment, an embodiment in which text data in an input field is copied to another input field has been described, but the text data does not have to be in an input field. As shown in FIG. 40, the text data may be handwritten in any location. FIG. 40 is a diagram for explaining copying when there is no input frame. FIG. 40(a) shows an example of copying using copy symbol 303, FIG. 40(b) shows an example of copying using copy symbol 304, FIG. 40(c) shows an example of copying using copy symbol 305, and FIG. 40(d) shows an example of copying using copy symbol 306.

[0200] In the case of copy symbol 303, the copy processing unit 30 takes text data within a certain distance directly above copy symbol 303 as the target of copying, and copies the same text data directly below (below) the text data with the same attributes as the text data. In the case of copy symbol 304, the copy processing unit 30 takes text data within a certain distance from the arrowhead of copy symbol 304 as the target of copying, and copies the text data near the arrowhead with the same attributes as the text data. In the case of copy symbol 305, the copy processing unit 30 takes text data overlapping with the arrowhead of copy symbol 305 as the target of copying, and copies as much text data as is within the range covered by copy symbol 305 with the same attributes as the text data. In the case of copy symbol 306, the copy processing unit 30 takes text data within a certain distance directly above copy symbol 306 as the target of copying, and copies the text data up to the maximum range of display 220. The duplicate symbol 307 may be the same as the duplicate symbols 305 and 306, and further the characters of a predetermined attribute are changed.

[0201] Furthermore, the shapes of the copy symbols 303 to 308 in this embodiment are merely examples, and the shape of the copy symbol to which the copy function is associated may be changed as appropriate.

[0202] In addition, in this embodiment, the copying of text data in input fields arranged vertically has been described, but copying of text data in input fields arranged horizontally can also be achieved in a similar manner. For example, when text data is input in a table or tabular format, the user can copy in any direction. The user may be able to set which direction to copy in.

[0203] In the present embodiment, the copying of an object in a state in which the object is converted into text data has been described, but if the object to be copied is handwritten data, the handwritten data may be copied as is. In addition, graphics or image data may be copied.

[0204] Furthermore, in this embodiment, stroke data is mainly converted into Japanese, but the language into which the stroke data is converted may be other languages ​​(English, Chinese, Hindi, Spanish, French, Arabic, Russian, etc.).

[0205] In addition, although the electronic whiteboard has been described as an example in the present embodiment, the electronic whiteboard may be called an electronic whiteboard, an electronic information board, or the like. In addition, the present embodiment can be suitably applied to any information processing device having a touch panel. The information processing device equipped with a touch panel may be, for example, an output device such as a projector or digital signage, a HUD (Head Up Display) device, an industrial machine, an imaging device, a sound collecting device, a medical device, a network home appliance, a notebook PC, a mobile phone, a smartphone, a tablet terminal, a game machine, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, or a desktop PC.

[0206] In this embodiment, the coordinates of the pen are detected by detecting the coordinates of the pen tip with a touch panel, but the display device 2 may detect the coordinates of the pen tip with ultrasonic waves. The pen also emits ultrasonic waves together with light, and the display device 2 calculates the distance based on the arrival time of the ultrasonic waves. The display device 2 can identify the position of the pen based on the direction and distance, and the projector draws (projects) the trajectory of the pen as stroke data.

[0207] Moreover, the configuration example in FIG. 4 and the like is divided according to main functions in order to facilitate understanding of the processing by the display device 2. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing by the display device 2 can be divided into more processing units according to the processing contents. Also, it can be divided so that one processing unit includes more processes.

[0208] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0209] Embodiments of the present invention provide significant improvements in computer power and functionality. These improvements allow users to utilize computers that provide more efficient and robust interaction with tables, which are ways of storing and presenting information in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of more efficient, powerful and robust user interfaces. Such user interfaces provide better interaction between humans and machines.

[0210] <Additional Notes> [Appendix 1] 1. A display device that displays a first object based on handwriting input, comprising: an input receiving unit that receives a handwritten input of a symbol that copies at least a part of the first object; a display control unit that displays a second object obtained by duplicating at least a portion of the first object based on a symbol that duplicates at least a portion of the first object; A display device comprising: [Appendix 2] The display device described in Appendix 1, wherein when a symbol that copies at least a portion of the first object has a predetermined relationship with the first object, the display control unit displays a second object in which at least a portion of the first object is copied. [Appendix 3] The display device according to claim 1 or 2, wherein when the first object is displayed in a first input field and a symbol copying at least a part of the first object is handwritten input into a second input field directly below the first input field, the display control unit displays the second object in the second input field. [Appendix 4] When the first object is displayed in a first input field, an arrowhead of a symbol shaped like an arrow for duplicating at least a part of the first object is at least around the first input field, and an arrowhead of a symbol shaped like an arrow for duplicating at least a part of the first object is at least around the second input field, The display device according to claim 1, wherein the display control unit causes the second object to be displayed in the second input field. [Appendix 5] when the input accepting unit finishes accepting the handwritten input of the symbol for copying at least a part of the first object, the display control unit displays candidates of text into which the symbol for copying at least a part of the first object is converted and candidates of operation commands corresponding to the symbol for copying at least a part of the first object; The display device according to claim 3 or 4, wherein, when a selection of the operation command is accepted, the display control unit causes the second object to be displayed in the second input field. [Appendix 6] When the first object is displayed in a first input field, and a symbol duplicating at least a part of the first object overlaps the first input field and one or more second input fields, 3. The display device according to claim 1, wherein the display control unit causes the second object to be displayed in the one or more second input fields. [Appendix 7] a copy processing unit that detects characters having a predetermined attribute from the first object, and creates the second object by changing the characters by an increment that is preset according to the characters having the predetermined attribute, the number of the second object being equal to the number of the one or more second input fields; The display device according to claim 6, wherein the display control unit causes one or more of the second objects created by the copy processing unit to be displayed in one or more second input fields, respectively. [Appendix 8] when the input accepting unit finishes accepting the handwritten input of the symbol for copying at least a part of the first object, the display control unit displays candidates of text into which the symbol for copying at least a part of the first object is converted and candidates of operation commands corresponding to the symbol for copying at least a part of the first object; The display device according to claim 6, wherein, when a selection of the operation command is accepted, the second object is displayed in the one or more second input fields. [Appendix 9] When the first object is displayed in a first input field, and a symbol that replicates at least a part of the first object is handwritten in a second input field immediately below the first input field, The display device according to claim 1 or 2, wherein the display control unit causes the second object to be displayed in all input fields arranged in the same direction as the first input field and the second input field. [Appendix 10] a copy processing unit that detects characters having a predetermined attribute from the first object, and creates the second object by changing the characters by an increment that is preset according to the characters, the number of the second object being the number of all input fields in a direction in which the first input field and the second input field are arranged, excluding the first input field; The display device described in Appendix 9, wherein the display control unit causes the second object created by the copy processing unit to be displayed in all input fields in the direction in which the first input field and the second input field are arranged, excluding the first input field. [Appendix 11] when the input accepting unit finishes accepting the handwritten input of the symbol for copying at least a part of the first object, the display control unit displays candidates of text into which the symbol for copying at least a part of the first object is converted and candidates of operation commands corresponding to the symbol for copying at least a part of the first object; The display device according to claim 9, wherein, when a selection of the operation command is accepted, the second object is displayed in all input fields in the direction in which the first input field and the second input field are arranged, excluding the first input field. [Appendix 12] When a first object is displayed in a first input field and a second input field located immediately below the first input field, and a symbol duplicating at least a part of the first object is overlapped from the first input field through the second input field to one or more third input fields, a copy processing unit that detects characters of a predetermined attribute from two first objects in each of the first input field and the second input field, determines a difference between the characters of the predetermined attribute in each of the first input field and the second input field, and creates the second objects by changing the characters in the second input field by the difference amount, the number of which is equal to the number of the one or more third input fields; 3. The display device according to claim 1, wherein the display control unit causes the second object created by the copy processing unit to be displayed in each of the one or more third input fields. [Appendix 13] when the input accepting unit finishes accepting the handwritten input of the symbol for copying at least a part of the first object, the display control unit displays candidates of text into which the symbol for copying at least a part of the first object is converted and candidates of operation commands corresponding to the symbol for copying at least a part of the first object; The display device according to claim 12, wherein, when a selection of the operation command is accepted, the second object is displayed in each of the one or more third input fields. [Appendix 14] When a first object is displayed in a first input field and a second input field immediately below the first input field, and a symbol that duplicates at least a part of the first object is handwritten input in a third input field immediately below the second input field, a copy processing unit that detects characters of a predetermined attribute from two first objects in each of the first input field and the second input field, determines a difference between the characters of the predetermined attribute in each of the first input field and the second input field, and creates the second objects by changing the characters in the second input field by the difference amount, the number of which is the same as the number of all input fields in a direction in which the first input field and the second input field are arranged, excluding the second input field; The display device according to claim 1 or 2, wherein the display control unit causes the second object created by the copy processing unit to be displayed in all input fields in the direction in which the first input field and the second input field are arranged, excluding the second input field. [Appendix 15] when the input accepting unit finishes accepting the handwritten input of the symbol for copying at least a part of the first object, the display control unit displays candidates of text into which the symbol for copying at least a part of the first object is converted and candidates of operation commands corresponding to the symbol for copying at least a part of the first object; The display device according to claim 14, wherein, when a selection of the operation command is accepted, the second object is displayed in all input fields in the direction in which the first input field and the second input field are arranged, excluding the second input field. [Appendix 16] an analysis unit that analyzes operation history information of an operation command for copying an object by a user; an operation reception unit that displays an analysis result by the analysis unit and receives a setting change related to the display of the operation command; 16. The display device according to any one of claims 1 to 15, comprising: [Appendix 17] When the operation reception unit receives the setting change, The display device described in Appendix 16, wherein the display control unit displays an operation command that has been selected a number of times in the operation history information that is greater than or equal to a threshold, even if a symbol that copies at least a portion of the first object is not handwritten. [Appendix 18] The display device according to claim 16, wherein the display control unit prioritizes displaying an operation command that has been selected more frequently in the operation history information. [Explanation of symbols]

[0211] 2 Display device [Prior art documents] [Patent documents]

[0212] [Patent Document 1] Patent No. 3933449

Claims

1. 1. A display device that displays a first object based on handwriting input, comprising: an input receiving unit that receives a handwritten input of a symbol that copies at least a part of the first object; a display control unit that displays a second object obtained by duplicating at least a portion of the first object based on a symbol that duplicates at least a portion of the first object; A display device comprising:

2. The display device according to claim 1 , wherein when a symbol that copies at least a portion of the first object has a predetermined relationship with the first object, the display control unit displays a second object in which at least a portion of the first object is copied.

3. 3. The display device according to claim 1, wherein when the first object is displayed in a first input field and a symbol copying at least a portion of the first object is handwritten input into a second input field directly below the first input field, the display control unit causes the second object to be displayed in the second input field.

4. When the first object is displayed in a first input field, an arrowhead of a symbol shaped like an arrow for duplicating at least a part of the first object is at least around the first input field, and an arrowhead of a symbol shaped like an arrow for duplicating at least a part of the first object is at least around the second input field, The display device according to claim 1 , wherein the display control unit causes the second object to be displayed in the second input field.

5. when the input accepting unit finishes accepting the handwritten input of the symbol for copying at least a part of the first object, the display control unit displays candidates of text into which the symbol for copying at least a part of the first object is converted and candidates of operation commands corresponding to the symbol for copying at least a part of the first object; The display device according to claim 3 , wherein, when a selection of the operation command is accepted, the display control unit causes the second object to be displayed in the second input field.

6. When the first object is displayed in a first input field, and a symbol duplicating at least a part of the first object overlaps the first input field and one or more second input fields, The display device according to claim 1 , wherein the display control unit causes the second object to be displayed in the one or more second input fields.

7. a copy processing unit that detects characters having a predetermined attribute from the first object, and creates the second object by changing the characters by an increment that is preset in accordance with the characters having the predetermined attribute, the copy processing unit being equal to the number of the one or more second input fields; The display device according to claim 6 , wherein the display control unit causes the one or more second objects created by the copy processing unit to be displayed in the one or more second input fields, respectively.

8. when the input accepting unit finishes accepting the handwritten input of the symbol for copying at least a part of the first object, the display control unit displays candidates of text into which the symbol for copying at least a part of the first object is converted and candidates of operation commands corresponding to the symbol for copying at least a part of the first object; The display device according to claim 6 , wherein when a selection of the operation command is accepted, the second object is displayed in the one or more second input fields.

9. When the first object is displayed in a first input field, and a symbol that replicates at least a part of the first object is handwritten in a second input field located immediately below the first input field, The display device according to claim 1 , wherein the display control unit causes the second object to be displayed in all input fields arranged in the same direction as a direction in which the first input field and the second input field are arranged.

10. a copy processing unit that detects characters having a predetermined attribute from the first object, and creates the second object by changing the characters by an increment that is preset according to the characters, the number of the second object being the number of all input fields in a direction in which the first input field and the second input field are arranged, excluding the first input field; The display device according to claim 9, wherein the display control unit causes the second object created by the copy processing unit to be displayed in all input fields in the direction in which the first input field and the second input field are arranged, excluding the first input field.

11. when the input accepting unit finishes accepting the handwritten input of the symbol for copying at least a part of the first object, the display control unit displays candidates of text into which the symbol for copying at least a part of the first object is converted and candidates of operation commands corresponding to the symbol for copying at least a part of the first object; The display device according to claim 9, wherein when a selection of the operation command is accepted, the second object is displayed in all input fields in a direction in which the first input field and the second input field are arranged, excluding the first input field.

12. When a first object is displayed in a first input field and a second input field located immediately below the first input field, and a symbol duplicating at least a part of the first object is overlapped from the first input field through the second input field to one or more third input fields, a copy processing unit that detects characters of a predetermined attribute from two first objects in each of the first input field and the second input field, determines a difference between the characters of the predetermined attribute in each of the first input field and the second input field, and creates the second objects by changing the characters in the second input field by the difference amount, the number of which is equal to the number of the one or more third input fields; The display device according to claim 1 , wherein the display control unit causes the second object created by the copy processing unit to be displayed in each of the one or more third input fields.

13. when the input accepting unit finishes accepting the handwritten input of the symbol for copying at least a part of the first object, the display control unit displays candidates of text into which the symbol for copying at least a part of the first object is converted and candidates of operation commands corresponding to the symbol for copying at least a part of the first object; The display device according to claim 12 , wherein, when a selection of the operation command is accepted, the second object is displayed in each of the one or more third input fields.

14. When a first object is displayed in a first input field and a second input field immediately below the first input field, and a symbol that duplicates at least a part of the first object is handwritten input in a third input field immediately below the second input field, a copy processing unit that detects characters of a predetermined attribute from two first objects in each of the first input field and the second input field, determines a difference between the characters of the predetermined attribute in each of the first input field and the second input field, and creates second objects in which the characters in the second input field are changed by the difference, the number of which corresponds to the number of all input fields in a direction in which the first input field and the second input field are arranged but not including the second input field; 3. The display device according to claim 1, wherein the display control unit causes the second object created by the copy processing unit to be displayed in all input fields in the direction in which the first input field and the second input field are arranged, excluding the second input field.

15. when the input accepting unit finishes accepting the handwritten input of the symbol for copying at least a part of the first object, the display control unit displays candidates of text into which the symbol for copying at least a part of the first object is converted and candidates of operation commands corresponding to the symbol for copying at least a part of the first object; The display device according to claim 14, wherein when a selection of the operation command is accepted, the second object is displayed in all input fields in a direction in which the first input field and the second input field are arranged, excluding the second input field.

16. an analysis unit that analyzes operation history information of an operation command for copying an object by a user; an operation reception unit that displays an analysis result by the analysis unit and receives a setting change related to the display of the operation command; The display device according to claim 1 ,

17. When the operation reception unit receives the setting change, The display device according to claim 16 , wherein the display control unit causes an operation command that has been selected a number of times in the operation history information that is equal to or greater than a threshold value to be displayed even if a symbol that copies at least a part of the first object is not handwritten.

18. The display device according to claim 16 , wherein the display control unit displays, with priority, an operation command that has been selected more frequently in the operation history information.

19. 1. A display method performed by a display device that displays a first object based on handwriting input, comprising: accepting handwritten input of a symbol that replicates at least a portion of the first object; A process of displaying a second object in which at least a part of the first object is copied based on a symbol for copying at least a part of the first object; A display method comprising:

20. a display device for displaying a first object based on handwriting input; an input receiving unit that receives a handwritten input of a symbol that copies at least a part of the first object; a display control unit that displays a second object obtained by duplicating at least a portion of the first object based on a symbol that duplicates at least a portion of the first object; A program to function as a

Images