Display device, display method, and program

Abstract

To provide a display device capable of selecting a line or a column with handwritten data.SOLUTION: A display device 2 for displaying a table created by handwriting includes: reception means for receiving an input of handwritten data by input means; and selection receiving means for receiving selection of a line or a column of a table with the hand written data. Thus, the display device improves operability since editing such as an addition, movement, deletion or a division of a line or a column can be performed with hand written data.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a display device, a display method, and a program. [Background technology]

[0002] There is a known display device that uses handwriting recognition technology to convert handwritten data into characters and display them on a display. Display devices equipped with relatively large touch panels are placed in conference rooms and public facilities and are used by multiple users as electronic whiteboards.

[0003] The display device can accept table input based on handwritten data (see, for example, Patent Document 1). Patent Document 1 discloses a method for deleting rows or columns of a table by operating an operation menu. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has a problem in that it is not possible to select rows or columns using handwritten data. For example, Patent Document 1 discloses a method for a user to create a table using handwritten data, but does not disclose a method for selecting rows or columns using handwritten data.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide a display device that allows selection of rows and columns using handwritten data. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides a display device for displaying a table created by handwriting, the display device including: a receiving unit for receiving an input of handwritten data by an input unit; and a selection receiving unit for receiving a selection of a row or column of the table by the handwritten data; and a control means for, when a row or column of the table is selected by the selection receiving means, editing the table with respect to the selected row or column of the table in response to a user operation, wherein the control means displays a button for adding a row or column near the row or column selected by the selection receiving means, and, when the receiving means receives a press of the button, adds a row or column before or after the row or column selected by the selection receiving means. . Effect of the Invention

[0007] The present invention provides a display device that allows for row and column selection with handwritten data. [Brief description of the drawings]

[0008] [Figure 1] 11A and 11B are diagrams illustrating an example of an operation in which the display device accepts input of a table. [Diagram 2] FIG. 13 is a diagram illustrating a row selection method. [Diagram 3] FIG. 1 is a schematic perspective view of a display device. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device. [Diagram 5] FIG. 2 is an example of a functional block diagram for explaining functions of the display device by dividing them into blocks. [Figure 6] 1 is a diagram illustrating an example of drawing object information held by a drawing object management unit; [Figure 7] 10 is a diagram illustrating an example of line object information; [Figure 8] 11 is a diagram illustrating an example of a method for selecting a row in a table. [Figure 9] FIG. 13 is a flowchart illustrating an example of a procedure for accepting a selection of an arbitrary row of a table object. [Figure 10] FIG. 13 is an example diagram illustrating how a user can select a row with an arrow. [Figure 11] FIG. 13 is a flowchart illustrating an example of a procedure for accepting a selection of an arbitrary row of a table object. [Figure 12] FIG. 13 is a diagram showing an example of a table on which buttons are displayed. [Figure 13] 13A and 13B are diagrams illustrating an example of table object information before and after a row is added. [Figure 14] FIG. 11 is a flowchart illustrating an example of a process in which the display device accepts addition of a row. [Figure 15] 1 is a diagram illustrating an example of a method for adding a row to a table using handwritten data; [Figure 16] FIG. 11 is a flowchart illustrating an example of a process in which the display device accepts addition of a row. [Figure 17] 13A and 13B are diagrams illustrating an example of drawing object information before and after a line is added. [Figure 18] 11 is a diagram illustrating an example of a method for adding a line using handwritten data of an insertion symbol; [Figure 19] FIG. 11 is a flowchart illustrating an example of a process in which the display device accepts addition of a row. [Figure 20] 13A and 13B are diagrams illustrating an example of drawing object information before and after a line is added. [Figure 21] 11A and 11B are diagrams illustrating an example of a method for deleting a row; [Figure 22] FIG. 11 is an example of a flowchart illustrating a process in which the display device accepts deletion of a row. [Diagram 23] 13A and 13B are diagrams illustrating an example of table object information before and after deletion; [Figure 24] 11 is a diagram illustrating an example of a method for deleting a line in handwritten data. [Diagram 25] FIG. 11 is an example of a flowchart illustrating a process in which the display device accepts deletion of a row. [Figure 26] 13A and 13B are diagrams illustrating an example of table object information before and after deletion. [Figure 27] 11 is a diagram illustrating an example of a method for deleting a line in handwritten data. [Figure 28] FIG. 11 is an example of a flowchart illustrating a process in which the display device accepts deletion of a row. [Figure 29] 13A and 13B are diagrams illustrating an example of drawing object information before and after deletion. [Diagram 30] 11 is a diagram illustrating an example of a method for copying or moving a row. [Diagram 31] FIG. 11 is a flowchart illustrating an example of a procedure for the display device to accept row movement. [Diagram 32] 13A and 13B are diagrams illustrating an example of drawing object information before and after movement. [Diagram 33] 11 is a diagram illustrating an example of a method for moving a line in handwritten data. [Diagram 34]FIG. 11 is an example of a flowchart illustrating a process in which the display device receives a row movement. [Diagram 35] 13A and 13B are diagrams illustrating an example of drawing object information before and after movement. [Diagram 36] 11 is a diagram illustrating an example of a method for moving a line in handwritten data. [Figure 37] FIG. 11 is an example of a flowchart illustrating a process in which the display device receives a row movement. [Figure 38] 13A and 13B are diagrams illustrating an example of drawing object information before and after a line is moved. [Figure 39] 11 is a diagram illustrating an example of a method for dividing a table. [Diagram 40] FIG. 11 is a flowchart showing a procedure in which the display device accepts a request to divide a table and divides the table. [Diagram 41] 11A and 11B are diagrams illustrating an example of drawing object information before and after a table is divided; [Diagram 42] FIG. 13 is a diagram showing another configuration example of the display device. [Diagram 43] FIG. 13 is a diagram showing another configuration example of the display device. [Diagram 44] FIG. 13 is a diagram showing another configuration example of the display device. [Diagram 45] FIG. 13 is a diagram showing another configuration example of the display device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a display device 2 as an example of an embodiment of the present invention and a display method performed by the display device 2 will be described with reference to the drawings. EXAMPLES

[0010] <Example of table editing method> Before describing the table editing method of this embodiment, a comparative example of the table editing method will be described. Note that the comparative example is not limited to the prior art.

[0011] When a user edits a table using an electronic pen, a method of operating some kind of menu is considered. In this case, for example, the following operations are required. To insert a row, the user selects the row to insert and long presses it to display the edit menu. From the edit menu, the user selects "Insert". To delete a row, the user selects the row to be deleted and long presses it to display the edit menu. The user selects "Delete" from the edit menu. To move a row, the user selects the row to be moved and long-presses it to display the edit menu. From the edit menu, the user selects "Cut". Next, the user selects the destination row and long-presses it to display the edit menu. From the edit menu, the user selects "Insert Cut Row".

[0012] In this way, since the user must repeatedly select from the menu for editing, the menu operations become cumbersome, and there is a risk that operability will decrease.

[0013] <How to create a table> Next, a method for selecting a table in the display device 2 of this embodiment will be described. First, a procedure for creating a table will be described with reference to Fig. 1. Fig. 1 shows an example of an operation in which the display device 2 accepts input of a table. (1) The user presses icon 301 for inputting a table (Figure 1(a)). (2) The user handwrites the table 302 freehand, that is, the user creates a table using handwritten data. (3) The display device 2 recognizes the handwritten data and converts it into a table 303 (FIG. 1(b)). That is, the display device 2 displays a table 303 formed of multiple parallel straight lines vertically and horizontally.

[0014] The display device 2 of this embodiment allows the user to edit the table with handwritten data for such a table 303. The flow of the editing operation is as follows. (1) The user selects part of a table using handwritten data. (2) When a user selects any row (column) in a table, the user can add a row (column) above (to the left) or below (to the right) of that row (column) with a simple operation. (3) Similarly, by selecting any row (column) in a table, the user can delete that row (column) with a simple operation. (4) Similarly, by selecting any Xth row (column) of a table, the user can easily move to any Yth row (column). (5) Similarly, by selecting any X row (column) in the table, the user can easily extract and display that X row (column) anywhere outside the table.

[0015] Fig. 2 is a diagram explaining a method for selecting a line. As an example, as shown in Fig. 2(a), the user can select any line by enclosing it with a frame 304 of handwritten data. The display device 2 judges whether or not there is a line object in the area inside the enclosed frame 304. If the object in the area is a line, the display device 2 selects the Osaka line as shown in Fig. 2(b).

[0016] The selected state is a state in which the display device 2 highlights the selected row to make the user aware of it. The display device 2 also enters a state in which it waits for an operation on the selected row. As will be described later, the user can edit the table by inputting handwritten data using an input means such as an electronic pen, without going through a menu or with a minimum of menu operations. For example, the user can add a row before or after the selected row, delete the selected row, or move it. The same applies to columns. It is not essential that the user selects a row or column.

[0017] In this way, the display device 2 of this embodiment can perform editing such as adding, moving, or deleting rows or columns on handwritten data for a selected row or column in response to user operations, thereby improving operability.

[0018] <Terminology> The input means may be any means capable of specifying coordinates on a touch panel and writing by hand, such as a pen, a human finger or hand, or a rod-shaped member.

[0019] A stroke refers to a series of operations in which a user presses an input means against a display, moves it continuously, and then releases it from the display. Stroke data is information displayed on the display based on a trajectory of coordinates input by the input means. Stroke data may be appropriately interpolated. Handwritten data is data having one or more stroke data. Handwritten input refers to handwritten data being input by a user.

[0020] An object is an object that is displayed on a display based on stroke data. Object means a target, but in this embodiment it means a display target. The character string converted from handwritten data through character recognition may include not only text data, but also data displayed based on user operations, such as fixed characters such as "Done," stamps displayed as marks, shapes such as circles and stars, and straight lines.

[0021] A character string is one or more character codes (fonts) that are converted from handwritten data by character recognition. A character string can be one or more characters that are handled by a computer. Characters include numbers, alphabets, and symbols. A character string is also called text data.

[0022] A display component is a display of colors, shapes, etc. that form the screen. It is not something that can be disassembled from the display device 2 as a component, but rather something that a human being can visually distinguish and recognize from others. For example, a menu for operations is a display component.

[0023] A table is a representation of information in an array. A table is also called a table. A table has cells arranged horizontally and vertically. Cells arranged horizontally are called "rows," and cells arranged vertically are called "columns."

[0024] Editing refers to modifying, adding, deleting, etc. a work. In this embodiment, it mainly refers to operations related to tables. For example, it includes adding, deleting, moving, copying, dividing a table, etc.

[0025] <Configuration example> 3 shows an example of a schematic perspective view of the display device 2. A user (user) uses the display device 2. The display device 2 of this embodiment allows the user to handwrite on the display using an input means such as a hand or an electronic pen 2500. Although the display device 2 in Fig. 3 is placed horizontally, it can also be placed vertically. The user can switch between horizontal and vertical orientations by rotating the display device 2 around the center of the display.

[0026] <Hardware configuration example> 4 is a diagram showing an example of a hardware configuration of the display device 2. The display device 2 of this embodiment includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, a network controller 205, and an external device connection I / F 206 (Interface), and is a shared terminal for sharing information among a plurality of users.

[0027] Of these, the CPU 201 controls the overall operation of the display device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201.

[0028] The SSD 204 stores various data such as programs for the display device 2. This program may be an application program that runs on an information processing device equipped with a general-purpose OS (Windows (registered trademark), Mac OS (registered trademark), Android (registered trademark), iOS (registered trademark), etc.). In this case, the display device 2 is usually used as a general-purpose information processing terminal, but when a user installs and executes an application program, the user can write by hand, etc., just like a dedicated device as a display device.

[0029] A network controller 205 controls communication with a network. An external device connection I / F 206 controls communication with a USB (Universal Serial Bus) memory 2600 and external devices (a camera 2900, a speaker 2800, and a microphone 2700).

[0030] The display device 2 also includes a capture device 211, a GPU 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication unit 219, and an antenna 219a of the short-range communication unit 219.

[0031] Of these, the capture device 211 displays video information as still or moving images on the display of the PC 10. The GPU (Graphics Processing Unit) 212 is a semiconductor chip that specializes in graphics. The display controller 213 controls and manages the screen display in order to output the output image from the GPU 212 to the display 220, etc.

[0032] The contact sensor 214 detects that the electronic pen 2500, the user's hand H, or the like has come into contact with the display 220. When the electronic pen 2500 and the hand H are not distinguished from each other, they are referred to as an input means 291.

[0033] The sensor controller 215 controls the processing of the contact sensor 214. The contact sensor 214 inputs and detects coordinates by an infrared blocking method. The method of inputting and detecting coordinates may be a method in which two light receiving and emitting devices installed at both ends of the upper side of the display 220 emit a plurality of infrared rays parallel to the display 220, and receive light that is reflected by a reflecting member installed around the display 220 and returns along the same optical path as the optical path of the light emitted by the light receiving element.

[0034] The contact sensor 214 outputs the ID of the infrared rays radiated by the two light receiving and emitting devices that are blocked by the object to the sensor controller 215, and the sensor controller 215 specifies the coordinate position that is the contact position of the object. The electronic pen controller 216 communicates with the electronic pen 2500 to determine whether the tip or the tail of the pen is touched to the display 220. The near field communication unit 219 is a communication circuit such as NFC or Bluetooth (registered trademark).

[0035] Furthermore, the display device 2 includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 201.

[0036] The contact sensor 214 is not limited to an infrared blocking type, and may use various detection means such as a capacitive touch panel that identifies the contact position by detecting a change in capacitance, a resistive film touch panel that identifies the contact position by a voltage change between two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by a contacting object touching the display unit. The electronic pen controller 216 may determine whether or not a part of the electronic pen 2500 that is held by the user or other parts of the electronic pen has been touched, in addition to the tip and bottom of the electronic pen 2500.

[0037] <About the function> Next, the functions of the display device 2 will be described with reference to Fig. 5. Fig. 5 is an example of a functional block diagram for explaining the functions of the display device 2 in blocks. The display device 2 has a contact position detection unit 40, a drawing data generation unit 41, a drawing object management unit 42, a table object management unit 43, an object selection unit 44, a display control unit 45, a conference data storage unit 46, a network communication unit 47, a symbol recognition unit 48, and a preview image generation unit 49. Each function of the display device 2 is a function or means realized by any of the components shown in Fig. 4 operating according to an instruction from the CPU 201 in accordance with a program expanded from the SSD 204 onto the RAM 203.

[0038] The contact position detection unit 40 detects the coordinates detected by the contact sensor 214 when the tip of the input means 291 touches the display. The drawing data generation unit 41 acquires the coordinates touched by the tip of the input means 291 from the contact position detection unit 40. The drawing data generation unit 41 generates stroke data by connecting the sequence of coordinate points by interpolating them. The drawing data generation unit 41 also generates figures such as circles and rectangles recognized by the symbol recognition unit 48 based on the handwritten data.

[0039] The drawing object management unit 42 manages the areas of handwritten data generated by the drawing data generation unit 41, areas of shapes such as circles and rectangles, and areas such as tables as drawing objects by assigning drawing object IDs to these areas. The drawing object management unit 42 also manages the display position (coordinates) of each drawing object. The drawing object management unit 42 manages the drawing object ID and display position of the table object, while the table object management unit 43 manages the objects of rows and columns written in the table. The table object management unit 43 manages the handwritten data and character strings in the table, and performs processing specific to tables such as changing the order of rows and columns.

[0040] The object selection unit 44 accepts the selection of an area formed by handwritten data, or a row or column selected by handwritten data of a predetermined shape such as an arrow. The object selection unit 44 may also accept the selection of a cell. The object selection unit 44 puts an object into a selected state.

[0041] The display control unit 45 displays handwritten data, character strings, tables, operation menus, etc. on the display. The network communication unit 47 connects to a network such as a LAN or the Internet, and transmits and receives data to and from other devices via the network.

[0042] The conference data storage unit 46 stores materials (files), handwritten data, image data, character strings, and the like used in the conference in a storage device such as a hard disk.

[0043] The preview image generating unit 49 displays the results of moving rows or dividing a table as a preview data image before the table object managing unit 43 actually moves or divides the rows or divides the table.

[0044] <Drawing object information, Line object information> 6 is a schematic diagram showing the drawing object information held by the drawing object management unit 42. The drawing object information is information for managing the drawing objects displayed by the display device 2.

[0045] The drawing object ID is identification information that identifies a drawing object. A drawing object refers to all of the various display objects that are displayed on the screen. In Figure 6, the drawing object with drawing object ID = 001 is each row of a table. The bottom row in Figure 6 has a drawing object ID of 002, which represents handwritten data for an encirclement line and arrow drawn by hand in the table.

[0046] The drawing object is a table, and the row object ID is set for the row object within the table. It is identification information that identifies the row object. A row object is one row in a table. Column objects are also assigned column object IDs in the same way.

[0047] The origin coordinates indicate the position of the origin of a drawing object relative to a predetermined origin on the screen of the display device 2. The origin of a drawing object is, for example, the upper left corner of the circumscribing rectangle of the drawing object. The coordinates are expressed, for example, in pixel units of the display.

[0048] The end point coordinates indicate the position of the end point of a drawing object relative to a predetermined origin on the screen of the display device 2. The end point of a drawing object is, for example, the lower right corner of the circumscribing rectangle of the drawing object.

[0049] The object type is the type of drawing object. Examples include table (row or column), handwriting, string, shape, image, etc. Table (row) indicates a row object. Handwriting is stroke data (coordinate point sequence). Characters are strings of characters (character codes) converted from handwritten data. Strings are sometimes called text data. Shapes are geometric shapes such as triangles and squares converted from handwritten data. Images are image data such as Jpeg, Png, Tiff, etc. imported from a PC or the Internet.

[0050] 7 shows a schematic diagram of line object information. Line object information is information that extracts only line objects from drawing objects.

[0051] · Row object ID is identification information that identifies a row (or column) in a table.

[0052] The starting coordinates are the coordinates of the starting point of a line.

[0053] The end coordinates are the coordinates of the end point of a line.

[0054] <Selecting a row or column in a table> Fig. 8 is a diagram for explaining a method for selecting rows in a table. First, the display position of a row object managed by the drawing object management unit 42 will be explained with reference to Fig. 8. The position of a display pixel on the display is expressed in display coordinates with the pixel position in the upper left corner of the display area as the origin, with the rightward direction being the positive direction of the X axis and the downward direction being the positive direction of the Y axis.

[0055] That is, Coordinates = (number of pixels in the X-axis direction from the pixel at the origin, number of pixels in the Y-axis direction from the pixel at the origin) It is.

[0056] Furthermore, the drawing object management unit 42 manages the area of ​​the drawing object by a rectangle circumscribing the drawing object, with the pixel position at the upper left corner of this rectangular area being the start coordinate and the pixel position at the lower right corner of the rectangular area being the end coordinate. In the case of a line object, the upper left corner of the circumscribing rectangle of the line is the start coordinate (Sxk, Syk) and the lower right corner of the circumscribing rectangle is the end coordinate (Exk, Eyk). Here, k is the line number assigned in order from the first line. For example, if the selected line is line m, the start coordinate is (Sxm, Sym) and the end coordinate is (Exm, Eym).

[0057] The interval between contact coordinates on the touch panel is the same as the interval between display coordinates on the display, and the coordinates of both are assumed to match.

[0058] As shown in Fig. 8(a), the display device 2 displays a table. When the user selects the object selection icon, the drawing object management unit 42 determines that the mode is the object selection mode based on the coordinates detected by the contact position detection unit 40. It is not necessary to press the object selection icon.

[0059] The user hand-draws a frame 304 around the row for which the branch office item is Osaka. The object selection unit 44 calculates the coordinates of the bottom end 311, the leftmost end 312, the top end 313, and the rightmost end 314 from the coordinates of the frame 304 detected by the contact position detection unit 40, and determines that the hand-drawn frame 304 surrounds the Osaka row (details will be explained in a flowchart). The object selection unit 44 then selects the Osaka row. Note that although only one row is selected in FIG. 8(a), the user can select multiple rows.

[0060] The details of the selection are explained using the flowchart in Figure 9. Figure 9 is a flowchart showing the procedure for accepting the selection of any row in a table object. Let the row object IDs be 1, 2, ..., n, starting from Sapporo in Figure 8(a). The coordinates of the upper left corner of the circumscribing rectangle of the mth row enclosed by the user's handwritten data be the starting coordinates (Sxm, Sym), and the coordinates of the lower right be the ending coordinates (Exm, Eym).

[0061] First, the outline of the process in Fig. 9 is as follows. When the user surrounds a line in object selection mode, the object selection unit 44 judges, from the coordinates detected by the contact position detection unit 40, whether the area surrounded by the stroke data includes an area d smaller than the kth line. d is a margin that enables selection of a line even if the stroke data does not completely surround a line. If the object selection unit 44 judges that the area surrounded by the stroke data includes an area d smaller than the kth line, it selects that line. The object selection unit 44 repeats this process from k=1 to the nth line.

[0062] In Fig. 8(a), the row for Osaka is selected, and therefore in Fig. 8(b), the row for Osaka is in a selected state. The object selection unit 44 emphasizes the row for Osaka by changing its color and surrounding it with a line 305.

[0063] 9 will be described. The object selection unit 44 sets a variable k to 1 (S1). k represents a row number. The object selection unit 44 judges whether or not a termination condition (k≦n is No) is met (S2). n is the number of rows in the table. The termination condition determines that the determination of whether or not the last row is surrounded by handwritten data has been completed.

[0064] If the termination condition is not met, the object selection unit 44 judges whether the coordinate of the top end 313 is greater than Syk+d (S3).

[0065] Similarly, the object selection unit 44 determines whether the coordinates of the bottom end 311 are smaller than Eyk-d (S4).

[0066] The object selection unit 44 determines whether the coordinate of the rightmost end 314 is greater than Exk-d (S5).

[0067] The object selection unit 44 determines whether the coordinate of the leftmost edge 312 is smaller than Sxk+d (S6).

[0068] If all of steps S3 to S6 are Yes (Yes in S7), the object selection unit 44 sets the kth row to a selected state (S8).

[0069] In step S7, if any one of steps S3 to S6 is No, the object selection unit 44 increments k by 1 (S9). That is, the object selection unit 44 determines whether or not the next row has been selected.

[0070] Note that the object selection unit 44 may not determine whether all lines are selected as in FIG. 9, but may determine only lines where the circumscribing rectangles of the stroke data overlap.

[0071] In this way, the user can select a line simply by surrounding it with stroke data.

[0072] <Another example of how to select rows> As a method for selecting a line, the user may select the line by drawing an arrow next to the line, in addition to the method of surrounding the line with stroke data.

[0073] Figure 10 is a diagram explaining how a user can select a row with an arrow. As shown in Figure 10(a), the user hand-draws an arrow 320 pointing toward Osaka next to the row whose branch office item is Osaka. The coordinates of the upper left corner of the circumscribing rectangle of the arrow 320 are set as the start coordinates (Sxp, Syp), and the coordinates of the lower right corner are set as the end coordinates (Exp, Eyp).

[0074] Also, a range of distance L from the frame is predefined as the range in which the user can handwrite a symbol to select a row (symbol detection range). This is because there are cases in which the user wants to handwrite something regardless of the selected row. Also, in Fig. 10(a), an arrow is handwritten to the left of the table, but the range of distance L from the right of the table can also be the symbol detection range.

[0075] When the object selection unit 44 determines based on the drawing object information that the handwritten data is an arrow and is next to the row for Osaka, it selects the row for Osaka (FIG. 10(b)).

[0076] FIG. 11 is a flow chart showing a procedure for accepting the selection of an arbitrary row of a table object.

[0077] When the user handwrites an arrow 320 next to the row for Osaka, the object selection unit 44 acquires the coordinates and size of the drawing object from the object information storage unit, and judges whether or not there is handwritten data within the distance L from the table (S11). In the example of Fig. 6, an arrow with drawing object ID = 002 is added as a drawing object to the drawing object consisting of only the table.

[0078] Next, table object management unit 43 judges whether the handwritten line is an arrow 320 (S12). That is, table object management unit 43 judges whether the handwritten data in the vicinity of the selected line is recognized as a predetermined symbol.

[0079] In the case of the arrow 320, the object selection unit 44 sets the variable k to 1 (S13). k represents the line number. Note that the recognition of the handwritten data is performed by the symbol recognition unit 48. The arrow is an example, and various symbols such as a check mark may be set as the symbol for selection.

[0080] The object selection unit 44 judges whether or not the termination condition (k≦n, No) is satisfied (S14), where n is the number of rows in the table.

[0081] If the end condition is not satisfied, the object selection unit 44 determines whether the center y coordinate (Syp+Eyp) / 2 of the arrow is between Sym and Eym, which correspond to the height of the selected line (S15).

[0082] If the determination in step S15 is Yes, the object selection unit 44 sets the kth row in a selected state (S17).

[0083] If the determination in step S15 is No, the object selection unit 44 increments k by 1 and returns to step S14 (S16).

[0084] In this way, the user can select any row by handwriting a symbol next to the row.

[0085] Note that the object selection unit 44 may not determine whether all the lines are selected as in FIG. 11, but may determine only the lines in which the Y coordinates of the stroke data overlap.

[0086] <How to add a row 1> When the user places a row in a selected state, the display control unit 45 displays buttons for adding and deleting a row before and after the row for Osaka.

[0087] Fig. 12(a) shows an example of a table in which buttons 321 to 323 are displayed. The three buttons 321 to 323 are displayed to the right of the selected row for Osaka. The positions of the buttons 321 to 323 are not limited to those shown in the figure. The buttons 321 to 323 may be located anywhere in the vicinity of the selected row.

[0088] Button 321 allows the user to add a row above the selected row. Button 323 allows the user to add a row below the selected row. Button 322 allows the user to delete the selected row.

[0089] When the user presses button 321, table object management unit 43 adds and displays a row above the row for Osaka (FIG. 12(b)). Table object management unit 43 sets new row object IDs for the added rows in consecutive order, starting from row 1. Therefore, the added row becomes row m, and the row object IDs below the selected row are incremented by one each.

[0090] Figure 13 shows table object information before and after a row is added. Figure 13(a) shows the table object information before a row is added, and Figure 13(b) shows the table object information after a row is added. The mth row corresponds to the added row. Because a row was added, the number of rows has increased by one, from n to n+1.

[0091] FIG. 14 is an example of a flowchart illustrating a process in which the display device 2 accepts the addition of a row.

[0092] First, as described with reference to FIGS. 9 and 11, the object selection unit 44 selects the mth row (S21).

[0093] As a result, the display control unit 45 displays a button 321 for adding a row to the mth row and a button 323 for adding a row after the mth row (the m+1th row) (S22). A button 322 is also displayed, but will be described in detail later.

[0094] The table object management unit 43 determines that the row adding button 321 has been pressed, based on the contact position of the input means 291 detected by the contact position detection unit 40 (S23).

[0095] The table object management unit 43 adds one blank row to the end of the table (S24).

[0096] Furthermore, if the pressed button 321 is the button 321 for adding a row to the mth row (Yes in S25), the table object manager 43 sets the variable k to n in order to set the last row (S26), where k represents the row number.

[0097] If the pressed button 323 is the button 323 for adding a row to the (m+1)th row (No in S25), the table object manager 43 increments m by 1 (S27). This is to end the copy process one row earlier.

[0098] Table object manager 43 judges whether a termination condition (m≦k, No) is satisfied (S28). This termination condition judges whether the target row (variable k) of the copy process is larger than the selected row m.

[0099] If the termination condition is not satisfied, the table object management unit 43 copies the drawing object in the kth row to the k+1th row (S29). The drawing object in the kth row is erased.

[0100] Then, table object manager 43 decrements k by 1 (S30) and repeats the process from step S28.

[0101] Therefore, the drawing object in the row above is copied to the row below until k=n, n-1, n-2, ..., m (or m+1).

[0102] In this manner, the user can add a line by enclosing the line with handwritten data and pressing buttons 321 and 323.

[0103] <How to add a row 2> Next, a second method for adding a row will be described with reference to Fig. 15. Fig. 15 is a diagram for explaining a method for adding a row to a table using handwritten data.

[0104] As shown in Fig. 15(a), assume that the row for Osaka is in a selected state. Next, the user handwrites a given mark (here, a + mark 325) above half the height of the row for Osaka (Fig. 15(b)). The coordinates of the upper left corner of the circumscribing rectangle of the + mark 325 are (Sxp, Syp), and the coordinates of the lower right corner are (Exp, Eyp).

[0105] The table object management unit 43 determines whether the handwritten data is a + mark 325. If it is a + mark 325, the table object management unit 43 determines whether the written position is above or below halfway of the selected Osaka row. If it is determined that the data was written above halfway of the Osaka row, the table object management unit 43 adds and displays a row above the Osaka row (FIG. 15(c)). If it is determined that the data was written below halfway of the Osaka row, the table object management unit 43 adds and displays a row below the Osaka row.

[0106] Fig. 16 is an example of a flowchart for explaining a process in which the display device 2 accepts the addition of a line. In the explanation of Fig. 16, the differences from Fig. 14 will be mainly explained. In Fig. 16, a line is added by handwritten data of a + mark 325, not by buttons 321 and 323.

[0107] The object selection unit 44 selects the mth row (S31).

[0108] When the user handwrites a + mark 325 on the boundary between the Osaka row and the Nagoya row, the object selection unit 44 obtains the coordinates and size of the drawn object from the object information storage unit and determines whether the handwritten data is within distance L from the table (S32).

[0109] Next, the table object management unit 43 judges (S33) whether the symbol recognition unit 48 has recognized the + mark 325. The + mark 325 is just an example, and other symbols may be set for adding a row.

[0110] When the + mark 325 is handwritten, the table object management unit 43 adds a blank line to the end of the table (S34).

[0111] Next, the table object management unit 43 judges whether the center y coordinate of the + mark (Syp+Eyp) / 2 is larger than the center of the height of the m-th row (Sym+Eym) / 2 (S35). The table object management unit 43 may determine the location of the + mark 325 by setting upper and lower limits for the height direction of the selected row, rather than simply checking whether it is larger or smaller.

[0112] If the determination in step S35 is No, table object manager 43 increments m by 1 (S36).

[0113] The processing of steps S37 to S40 may be the same as steps S27 to S30 in Fig. 14. That is, if the determination in step S35 is Yes, table object management unit 43 repeats copying the text of the kth line to the k+1th line from k=n to k=m so that a new line is added above the selected line.

[0114] If the determination in step S35 is No, the text in the kth line is copied to the k+1th line so that a new line is added below the selected line, and this is repeated from k=n to k=m+1.

[0115] In this way, the user can surround a line with handwritten data and then add additional lines with handwritten data of + marks.

[0116] Fig. 17 shows the drawing object information before and after a line is added. Fig. 17(a) shows the drawing object information before a line is added, and Fig. 17(b) shows the drawing object information after a line is added. The mth line in Fig. 17(a) is the selected line. If the y coordinate of the center of the + mark 325 is greater than the center of the height of the mth line, the mth line is added. Also, in Fig. 17(a), handwritten data of the + mark 325 is registered in drawing object ID=002.

[0117] 17B, ​​since a row has been added, the number of rows has increased by one from n to n+1. Also, the + mark 325 of the drawing object ID=002 has been erased, and therefore has also been erased from the drawing object information.

[0118] <How to add a row 3> Next, a third method for adding a line will be described with reference to Fig. 18. Fig. 18 is a diagram for explaining a method for adding a line using handwritten data of an insertion symbol.

[0119] As shown in Fig. 18(a), assume that the row for Osaka is in a selected state. Next, the user handwrites a specified mark (here, < mark 326) between Nagoya and Osaka. The table object management unit 43 determines that the handwritten data is the < mark 326. In the case of the < mark 326, the table object management unit 43 determines which line space the tip of the < mark 326 is located in. If it is determined that it is located between Nagoya and Osaka, the table object management unit 43 adds and displays a row above Osaka (Fig. 18(b)).

[0120] Fig. 19 is an example of a flow chart for explaining the process in which the display device 2 accepts the addition of a line. In the explanation of Fig. 19, the differences from Fig. 16 will be mainly explained. In Fig. 19, a line is added by handwritten data of a < mark 326 instead of a + mark 325. First, steps S41 and S42 may be the same as steps S31 and S32 in Fig. 16.

[0121] In step S43, the table object management unit 43 determines whether or not the symbol recognition unit 48 has detected the < mark 326 (S43).

[0122] Step S44 may be similar to step S34 in FIG.

[0123] Then, the table object manager 43 judges whether the tip of the < mark 326 is within ±h (the range of Sym±h) from the top end of the selected line (S45), where h is the margin.

[0124] If the determination in step S45 is No, the table object manager 43 determines whether the tip of the < mark 326 is within ±h (a range of Eym±h) from the bottom end of the selected line (S46), where h is the margin.

[0125] Subsequent processing steps S47 to S51 may be similar to steps S36 to S40 in Fig. 16. That is, if the determination in step S45 is Yes, table object management unit 43 repeats copying the text of the kth line to the k+1th line from k=n to k=m so that a new line is added above the selected line.

[0126] If the determination in step S46 is Yes, the text in the kth line is copied to the k+1th line so that a new line is added below the selected line, and this is repeated from k=n to k=m+1.

[0127] In this way, the user can surround a line with handwritten data and then add additional lines with handwritten data marked with the < symbol.

[0128] Fig. 20 shows drawing object information before and after a line is added. In explaining Fig. 20, the differences from Fig. 17 will be mainly explained. Fig. 20 has the same format as Fig. 17, but in Fig. 20(a), handwritten data of < mark 326 is registered in drawing object ID=002. Fig. 20(b) can be the same as Fig. 17(b).

[0129] <How to delete a row 1> Next, a method for deleting a row will be described with reference to Fig. 21 etc. Fig. 21 is a diagram for explaining a method for deleting a row. The process may be the same as Fig. 12 until the row is selected and buttons 321 to 323 are displayed.

[0130] As shown in Fig. 21(a), when the row for Osaka is selected, the display control unit 45 displays a button 322 for deleting the row to the right of the selected row. When the user presses button 322 for deleting the row for Osaka, the table object management unit 43 deletes the row for Osaka and displays the row for Fukuoka consecutively below the row for Nagoya (Fig. 21(b)).

[0131] FIG. 22 is an example of a flowchart illustrating a process in which the display device 2 accepts the deletion of a row.

[0132] First, the object selection unit 44 selects the mth row (S61).

[0133] As a result, the display control unit 45 displays the button 322 for deleting the mth line (S62).

[0134] The table object management unit 43 judges whether the row deletion button 322 has been pressed or not, according to the contact position of the input means 291 detected by the contact position detection unit 40 (S63).

[0135] When the delete button 322 is pressed, the table object management unit 43 sets the variable k to m (S64). This is to move the drawing objects from the m+1th row onwards by one row (in the case of a column operation, the drawing objects from the m+1th column onwards by one column).

[0136] Table object manager 43 judges whether or not the termination condition (k≦n, No) is met (S65).

[0137] If the termination condition is not satisfied, table object manager 43 copies the text in the k+1th line to the kth line (S66).

[0138] Then, table object manager 43 increments k by 1 (S67) and repeats the process from step S65. Therefore, table object manager 43 repeats copying the text in the (k+1)th line to the kth line from k=m to k=n.

[0139] If the termination condition is satisfied, the table object manager 43 deletes the n-th row (S68). That is, the table object manager 43 deletes the bottom row.

[0140] In this way, the user can delete a line by enclosing it in handwritten data and pressing a button.

[0141] Figure 23 is a diagram explaining table object information before and after deletion. Figure 23(a) shows the table object information before deletion, and Figure 23(b) shows the table object information after deletion. The table object information that had n rows has now become n-1 rows. Because the mth row in Figure 23(a) has been deleted, the drawing object in the mth row in Figure 23(b) has been replaced with the contents of the m+1th row in Figure 23(a).

[0142] <How to delete a row 2> Next, a line deletion method (part 2) will be described with reference to Fig. 24. Fig. 24 is a diagram for explaining a method for deleting a line in handwritten data.

[0143] As shown in Fig. 24(a), assume that the row for Osaka is in a selected state. Next, the user handwrites a specified mark (here, an x ​​mark 327) next to the row for Osaka. The table object management unit 43 determines that the handwritten data is an x ​​mark 327. In the case of the x mark 327, the table object management unit 43 determines which row the mark was written in. If it is determined that the mark is next to the row for Osaka, the table object management unit 43 deletes the row for Osaka and displays the row for Fukuoka following the row for Nagoya (Fig. 24(b)).

[0144] Fig. 25 is an example of a flowchart for explaining the process in which the display device 2 accepts the deletion of a line. In the explanation of Fig. 25, the differences from Fig. 19 or Fig. 22 will be mainly explained. In Fig. 25, a line is deleted by handwritten data of an x ​​mark instead of a button.

[0145] First, steps S71 and S72 may be similar to steps S41 and S42 in FIG.

[0146] Next, the drawing object management unit 42 judges whether or not the cross mark 327 has been handwritten (S73). The cross mark 327 is recognized by the symbol recognition unit 48.

[0147] Steps S74 to S78 may be similar to steps S64 to S68 in FIG.

[0148] That is, when an X mark is handwritten, the table object management unit 43 repeats copying the text of the k+1th line to the kth line from k=m to k=n. When the repeated process is completed, the table object management unit 43 deletes the nth line.

[0149] In this way, the user can surround a line with handwritten data and further delete the line with handwritten data of an X mark.

[0150] Fig. 26 is a diagram for explaining table object information before and after deletion. Fig. 26(a) shows the drawing object information before deletion, and Fig. 26(b) shows the drawing object information after deletion. In the explanation of Fig. 26, the differences from Fig. 23 will be mainly explained.

[0151] The drawing object information in Fig. 26(a) has handwritten data of drawing object ID=002 corresponding to an x ​​mark. As shown in Fig. 26(b), the number of line objects, which was n, has become n-1 lines. Because the mth line in Fig. 26(a) has been deleted, the drawing object in the mth line in Fig. 26(b) has been replaced with the content of the m+1th line in Fig. 26(a). In addition, the x mark 327 of drawing object ID=002 is erased, so it is also erased from the drawing object information.

[0152] <How to delete a row 3> Next, a line deletion method (part 3) will be described with reference to Fig. 27. Fig. 27 is a diagram for explaining a method for deleting a line in handwritten data.

[0153] As shown in Fig. 27(a), assume that the row for Osaka is in a selected state. Next, the user handwrites a diagonal line 328 in the row for Osaka. The table object management unit 43 determines whether the diagonal line 328 has been handwritten superimposed on the selected row. If the diagonal line 328 has been handwritten superimposed on the selected row, the table object management unit 43 determines which row the line has been written in. If it is determined that the written line is the selected row for Osaka, the table object management unit 43 deletes the row for Osaka and displays the row for Fukuoka following the row for Nagoya (Fig. 27(b)).

[0154] Fig. 28 is an example of a flowchart for explaining the process in which the display device 2 accepts the deletion of a line. In the explanation of Fig. 28, the differences from Fig. 25 will be mainly explained. In Fig. 28, a line is deleted by handwritten data of a diagonal line 328 instead of a cross mark 325.

[0155] First, step S81 may be similar to step S71 in FIG.

[0156] The table object management unit 43 judges whether the handwritten data has been handwritten (S82). That is, the table object management unit 43 simply needs to judge whether the handwritten data has been handwritten.

[0157] Next, the table object management unit 43 judges whether or not the diagonal line 328 is handwritten on the mth line (S83). That is, the table object management unit 43 judges whether or not the start point (Sxp, Syp) of the handwritten data is within a radius d of the upper left coordinate (Sxm, Sym) of the mth line, and the end point (Exp, Eyp) is within a radius d of the lower right coordinate (Exm, Eym) of the mth line. d is a margin.

[0158] Steps S84 to S88 may be the same as steps S74 to S78 in Fig. 25. That is, when a diagonal line is handwritten on the mth line, the table object management unit 43 repeats copying the text on the k+1th line to the kth line from k=m to k=n. When the repeated process is completed, the nth line is deleted.

[0159] In this manner, the user can enclose a line in handwritten data and then delete the line by handwriting a diagonal line 328 .

[0160] Fig. 29 is a diagram for explaining drawing object information before and after deletion. Fig. 29(a) shows the drawing object information before deletion, and Fig. 29(b) shows the drawing object information after deletion. In the explanation of Fig. 29, differences from Fig. 26 will be mainly explained.

[0161] The drawing object information in Fig. 29 is formally similar to that in Fig. 26. However, in Fig. 29(a), the handwritten data of drawing object ID=002 is a diagonal line 328. Fig. 29(b) may be similar to Fig. 26(b).

[0162] <Copy or move lines 1> Next, a method of copying or moving a line will be described with reference to Fig. 30. Fig. 30 is a diagram for explaining a method of copying or moving a line.

[0163] As shown in Fig. 30(a), assume that the row for Osaka is in a selected state. The user operates the input means 291 to drag the row for Osaka to the position of Sapporo in the table. The table object management unit 43 detects that the touch coordinates of the tip of the input means 291 overlap with the row for Sapporo. As a result, the preview data generation unit creates and displays an image of the table in which Osaka has been moved to the position of Sapporo (Fig. 30(b)). A preview refers to simulating the output result and checking it on the screen before actually performing some output on a computer or the like.

[0164] When a predetermined time t seconds has elapsed without the user removing the input means 291 from the display while the preview image in Fig. 30(b) is being displayed, the table object management unit 43 creates and displays a table in which the character string in the Osaka row is copied to the Sapporo row (Fig. 30(c)). Therefore, the Sapporo row is deleted by overwriting.

[0165] On the other hand, if the user drops (moves input means 291 away from the display) before the predetermined time t seconds have elapsed while the preview image in Fig. 30(b) is displayed, table object management unit 43 displays a table in which Osaka has been moved to Sapporo's position. The rows for Nagoya are moved down one row at a time from the row for Sapporo. Also, preview image generation unit 49 makes the preview image non-displayable (Fig. 30(d)).

[0166] FIG. 31 is an example of a flowchart showing a procedure by which the display device 2 accepts row movement.

[0167] Table object management unit 43 sets the mth row (an example of the first row) to a selected state (S91).

[0168] The table object management unit 43 judges whether or not the m-th line has been pressed by the input means 291 based on the contact coordinates of the input means 291 detected by the contact position detection unit 40 (S92). When the user moves the position of the input means 291 without releasing the input means 291, the table object management unit 43 detects this based on the contact coordinates of the input means 291 detected by the contact position detection unit 40 (S93).

[0169] When the input means 291 is dragged from the selected row m to row m' (an example of the second row), the table object management unit 43 writes the touch coordinate m' into memory, and the preview image generation unit 49 displays a preview image (S94). The preview image is generated by the same process as the process described later in Fig. 31. In the case of columns, the m column and the m' column are examples of the first column and the second column, respectively.

[0170] When the input means 291 is removed from the display, the table object management unit 43 judges whether the touch coordinate m' is smaller than the selected row m (S95).

[0171] If the touch coordinate m' is smaller than the selected row m (if the touch coordinate has moved up), the table object management unit 43 copies the text of the mth row to a variable (S96). A variable is a container for data. This is because the mth row is to be overwritten.

[0172] Then, table object manager 43 sets variable k to m (S97).

[0173] Table object manager 43 judges whether or not a termination condition (No for k=m') is met (S98). This termination condition judges whether copying from row m to row m' has been completed.

[0174] If the termination condition is not met, table object manager 43 copies the text in the (k-1)th line to the kth line (S99).

[0175] Table object manager 43 decrements k by 1 and repeats the process from step S98 (S100).

[0176] If the termination condition is met, table object management unit 43 copies the text in the mth line stored in the variable to the m'th line (S101).

[0177] If the touch coordinate m' is larger than the selected row m (if the touch coordinate m' is moved downwards), the table object management unit 43 copies the text of the mth row to a variable (S102). A variable is a container for data. This is because the mth row is to be overwritten.

[0178] Then, the table object manager 43 sets the variable k to m (S103).

[0179] Table object manager 43 judges whether or not the termination condition (No for k=m') is satisfied (S104).

[0180] If the termination condition is not met, the table object manager 43 copies the text in the (k+1)th line to the kth line (S105).

[0181] Table object manager 43 increments k by 1 and repeats the process from step S104 (S106).

[0182] If the termination condition is met, table object management unit 43 copies the text in the mth line stored in the variable to the m'th line (S107).

[0183] In this manner, the user can select a row and move it by dragging it with input means 291 .

[0184] Figure 32 shows the drawing object information before and after the line is moved. Figure 32(a) shows the drawing object information before the line is moved, and Figure 32(b) shows the drawing object information after the line is moved. The drawing object information in Figure 32 has a new text column.

[0185] In the text column of Fig. 32(a), T1 to Tn are drawing objects stored in the respective rows. Text Tm in Fig. 32(a) is moved to the first row in Fig. 32(b), and texts T1 to Tm-1 are moved down by one row.

[0186] <Line movement part 2> Next, a method of moving a line using handwritten data (part 2) will be described with reference to Fig. 33. Fig. 33 is a diagram for explaining a method of moving a line using handwritten data.

[0187] As shown in Fig. 33(a), assume that the row for Osaka is in a selected state. The user handwrites an arrow 329 pointing from the row for Osaka to the row for Sapporo (Fig. 33(b)). When the table object management unit 43 determines that the start point of the handwritten data (arrow 329) is the selected row, it determines in which row of the table the end point of the handwritten data (arrow 329) is written. For example, if it is determined that the start point is Osaka and the end point is Sapporo, the table object management unit 43 creates and displays a table in which the row for Osaka has been moved to the position of Sapporo (Fig. 33(c)).

[0188] Fig. 34 is an example of a flowchart for explaining the process in which the display device 2 accepts the movement of a line. In the explanation of Fig. 34, the differences from Fig. 31 will be mainly explained. In Fig. 34, a line is moved by handwritten data of an arrow 329, not by dragging.

[0189] For this reason, the table object management unit 43 judges whether or not the starting point of the handwritten data (arrow 329) is handwritten on the m-th line (S112). The table object management unit 43 may also judge whether or not the shape is an arrow.

[0190] If it is handwritten on the mth line, the table object management unit 43 writes the line m' on which the end point of the arrow is handwritten into memory (S113). The table object management unit 43 may determine whether the end point is inside the table.

[0191] The subsequent steps S115 to S127 may be similar to steps S95 to S107 in FIG.

[0192] In this manner, the user can select a row and move the row by hand-drawing arrows 329 .

[0193] Figure 35 shows drawing object information before and after the movement. Figure 35(a) shows the drawing object information before the line is moved, and Figure 35(b) shows the drawing object information after the line is moved. In the explanation of Figure 35, the differences from Figure 32 will be mainly explained.

[0194] The drawing object information in Fig. 35(a) has handwritten data of drawing object ID=002 corresponding to the arrow 329. As shown in Fig. 35(b), the text Tm in Fig. 35(a) is moved to the first line in Fig. 35(b), and the texts T1 to Tm-1 are moved down by one line.

[0195] Furthermore, the arrow 329 with the drawing object ID=002 is deleted, and is therefore also deleted from the drawing object information.

[0196] <Line movement part 3> Next, a method of moving a line using handwritten data (part 3) will be described with reference to Fig. 36. Fig. 36 is a diagram for explaining a method of moving a line using handwritten data.

[0197] As shown in Fig. 36(a), assume that the row for Osaka is in a selected state. The user handwrites an arrow 330 from next to the row for Osaka toward the row for Sapporo (Fig. 36(b)). When the table object management unit 43 determines that the start point of the handwritten data (arrow 330) is the selected row, it determines in which row of the table the end point of the handwritten data (arrow 330) is written. For example, if it is determined that the start point is Osaka and the end point is Sapporo, the table object management unit 43 creates and displays a table in which the row for Osaka has been moved to the position of Sapporo (Fig. 36(c)).

[0198] Fig. 37 is an example of a flowchart for explaining the process in which the display device 2 accepts row movement. In the explanation of Fig. 37, the differences from Fig. 34 will be mainly explained. In Fig. 37, the row is moved by the handwritten data of the arrow 330 outside the table.

[0199] For this reason, the table object management unit 43 judges whether the starting point of the handwritten data (arrow 330) is on the mth row and whether it is handwritten within a distance L (within a predetermined distance) from the table (S132). The table object management unit 43 may judge whether the shape is an arrow.

[0200] If the arrow is handwritten on the mth line within the distance L from the table, the table object management unit 43 writes the line m' on which the arrow's end point is handwritten into memory (S133). The table object management unit 43 may determine whether the arrow's end point is handwritten within the distance L from the table.

[0201] The subsequent steps S135 to S147 may be similar to steps S115 to S127 in FIG.

[0202] This way, the user can select a row and move it around by hand-drawing an arrow.

[0203] Figure 38 shows the drawing object information before and after the row has been moved. In explaining Figure 38, the differences from Figure 35 will be mainly explained. Figure 38 has the same format as Figure 35, but in Figure 38(a), handwritten data for an arrow 329 handwritten outside the table is registered in drawing object ID=002. Figure 38(b) can be the same as Figure 35(b).

[0204] Next, a method for a user to divide a part of a table will be described with reference to Fig. 39. Fig. 39 is a diagram for explaining a method for dividing a table.

[0205] As shown in FIG. 39(a), the row for Osaka is in a selected state. The user drags and drops the row for Osaka outside the table using the input means 291 (FIG. 39(b)). The table object management unit 43 creates a second table 340 that inherits the title row for the Osaka row at the coordinates where it was dropped. A title row is a row that includes at least the first row "First half of 2020" and further has a different color set from the general rows. The table object management unit 43 deletes the row for Osaka from the original table (FIG. 39(c)).

[0206] FIG. 40 is a flow chart showing a procedure in which the display device 2 accepts a request to divide a table and divides the table.

[0207] First, the object selection unit 44 selects the mth row (S151).

[0208] Next, the table object management unit 43 judges whether the m-th row has been touched (S152). When the user moves the input means 291 without removing it from the display (S153), the table object management unit 43 judges whether the touched coordinates are inside the table (S154).

[0209] If it is within a table, since it is a move or copy operation, table object management unit 43 holds m' rows in a variable, and preview image generation unit 49 displays a preview image (S156). This preview image is the same as step S94 in FIG.

[0210] When the user drags the input means 291 outside the table, the preview image generating unit 49 displays a preview image of the table combining the title row and the mth row of the table at the coordinates touched by the tip of the input means 291 (S155). The title row is predetermined, for example, the first row. Alternatively, a row that is colored may be determined to be the title row. In this embodiment, "First Half of 2020" and "Branch April May June" are the title rows.

[0211] Then, when the user removes the input means 291 from the display, the table object management unit 43 judges whether m'=null (S157). Null indicates the outside of the table.

[0212] When the user touches the outside of the table with input means 291 and removes it from the display, a deletion process is executed for the rows of the original table that are to be divided into second table 340. Therefore, the processes in steps S158 to S162 are the same as those in steps S64 to S68 in FIG.

[0213] Then, the table object management unit 43 displays the second table 340 by combining the title row and the m-th row of the original table at the coordinates last touched by the input means 291 (S163).

[0214] If the user touches the inside of the table, it is determined that the user has performed a row movement process. Therefore, the processes in steps S164 to S176 may be the same as those in steps S135 to S147 in FIG.

[0215] This way, a user can split a table by selecting a row and dragging it outside the table.

[0216] Figure 41 is a diagram explaining the drawing object information before and after dividing a table. Figure 41(a) shows the object information before division, and Figure 41(b) shows the object information after division. As a result of the division, a new second table 340 was generated, so in Figure 40(b), the last drawing object ID has changed from 001 to 002.

[0217] In the rows of row object IDs 001 to 00n in Fig. 41(a), since the mth row has been divided, the row of row object ID 00m corresponds to the drawing object whose drawing object ID is 002. Also, the rows from row m+1 onwards are moved up by one row.

[0218] Also, the row object ID of the drawing object ID=002 becomes 001. This is because row object IDs are assigned starting from the first row in the second table 340 after division. Note that the start point coordinates and end point coordinates are also new in the table after division.

[0219] <Major Effects> As described above, the display device 2 of the present embodiment can execute editing such as adding, moving, deleting, or dividing rows and columns on handwritten data, thereby improving operability. EXAMPLES

[0220] In the following embodiment, another configuration example of the display device 2 will be described.

[0221] <<Another configuration example 1 of the display device>> Although the display device 2 of the present embodiment has been described as having a large touch panel, the display device 2 is not limited to having a touch panel.

[0222] Fig. 42 is a diagram showing another example of the configuration of the display device 2. In Fig. 42, a projector 411 is installed on the upper side of a normal whiteboard 413. This projector 411 corresponds to the display device 2. The normal whiteboard 413 is not a flat panel display integrated with a touch panel, but a whiteboard on which a user directly writes by hand with a marker. The whiteboard may be a blackboard, and may be any flat surface large enough to project an image.

[0223] The projector 411 has an ultra-short focal length optical system and can project an image with little distortion from a distance of about 10 cm onto the whiteboard 413. This image may be transmitted from a PC connected wirelessly or via a wire, or may be stored in the projector 411.

[0224] A user writes handwriting on the whiteboard 413 using a dedicated electronic pen 2501. The electronic pen 2501 has a light-emitting part, for example at the tip, that is switched on and emits light when the user presses it against the whiteboard 413 to write. The wavelength of the light is near-infrared or infrared, so it is invisible to the user's eyes. The projector 411 has a camera that captures the light-emitting part, analyzes the image, and identifies the direction of the electronic pen 2501. The electronic pen 2501 also emits sound waves along with the light emission, and the projector 411 calculates the distance based on the arrival time of the sound waves. The projector 411 can identify the position of the electronic pen 2501 based on the direction and distance. Handwritten data is drawn (projected) at the position of the electronic pen 2501.

[0225] Since the projector 411 projects the menu 430, when the user presses a button with the electronic pen 2501, the projector 411 identifies the position of the electronic pen 2501 and the pressed button by the ON signal of the switch. For example, when the save button 431 is pressed, the handwritten data (coordinate point sequence) handwritten by the user is saved by the projector 411. The projector 411 saves the handwritten information in a predetermined server 412 or USB memory 2600 or the like. The handwritten information is saved for each page. Since the information is saved as coordinates rather than as image data, the user can re-edit it. However, in this embodiment, the menu 430 does not need to be displayed because an operation command can be called up by handwriting. EXAMPLES

[0226] <<Another configuration example 2 of the display device>> 43 is a diagram showing another example of the configuration of the display device 2. In the example of FIG.

[0227] The terminal device 600 is connected by wire to the image projection device 700A and the pen operation detection device 810. The image projection device 700A projects image data input by the terminal device 600 onto the screen 800.

[0228] The pen operation detection device 810 communicates with the electronic pen 820 and detects the operation of the electronic pen 820 in the vicinity of the screen 800. Specifically, the electronic pen 820 detects coordinate information indicating a point pointed to by the electronic pen 820 on the screen 800 (the detection method may be the same as that in FIG. 42 ), and transmits the coordinate information to the terminal device 600.

[0229] The terminal device 600 generates image data of handwritten data input by the electronic pen 820 based on the coordinate information received from the pen operation detection device 810, and causes the image projection device 700A to draw an image of the handwritten data on the screen 800.

[0230] Furthermore, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by combining the background image projected by the image projection device 700A and the image of the handwritten data input by the electronic pen 820. EXAMPLES

[0231] <<Another configuration example 3 of the display device>> Fig. 44 is a diagram showing a configuration example of the display device 2. In the example of Fig. 44, the display device 2 includes a terminal device 600, a display 800A, and a pen motion detection device 810A.

[0232] The pen action detection device 810A is disposed near the display 800A, detects coordinate information indicating a point indicated by the electronic pen 820A on the display 800A (the detection method may be the same as that in FIG. 42), and transmits the coordinate information to the terminal device 600. In the example of FIG. 44, the electronic pen 820A may be charged by the terminal device 600 via a USB connector.

[0233] Based on the coordinate information received from the pen operation detection device 810A, the terminal device 600 generates image data of the image of the handwritten data input by the electronic pen 820A, and displays it on the display 800A. EXAMPLES

[0234] <<Another Configuration Example 4 of the Display Device>> Fig. 45 is a diagram showing a configuration example of the display device 2. In the example of Fig. 45, the display device 2 includes a terminal device 600 and an image projection device 700A.

[0235] The terminal device 600 performs wireless communication (Bluetooth (registered trademark) or the like) with the electronic pen 820B to receive coordinate information of a point indicated by the electronic pen 820B on the screen 800. The coordinate information may be obtained by the electronic pen 820B reading minute position information formed on the screen 800, or the coordinate information may be received from the screen 800.

[0236] Then, based on the received coordinate information, the terminal device 600 generates image data of an image of the handwritten data input by the electronic pen 820B, and causes the image projection device 700A to project the image of the handwritten data.

[0237] Furthermore, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by combining the background image projected by the image projection device 700A and the image of the handwritten data input by the electronic pen 820B.

[0238] As described above, each of the above-described embodiments can be applied to various system configurations.

[0239] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0240] For example, although the present embodiment has been described with respect to editing of rows, the user can edit columns in the same manner. For columns, the title column is at the left end.

[0241] Although the present embodiment has been described mainly with respect to input of handwritten data, the display device is capable of character recognition of handwritten data. Drawing objects in a table may be handwritten data or character strings.

[0242] Also, in this embodiment, a display device that can be used as an electronic whiteboard has been described, but the display device may be any device capable of displaying images, such as digital signage. Also, a projector may display display data instead of a display. In this case, in this embodiment, the coordinates of the pen are detected by a method in which the coordinates of the pen tip are detected by a touch panel, but the display device 2 may detect the coordinates of the pen tip by ultrasonic waves. The pen emits light and also emits ultrasonic waves, and the display device 2 calculates the distance based on the arrival time of the ultrasonic waves. The display device 2 can identify the position of the pen based on the direction and distance. The projector draws (projects) the trajectory of the pen as a stroke.

[0243] In addition, although the present embodiment has been described as an example of an electronic whiteboard, any information processing device having a touch panel can be suitably applied. A device having the same function as an electronic whiteboard is also called an electronic whiteboard, an electronic information board, an interactive board, etc. Examples of information processing devices equipped with a touch panel include output devices such as PJ (Projector), digital signage, HUD (Head Up Display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, network home appliances, notebook PCs (Personal Computers), mobile phones, smartphones, tablet terminals, game consoles, PDAs, digital cameras, wearable PCs, desktop PCs, etc.

[0244] Moreover, the configuration examples in Fig. 5 and the like are divided according to main functions in order to facilitate understanding of the processing by the display device 2. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing by the display device 2 can be divided into more processing units according to the processing contents. Also, it can be divided so that one processing unit includes more processes.

[0245] Although the display device 2 has the functions of the present embodiment by itself in the above description, the functions of the display device 2 may be provided by a server. In this case, the display device 2 and the server communicate via a network. The display device 2 displays handwritten data and transmits it to the server, and the server provides functions such as character recognition and table functions.

[0246] In addition, in this embodiment, even if a threshold is exemplified as a comparison target, the threshold is not limited to the exemplified value. Therefore, in this embodiment, for all thresholds, the descriptions "less than threshold" and "below threshold" have the same meaning, and the descriptions "exceeding threshold" and "above threshold" have the same meaning. For example, the description "less than threshold" when the threshold is 11 has the same meaning as the description "below threshold" when the threshold is 10. Furthermore, the description "exceeding threshold" when the threshold is 10 has the same meaning as the description "below threshold" when the threshold is 11.

[0247] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0248] The contact position detection unit 40 is an example of a receiving means. The object selection unit 44 is an example of a selection receiving means. The table object management unit 43 is an example of a control means. The symbol recognition unit 48 is an example of a recognition means. [Explanation of symbols]

[0249] 2 Display device [Prior art documents] [Patent documents]

[0250] [Patent Document 1] JP 2015-219628 A

Claims

1. A display device for displaying a handwritten table, a receiving means for receiving handwritten data input by an input means; a selection receiving means for receiving a selection of a row or column of the table by the handwritten data; a control means for editing the table with respect to the selected row or column of the table in response to a user operation when a row or column of the table is selected by the selection receiving means, the control means displays a button for adding a row or a column near the row or column selected by the selection receiving means; When the reception means receives a press of the button, a row or a column is added before or after the row or column selected by the selection reception means. Display device.

2. The display device according to claim 1 , wherein the selection receiving means receives a selection of a row or column surrounded by the handwritten data received by the receiving means.

3. a recognition unit that recognizes the handwritten data received by the reception unit, 3. The display device according to claim 1 or 2, wherein when the recognition means recognizes the handwritten data handwritten next to a row or column for which the selection accepting means has accepted a selection as a predetermined symbol, the control means adds a row or column before or after the row or column for which the selection accepting means has accepted a selection.

4. a recognition unit that recognizes the handwritten data received by the reception unit, 3. The display device according to claim 1, wherein when the recognition means recognizes the handwritten data handwritten at the boundary of a row or column for which the selection receiving means has accepted a selection as a predetermined symbol, the control means adds a row or column to the boundary.

5. the control means displays a button for deleting a row or column near the row or column selected by the selection receiving means; 5. The display device according to claim 1, wherein, when the accepting means accepts pressing of the button, the row or column selected by the selection accepting means is deleted.

6. a recognition unit that recognizes the handwritten data received by the reception unit, A display device as described in any one of claims 1 to 4, wherein when the recognition means recognizes the handwritten data handwritten next to a row or column for which the selection accepting means has accepted a selection as a predetermined symbol, the control means deletes the row or column for which the selection accepting means has accepted a selection.

7. When the handwritten data for deleting a row or column selected by the selection receiving means is received by the receiving means, 5. The display device according to claim 1, wherein the row or column for which the selection receiving means has received a selection is deleted.

8. A display method performed by a display device that displays a handwritten table, comprising the steps of: A step in which a receiving means receives input of handwritten data by an input means; a selection receiving means for receiving a selection of a row or column of the table by the handwritten data; a step of, when a row or column of the table is selected by the selection receiving means, editing the table with respect to the selected row or column of the table in response to a user operation by a control means; a step of the control means displaying a button for adding a row or column near the row or column selected by the selection receiving means; adding a row or column before or after the row or column selected by the selection receiving means when the receiving means receives a press of the button; A display method having the following structure:

9. A display device for displaying a table created by handwriting, a receiving means for receiving handwritten data input by an input means; a selection receiving means for receiving a selection of a row or column of the table by the handwritten data; when a row or column of the table is selected by the selection receiving means, the control means functions as a control means for editing the table with respect to the selected row or column of the table in response to a user operation; the control means displays a button for adding a row or a column near the row or column selected by the selection receiving means; When the reception means receives a press of the button, a row or a column is added before or after the row or column selected by the selection reception means. program.

Images