Display device, display method, and program

Abstract

To provide a display device with which it is possible to expand a frame for characters to fit in.SOLUTION: Provided is a display device comprising: acceptance means for accepting input of characters handwritten by input means 291 in a frame 402 enclosed by a straight line; calculation means for calculating the size of the handwritten characters; and control means for expanding the frame for the handwritten characters to fit in, on the basis of the size of the characters calculated by the calculation means and a space between the frame and the handwritten characters in the handwriting direction in which the handwritten characters are added on.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a display device, a display method, and a program. [Background technology]

[0002] Display devices that use handwriting recognition technology to convert handwritten data into characters and display them on a display are known. Display devices equipped with relatively large touch panels are placed in conference rooms and public facilities and are used by multiple users as electronic whiteboards, etc.

[0003] In a display device, a user can handwrite characters within a frame such as a figure or a table. When the frame is too small and the characters cannot fit within the frame, a technique is known for fitting the characters into the frame (see, for example, Patent Document 1). Patent Document 1 discloses a technique for automatically reducing the characters to an appropriate size so that the entire character string fits within the frame when the handwritten characters overflow. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has a problem in that the frame cannot be expanded, and if the characters are reduced in size without changing the size of the frame, the characters may become difficult to read.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide a display device that can expand a frame so that characters can be inserted. [Means for solving the problem]

[0006] The present invention provides a method for inputting a character by handwriting into a box surrounded by straight lines, comprising: When the space between the handwriting direction frame into which the handwritten character is added and the handwritten character is equal to or smaller than a threshold based on the width of the character size in the case where the handwriting direction is horizontal writing, or equal to or smaller than a threshold based on the height of the character size in the case where the handwriting direction is vertical writing, the frame is expanded so that the handwritten character can be included. A display device having a control means is provided. Effect of the Invention

[0007] It is possible to provide a display device whose frame can be expanded to accommodate characters. [Brief description of the drawings]

[0008] [Figure 1] 13A and 13B are diagrams illustrating how the size of a table frame is changed. [Diagram 2] 11 is a diagram for explaining an outline of a process in which the display device expands a frame. FIG. [Diagram 3] FIG. 1 is a schematic perspective view of a display device. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device. [Diagram 5] FIG. 2 is an example of a functional block diagram for explaining functions of the display device by dividing them into blocks. [Figure 6] FIG. 13 is a diagram illustrating a method for expanding a frame. [Figure 7] FIG. 13 is a diagram illustrating a method for determining the size of one character. [Figure 8] 11 is a flowchart illustrating an example of a procedure in which the display device expands a frame based on handwritten data handwritten in the frame. [Figure 9] FIG. 13 is a diagram illustrating the expansion of the bottom end of the frame. [Figure 10] 13A and 13B are diagrams illustrating the expansion of the right and bottom ends of the frame. [Figure 11] FIG. 13 is a diagram illustrating the deletion of spaces after handwriting is completed. [Figure 12] 13 is a diagram illustrating a frame that is expanded in response to pressing an expansion button. FIG. [Figure 13] 13A and 13B are diagrams illustrating a frame that is reduced in response to pressing of a reduce button. [Figure 14] 11 is a flowchart illustrating an example of a procedure in which the display device expands a frame in response to pressing of an expansion button. [Figure 15] FIG. 13 is a diagram showing another configuration example of the display device. [Figure 16] FIG. 13 is a diagram showing another configuration example of the display device. [Figure 17] FIG. 13 is a diagram showing another configuration example of the display device. [Figure 18] FIG. 13 is a diagram showing another configuration example of the display device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A display device and a display method performed by the display device will be described below as an example of an embodiment of the present invention with reference to the drawings. EXAMPLES

[0010] <Comparative Example> First, a comparative example suitable for explaining the display device 2 of the present embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining how the size of frame 402 of table 401 is changed. Note that the comparative example is not a conventional technique.

[0011] 1(a), a table 401 has four boxes 402. When the user is writing characters by hand using the input means 291 in the boxes 402, the display device 2 expands the boxes 402 to fit the stroke data 403 being written.

[0012] As shown in FIG. 1(b), the display device 2 expands the frame 402 to the right as the coordinates of the handwritten stroke data move to the right.

[0013] However, in the method of expanding the frame 402 according to the coordinates of the stroke data being handwritten, there is always a situation where there is not enough space between the right end of the handwritten data and the frame 402, which can make it difficult for the user to write. For example, when the user wants to handwrite the next stroke data to the right of the right end of the stroke data (when writing from the right end, such as "イ", or when typing a punctuation mark or a voiced consonant), there may be a situation where there is not enough space. This can be seen from the fact that when the user stops handwriting, as in Figure 1(c), there is no space to start writing the next character. In this case, the user needs to change the size of the frame 402 himself before writing.

[0014] <Outline of the operation of the display device> 2 is a diagram for explaining an outline of the process in which the display device 2 expands the frame 402. As shown in Fig. 2(a), there is no more space to write characters at the right end of the frame 402. When the tip of the input means 291 leaves the display, the display device 2 judges how much space there is to write, and if it judges that there is not enough space, it expands the frame 402 so that there is space for about one character, as shown in Fig. 2(b).

[0015] As an example, the display device 2 calculates the character size of one character from stroke data that has already been handwritten in the frame 402. When the space at the right end in the handwriting direction (the space between the frame in the handwriting direction and the right end of the last handwritten character) becomes smaller than the character size, the display device 2 expands the frame 402 in the right direction in the handwriting direction by at least one character. This allows the user to handwrite the next character in the space 405 that is larger than one character.

[0016] <Terminology> The input means may be any means capable of specifying coordinates on a touch panel and writing by hand, such as a pen, a human finger or hand, or a rod-shaped member.

[0017] A stroke refers to a series of operations in which a user presses an input means against a display, moves it continuously, and then releases it from the display. Stroke data is information displayed on the display based on a trajectory of coordinates input by the input means. Stroke data may be appropriately interpolated. Handwritten data is data having one or more stroke data. Handwritten input refers to handwritten data being input by a user.

[0018] An object that is displayed on the display based on stroke data is called an object. Object means a target, but in this embodiment, it means a display target. The character string converted from handwritten data through character recognition may include not only text data, but also data displayed based on user operations, such as fixed characters such as "Done," stamps displayed as marks, shapes such as circles and stars, and straight lines.

[0019] A character string is one or more character codes (fonts) that are converted from handwritten data by character recognition. A character string can be one or more characters that are handled by a computer. Characters include numbers, alphabets, and symbols. A character string is also called text data.

[0020] A frame is a contour that is bounded by at least four straight lines. Examples of frames include each cell that makes up a table and a rectangle that is a shape.

[0021] Space means an empty space, a margin, a blank space, etc. In this embodiment, the blank space between a character and a frame is called a space.

[0022] To expand means to make something larger by widening its scope. To reduce means to make something smaller by shrinking it.

[0023] <Configuration example> 3 shows an example of a schematic perspective view of the display device 2. A user (user) uses the display device 2. The display device 2 of this embodiment allows the user to handwrite on the display using an input means 291 such as a hand or an electronic pen 2500.

[0024] 3 is placed horizontally, but it can also be placed vertically. The user can switch between horizontal and vertical orientations by rotating the display device 2 around the center of the display.

[0025] <Hardware configuration example> 4 is a diagram showing an example of a hardware configuration of the display device 2. The display device 2 of this embodiment includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, a network controller 205, and an external device connection I / F 206 (Interface), and is a shared terminal for sharing information among a plurality of users.

[0026] Of these, the CPU 201 controls the overall operation of the display device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201.

[0027] The SSD 204 stores various data such as programs for the display device 2. The programs may be application programs that run on an information processing device equipped with a general-purpose OS (Windows (registered trademark), Mac OS (registered trademark), Android (registered trademark), iOS (registered trademark), etc.).

[0028] A network controller 205 controls communication with a network. An external device connection I / F 206 controls communication with a USB (Universal Serial Bus) memory 2600 and external devices (a camera 2900, a speaker 2800, and a microphone 2700).

[0029] The display device 2 also includes a capture device 211, a GPU 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication unit 219, and an antenna 219a of the short-range communication unit 219.

[0030] Of these, the capture device 211 displays video information as still images or moving images on the display of the PC 10. The GPU (Graphics Processing Unit) 212 is a semiconductor chip that specializes in graphics. The display controller 213 controls and manages the screen display in order to output the output image from the GPU 212 to the display 220 etc.

[0031] The contact sensor 214 detects that the electronic pen 2500, the user's hand H, or the like has come into contact with the display 220. When the electronic pen 2500 and the hand H are not distinguished from each other, they are referred to as an input means 291.

[0032] The sensor controller 215 controls the processing of the contact sensor 214. The contact sensor 214 inputs and detects coordinates by an infrared blocking method. The method of inputting and detecting coordinates may be a method in which two light receiving and emitting devices installed at both ends of the upper side of the display 220 emit a plurality of infrared rays parallel to the display 220, and receive light that is reflected by a reflecting member installed around the display 220 and returns along the same optical path as the optical path of the light emitted by the light receiving element.

[0033] The contact sensor 214 outputs the ID of the infrared rays radiated by the two light receiving and emitting devices that are blocked by the object to the sensor controller 215, and the sensor controller 215 specifies the coordinate position that is the contact position of the object. The electronic pen controller 216 communicates with the electronic pen 2500 to determine whether the pen tip or the pen tail has touched the display 220. The near field communication unit 219 is a communication circuit such as NFC or Bluetooth (registered trademark).

[0034] Furthermore, the display device 2 includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 201.

[0035] The contact sensor 214 is not limited to an infrared blocking type, and may use various detection means such as a capacitive touch panel that identifies the contact position by detecting a change in capacitance, a resistive film touch panel that identifies the contact position by a voltage change between two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by a contact object touching a display. The electronic pen controller 216 may determine whether or not a part of the electronic pen 2500 that is held by the user or other parts of the electronic pen have been touched, in addition to the tip and bottom of the electronic pen 2500.

[0036] <About the function> Fig. 5 is a diagram showing an example of a functional block diagram for explaining functions of the display device 2 in blocks. Each functional unit of the display device 2 shown in Fig. 5 is a function or means realized by any of the components shown in Fig. 4 operating in response to an instruction from the CPU 201 in accordance with a program loaded from the SSD 204 onto the RAM 203.

[0037] The display device 2 includes a contact detection unit 301 that detects that the input means 291 has contacted the display 220, and a coordinate detection unit 302 that detects the coordinates of the contact position. The contact detection unit 301 mainly transmits a signal indicating that the pen tip or the pen tail of the electronic pen 2500 has been pressed. The contact detection unit 301 detects contact based on this signal.

[0038] When a contact is detected by the contact detection unit 301, the coordinate detection unit 302 notifies an event notifying the contact together with the coordinate information detected by the coordinate detection unit 302 to an event allocation unit 303.

[0039] The event allocation unit 303 determines whether the contact is a UI control operation, a gesture operation, or a stroke drawing, and allocates the event to the appropriate processing unit. A UI control operation refers to the user pressing a button or icon displayed as an operation menu. UI is an abbreviation for user interface. A gesture operation refers to the selection, movement, expansion or reduction of handwritten data, etc.

[0040] When the contact of the input means 291 is determined to be an operation of a UI control, the event is sent to the operation processing unit 304, which executes processing corresponding to the user request. The result of the processing is sent to an image processing unit 313 (described later) to be reflected in the screen display.

[0041] When the contact of the input means 291 is determined to be a gesture operation, the event is transmitted to the gesture processing unit 305, which determines the operation associated with the gesture performed by the user and instructs the operation processing unit 304 to execute the corresponding operation. The operation processing unit 304 executes the process instructed by the gesture processing unit, similar to the case of an operation of a UI control.

[0042] When the contact of the input means 291 is determined to be a stroke drawing, the event is transmitted to the stroke processing unit 306. The stroke processing unit 306 draws handwritten data based on the coordinate points of the stroke data, deletes the drawn handwritten data, or edits the drawn handwritten data.

[0043] The character size calculation unit 307 calculates the size (width and height) of one stroke data or the size (width and height) of one character in the character string drawn by the stroke processing unit 306. The method of calculating the character size will be described later.

[0044] The drawing object processing unit 308 is a part that performs conversion and editing of drawing objects such as lines, figures, tables, and images. For example, the drawing object processing unit 308 converts stroke data into lines, figures, tables, and the like, and converts it into text data using an OCR (Optical Character Recognition) function. It also performs scaling (expansion / contraction) and deformation of drawing objects such as lines, figures, tables, and images. The drawing object processing unit 308 performs the expansion and contraction of a frame 402, which will be described later. The drawing object processing unit 308 expands the frame depending on the result of comparison between the calculated character size and the space between the frame in the handwriting direction and the handwritten character.

[0045] The stroke data output by the stroke processing unit 306 and the drawing objects output by the drawing object processing unit 308 are drawn on pages managed by a page processing unit 310. The page processing unit 310 manages pages, such as adding and deleting pages and switching between displayed pages. Information on each page, such as the stroke data and drawing objects, is stored in a page data storage unit 311.

[0046] The UI image generation unit 309 generates an operation menu to be displayed on the screen based on information (which menu to display) output from the page processing unit 310 and user operations. The generated operation menu is output to the display unit 312 and displayed on the display of the display device 2.

[0047] [Table 1] Table 1 shows a schematic diagram of the object information stored in the page data storage unit 311. The object information is information for managing various objects displayed by the display device 2.

[0048] The object ID is identification information that identifies an object on the screen. An object refers to any of the various display objects displayed on the screen.

[0049] Type is the type of object. Examples include table (row or column), handwriting, character, figure, image, etc. A table indicates a table object. Handwriting is stroke data (sequence of coordinate points). Characters are character strings (character codes) converted from handwritten data. Character strings are sometimes called text data. Figures are geometric shapes such as triangles and squares converted from handwritten data. Images are image data such as Jpeg, Png, and Tiff imported from a PC or the Internet. The data for each object is managed in association with an object ID.

[0050] The coordinates indicate the position of the start point of an object relative to a predetermined origin on the screen of the display device 2. The start point of an object is, for example, the upper left vertex of the circumscribing rectangle of a drawing object. The coordinates are expressed, for example, in pixel units of the display.

[0051] The size is the size (width and height) of each object. The end point position may also be recorded.

[0052] [Table 2] Table 2 shows detailed information about a table. Table objects are managed by associating an object ID with the table. A table has a frame identified by rows and columns. A frame is an area of ​​a table that is separated by rows and columns. Detailed information has coordinates and size for each frame. As the position of each object, such as stroke data, is known as shown in Table 1, it is possible to identify which frame each object is contained in.

[0053] A method of expanding frame 402 will be described with reference to Fig. 6. As shown in Fig. 6(a), the user handwrites "Whiteboard" in frame 402 of table 401 and removes the tip of input means 291 from the display. There is almost no space from the right end of the handwritten data "Whiteboard" (the rightmost coordinate of "-") to frame 402 in the handwriting direction (to the right).

[0054] When there is no more space for one character, the drawing object processing unit 308 expands the frame 402 in the handwriting direction so that a space for at least one character is created. The size of one character is determined according to the stroke data already handwritten in the frame 402 or the size of one character.

[0055] 6(b) shows the expanded box 402. A space 405 has been generated at the right end of the handwritten data. By expanding the box 402, a space 405 is generated that is the sum of a space determined to be smaller than one character and a space the size of one character. Therefore, the user can handwrite the next character in the space 405.

[0056] The handwriting direction is determined based on the circumscribing rectangle of the entire handwritten data ("whiteboard"). The drawing object processing unit 308 determines that the writing is horizontal if the aspect ratio of the circumscribing rectangle of the entire handwritten data within the frame indicates horizontal writing, and determines that the writing is vertical if the aspect ratio indicates vertical writing. Alternatively, the drawing object processing unit 308 may monitor the width and height of the entire handwritten data within the frame, and determine that the direction with the largest change is the handwriting direction.

[0057] The direction, whether it is from right to left, left to right, top to bottom, or bottom to top, may be determined based on the direction in which the coordinates of the handwritten data increase.

[0058] FIG. 7 is a diagram for explaining a method for determining the size of one character. The character size calculation unit 307 determines the largest width and height of the stroke data already handwritten as the width and height of one character, respectively. That is, the character size calculation unit 307 obtains a circumscribing rectangle for each stroke data and extracts the maximum width and height. When expanding the frame 402 in the horizontal direction, the drawing object processing unit 308 expands the frame 402 by at least the maximum width. When expanding the frame 402 in a vertical downward direction relative to the handwriting direction, the drawing object processing unit 308 expands the frame 402 by at least the maximum height. Note that the drawing object processing unit 308 may adopt a minimum value, an average value, or a median value instead of a maximum value.

[0059] Furthermore, the character size calculation unit 307 may focus on one character instead of one stroke data. For example, the drawing object processing unit 308 converts handwritten data into characters in real time by OCR processing. The character size calculation unit 307 obtains a circumscribing rectangle for each character and extracts the maximum width and height. The character size calculation unit 307 determines the maximum width or maximum height of one character as the size of one character. Note that various algorithms have been devised for the recognition method, but details will be omitted since known techniques can be used in this embodiment.

[0060] <Processing procedure for display device to expand frame> 8 is an example of a flowchart illustrating a procedure in which the display device 2 expands the frame 402 based on handwritten data handwritten in the frame 402. Note that this flowchart is based on the premise that the user is writing within the area surrounded by the frame 402, such as a table or a figure. If the writing is not within the area surrounded by the frame 402, the display device 2 does not need to perform the process of this flowchart, and may determine NO in step S3 described below so that subsequent processes are not executed.

[0061] When a user writes by hand, the coordinate detection unit 302 detects the coordinates touched by the input means 291. The event allocation unit 303 determines that the event is a stroke drawing event based on the sequence of coordinate points, and the stroke processing unit 306 displays the stroke data on the display (S1).

[0062] When the user removes the tip of the input means 291 from the display, the character size calculation unit 307 calculates the character size of one character (S2).

[0063] The timing when the character size calculation unit 307 calculates the character size of one character does not have to be the timing when the tip of the input means 291 is removed from the display. The character size calculation unit 307 may calculate the character size of one character in real time while the user is writing by hand.

[0064] Next, the drawing object processing unit 308 judges whether there is space within the frame 402 to write the next character (S3). In this embodiment, the character size of one character calculated in step S2 is used as a threshold for the drawing object processing unit 308 to judge whether there is space. The drawing object processing unit 308 judges that there is no space when the space at the right end of the frame 402 (the distance from the right end of the rightmost character to the frame) is smaller than the character size of one character.

[0065] The threshold value used by the drawing object processing unit 308 to determine whether or not there is a space does not need to match the character size of one character, and may be a value obtained by adding a margin to the character size of one character. The drawing object processing unit 308 can make the space larger than one character by adding a margin. The margin may be a fixed value, or may be determined by the drawing object processing unit 308 according to the character size, such as 10% of the character size.

[0066] Furthermore, the threshold for determining whether or not a space is present may be two characters or more. From the user's perspective, a space of two or more characters is secured, making it easier to concentrate on handwriting. Also, the user may be able to set the threshold (one character, two characters, etc.) for the drawing object processing unit 308 to determine whether or not a space is present.

[0067] In this embodiment, the handwriting direction is from left to right, so the rightmost space is used for the judgment, but if the user writes from right to left, the leftmost space should be used. Similarly, if the user writes from top to bottom, the bottommost space should be used. If the user writes from bottom to top, the topmost space should be used.

[0068] Even if the handwriting direction is to the right, if there is already other handwritten data to the right, the drawing object processing unit 308 may expand downward (see FIG. 9). Since the coordinates of each drawing object are stored in the page data storage unit 311, the drawing object processing unit 308 determines whether the circumscribing rectangle of the drawing object overlaps with the frame. In this case, when determining whether there is space in step S3, not only the right end but also the bottom end is determined. If there is space for one or more characters, there is no need to expand.

[0069] If the space at the right end is greater than the threshold in step S3, the drawing object processing unit 308 does nothing and returns to step S1.

[0070] If the space at the right end is equal to or smaller than the threshold in step S3, the drawing object processing unit 308 expands the frame 402 (S4). Expanding the frame 402 means creating a space for at least one character at the right end of the frame 402. In the case of a table, the drawing object processing unit 308 creates the space by expanding the column width, and in the case of a shape, the width of the shape.

[0071] Note that the drawing object processing unit 308 may expand the frame to provide space for one character, taking into account the space that exists before the frame is expanded. In other words, it is sufficient that the space after expansion is one character. The drawing object processing unit 308 expands the frame by subtracting the space that exists before the frame is expanded from the size of one character calculated in step S2.

[0072] Furthermore, the drawing object processing unit 308 may expand the frame 402 so that space for two or three characters is available instead of one character. This reduces the frequency with which the frame 402 is expanded, thereby reducing the load on the display device 2. Also, since the size of the frame 402 does not change frequently as seen by the user, it becomes easier to concentrate on handwriting. Furthermore, the user may be able to set the size (one character, two characters, etc.) that the drawing object processing unit 308 expands at one time.

[0073] Next, the drawing object processing unit 308 judges whether or not the user has finished handwriting (S5). For example, when a certain time has passed since the last handwriting, the drawing object processing unit 308 judges that handwriting has finished. This judgment method is one example, and as another example, in a display device 2 having a writing mode and other modes, it may be possible to judge that writing has finished when the user exits the writing mode by pressing a button, for example.

[0074] If it is determined in step S5 that handwriting is continuing, the process returns to step S1, and the display device 2 continues the process.

[0075] If it is determined in step S5 that handwriting has ended, the drawing object processing unit 308 reduces the frame 402 expanded in step S4 to return it to its pre-expansion state (S6). That is, the drawing object processing unit 308 reduces the frame 402 in the opposite direction by the amount of expansion in step S4. This prevents the space on the right edge that was automatically created by the expansion of the frame 402 from remaining unnecessary, and the size of the frame 402 becomes an appropriate size to accommodate the character string.

[0076] <Extend the bottom edge of the frame> As shown in Fig. 9, the drawing object processing unit 308 may expand the frame 402 so as to create space at the bottom end (vertically downward relative to the handwriting direction) rather than at the right end of the frame 402. Fig. 9 is a diagram for explaining the expansion of the bottom end of the frame.

[0077] Fig. 9(a) is the same as Fig. 6(a). There is almost no space between the right end of the handwritten data "whiteboard" and the frame 402 in the handwriting direction (to the right).

[0078] However, there are cases where the drawing object processing unit 308 cannot create a space at the right end of the handwritten data. For example, there are cases where other handwritten data 409 or a character string is already displayed. The coordinates of the other handwritten data 409, etc. are stored in the page data storage unit 311.

[0079] In such a case, the drawing object processing unit 308 expands the frame 402 so that a space at least one character high is created at the bottom end of the handwritten data. In other words, the drawing object processing unit 308 increases the height of the frame 402 by at least one character by expanding the bottom end of the frame 402. The drawing object processing unit 308 identifies the coordinates of the lowest stroke data among all stroke data within the frame, and expands the frame 402 by at least one character high from there. For example, if the bottom end of "to" in "whiteboard" is the coordinates of the lowest stroke data, the drawing object processing unit 308 expands the frame by one character from the bottom end of "to".

[0080] 9B, a space 406 is generated at the bottom of the handwritten data, so that the user can handwrite subsequent characters in the space 406 at the bottom of the handwritten data.

[0081] 10, the drawing object processing unit 308 may expand the frame 402 so as to create space on both the right end and the bottom end (vertically downward relative to the handwriting direction) of the frame 402. FIG 10 is a diagram for explaining the expansion of the right end and the bottom end of the frame.

[0082] Fig. 10(a) is the same as Fig. 6(a). There is almost no space between the right end of the handwritten data "whiteboard" and the frame 402 in the handwriting direction (to the right).

[0083] The drawing object processing unit 308 expands the frame 402 so that a space for at least one character is created at each of the right and bottom ends of the handwritten data. In other words, the drawing object processing unit 308 increases the width and line height of the frame 402 by at least one character.

[0084] 10B, a space 405 is generated at the right end of the handwritten data, and a space 406 is generated at the bottom end. Therefore, the user can handwrite the next character in the space 405 at the right end or the space 406 at the bottom end.

[0085] 10 can be used when the user wants to write by changing lines. If the user writes by hand only to the right, the drawing object processing unit 308 may delete the space at the bottom after the user finishes writing by hand.

[0086] <Delete spaces after finishing writing> 11 is a diagram for explaining the deletion of spaces after handwriting has ended. When it is determined that handwriting has ended, the drawing object processing unit 308 reduces the expanded frame 402 to return it to the size of the frame 402 before expansion.

[0087] 11(a) shows the expanded state of the frame 402. FIG 11(b) shows the frame 402 reduced to its size before expansion. In this way, the size of the frame 402 can be made appropriate to accommodate the handwritten data.

[0088] Note that the drawing object processing unit 308 may reduce the frame 402 to a size that accommodates the handwritten data, regardless of whether the frame has been expanded or regardless of the amount of expansion of the frame 402. In this case, the drawing object processing unit 308 may reduce the frame 402 to the right end of the handwritten data.

[0089] Furthermore, as shown in FIG. 11(b), when the size of the frame 402 is reduced so that the handwritten data can be accommodated, the user can expand the frame using the expand button described in the second embodiment.

[0090] 9, the drawing object processing unit 308 may not automatically reduce the expanded frame 402, but may instead reduce the frame 402 by pressing a reduce button, which will be described in the second embodiment. In this case, a space remains on the right edge, allowing the user to write additional text by hand.

[0091] <Major Effects> When the space on the right edge becomes smaller than the character size, the display device 2 of this embodiment expands the frame 402 so that there is at least space for one character, so that the user can handwrite the next character in a space larger than one character. EXAMPLES

[0092] In this embodiment, a display device 2 that expands a frame 402 when a user presses an expansion button 410 of the frame 402 will be described.

[0093] In this embodiment, it is assumed that the hardware configuration diagram shown in FIG. 4 and the functional block diagram shown in FIG. 5 described in the above embodiment can be used.

[0094] <Example of an expansion button> Fig. 12 is a diagram for explaining the frame 402 that is expanded in response to pressing the expand button 410. Fig. 12(a) is the same as Fig. 6(a). The UI image generating unit 309 of this embodiment displays the expand button 410 when the space at the right end becomes smaller than one character. Then, when the expand button 410 is pressed, the drawing object processing unit 308 expands the right end of the frame 402 so that space is created at the right end of the handwritten data (Fig. 12(b)). The expansion method may be the same as in the first embodiment.

[0095] Since the frame 402 is expanded only when the expand button 410 is pressed, the frame 402 can be expanded at the user's discretion.

[0096] As the extension button 410, the UI image generating unit 309 may display only the bottom extension button 410, or may display both the right-side extension button 410 and the bottom extension button 410.

[0097] As shown in FIG. 13, pressing a shrink button 411 causes the drawing object processing unit 308 to shrink the frame 402 .

[0098] Fig. 13 is a diagram for explaining the frame 402 that is reduced in response to pressing the reduce button 411. Fig. 13(a) is the same as Fig. 12(b), but shows a state in which a space 405 has been generated at the right end of the handwritten data.

[0099] When the frame 402 is expanded, the UI image generating unit 309 of this embodiment displays a shrink button 411. When the shrink button 411 is pressed, the drawing object processing unit 308 shrinks the frame 402 to the size before expansion. The method of reduction may be the same as in the first embodiment.

[0100] When the frame 402 is expanded to the bottom end, the UI image generating unit 309 may display a shrink button 411 for undoing the expansion at the bottom end. The UI image generating unit 309 may display both the shrink button 411 for undoing the expansion at the bottom end and the shrink button 411 for undoing the expansion at the right end.

[0101] <Processing procedure for display device to expand frame> 14 is an example of a flowchart illustrating a procedure in which the display device 2 expands the frame 402 in response to pressing of the expand button 410. Note that the explanation of FIG. 14 will mainly focus on the differences from FIG.

[0102] Steps S11, S12, and S13 are similar to steps S1, S2, and S3 in FIG. 8, respectively.

[0103] If it is determined in step S13 that the space within the frame 402 is equal to or less than the threshold, the UI image generating unit 309 displays an expansion button 410 for expanding the frame 402 (S14).

[0104] The drawing object processing unit 308 judges whether or not the Expand button 410 has been pressed by the user (S15). Since the coordinates of the input means 291 detected by the coordinate detection unit 302 overlap with the Expand button 410, the event allocation unit 303 allocates a UI operation event to the operation processing unit 304. Since the Expand button 410 has been pressed, the operation processing unit 304 requests the drawing object processing unit 308 to expand the frame 402.

[0105] If it is determined in step S15 that the expand button 410 has been pressed, the drawing object processing unit 308 expands the frame 402 (S16). The expansion process of the frame 402 in step S16 may be the same as S4 in Fig. 8. The drawing object processing unit 308 may expand the frame 402 only in the right direction (handwriting direction), or may expand the frame 402 only in the downward direction (direction perpendicular to the handwriting direction).

[0106] Furthermore, the UI image generating unit 309 erases the expand button 410 and displays a reduce button 411 for reducing the size of the frame 402 (S17).

[0107] The method of determining in step S18 whether the user has finished handwriting may be the same as in step S5 of Fig. 8. If it is determined in step S18 that handwriting is continuing, the process returns to step S11 and continues.

[0108] If it is determined in step S18 that handwriting has ended, the drawing object processing unit 308 determines whether the user has pressed the reduce button 411 for reducing the frame 402 (S19). Since the coordinates of the input means 291 detected by the coordinate detection unit 302 overlap with the reduce button 411, the event allocation unit 303 allocates a UI operation event to the operation processing unit 304. Since the reduce button 411 has been pressed, the operation processing unit 304 requests the drawing object processing unit 308 to reduce the frame 402.

[0109] If it is determined in step S19 that the reduce button 411 has not been pressed, the drawing object processing unit 308 ends the process without doing anything. In this case, the reduce button 411 may continue to be displayed, or may be erased after a certain period of time.

[0110] If it is determined in step S19 that the shrink button 411 has been pressed, the drawing object processing unit 308 performs a process of shrinking the frame 402 (S20). The process of shrinking the frame 402 in step S20 is the same as the process of shrinking in step S6 in Fig. 8, and is a process of returning the frame 402 expanded in step S1 to its original state.

[0111] 14, in order to show the flow of processing, the pressing of the expand button 410 or the reduce button 411 is determined in step S15 or step S19. However, the drawing object processing unit 308 may be configured to asynchronously determine whether the expand button 410 or the reduce button 411 is pressed by the user, and to immediately expand or reduce the frame 402 when the button is pressed.

[0112] <Major Effects> According to the display device 2 of this embodiment, the user can expand the frame by pressing the expand button 410, and can return the frame to the size before expansion by pressing the reduce button 411. EXAMPLES

[0113] In the following embodiment, another configuration example of the display device 2 will be described.

[0114] <<Another configuration example 1 of the display device>> Although the display device 2 of the present embodiment has been described as having a large touch panel, the display device 2 is not limited to having a touch panel.

[0115] Fig. 15 is a diagram showing another example of the configuration of the display device 2. In Fig. 15, a projector 451 is installed on the upper side of a normal whiteboard 452. This projector 451 corresponds to the display device 2. The normal whiteboard 452 is not a flat panel display integrated with a touch panel, but a whiteboard on which a user directly writes by hand with a marker. The whiteboard may be a blackboard, and may be any flat surface large enough to project an image.

[0116] The projector 451 has an ultra-short focal length optical system and can project an image with little distortion from a distance of about 10 cm onto the whiteboard 452. This image may be transmitted from a PC connected wirelessly or via a wire, or may be stored in the projector 451.

[0117] A user writes handwriting on the whiteboard 452 using a dedicated electronic pen 2501. The electronic pen 2501 has a light-emitting part, for example at the tip, that is switched on and emits light when the user presses it against the whiteboard 452 to write. The wavelength of the light is near-infrared or infrared, so it is invisible to the user's eyes. The projector 451 has a camera that captures the light-emitting part, analyzes the image, and identifies the direction of the electronic pen 2501. The electronic pen 2501 also emits sound waves along with the light emission, and the projector 451 calculates the distance based on the arrival time of the sound waves. The projector 451 can identify the position of the electronic pen 2501 based on the direction and distance. Handwritten data is drawn (projected) at the position of the electronic pen 2501.

[0118] Since the projector 451 projects the menu 454, when the user presses a button with the electronic pen 2501, the projector 451 identifies the position of the electronic pen 2501 and the pressed button by the ON signal of the switch. For example, when the save button 455 is pressed, the handwritten data (coordinate point sequence) handwritten by the user is saved by the projector 451. The projector 451 saves the handwritten information in a predetermined server 453 or USB memory 2600 or the like. The handwritten information is saved for each page. Since the information is saved as coordinates rather than as image data, the user can re-edit it. However, in this embodiment, the menu 454 does not need to be displayed because the operation command can be called up by handwriting. EXAMPLES

[0119] <<Another configuration example 2 of the display device>> 16 is a diagram showing another example of the configuration of the display device 2. In the example of FIG.

[0120] The terminal device 600 is connected by wire to the image projection device 700A and the pen motion detection device 810. The image projection device 700A projects image data input by the terminal device 600 onto the screen 800.

[0121] The pen operation detection device 810 communicates with the electronic pen 820 and detects the operation of the electronic pen 820 in the vicinity of the screen 800. Specifically, the electronic pen 820 detects coordinate information indicating a point pointed to by the electronic pen 820 on the screen 800 (the detection method may be the same as that in FIG. 15 ), and transmits the coordinate information to the terminal device 600.

[0122] The terminal device 600 generates image data of handwritten data input by the electronic pen 820 based on the coordinate information received from the pen operation detection device 810, and causes the image projection device 700A to draw an image of the handwritten data on the screen 800.

[0123] Furthermore, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by combining the background image projected by the image projection device 700A and the image of the handwritten data input by the electronic pen 820. EXAMPLES

[0124] <<Another configuration example 3 of the display device>> Fig. 17 is a diagram showing an example of the configuration of the display device 2. In the example of Fig. 17, the display device 2 includes a terminal device 600, a display 800A, and a pen motion detection device 810A.

[0125] The pen action detection device 810A is disposed near the display 800A, detects coordinate information indicating a point indicated by the electronic pen 820A on the display 800A (the detection method may be the same as in FIG. 15), and transmits the coordinate information to the terminal device 600. In the example of FIG. 17, the electronic pen 820A may be charged by the terminal device 600 via a USB connector.

[0126] Based on the coordinate information received from the pen operation detection device 810A, the terminal device 600 generates image data of the image of the handwritten data input by the electronic pen 820A, and displays the image data on the display 800A. EXAMPLES

[0127] <<Another Configuration Example 4 of the Display Device>> Fig. 18 is a diagram showing a configuration example of the display device 2. In the example of Fig. 18, the display device 2 includes a terminal device 600 and an image projection device 700A.

[0128] The terminal device 600 performs wireless communication (Bluetooth (registered trademark) or the like) with the electronic pen 820B to receive coordinate information of a point indicated by the electronic pen 820B on the screen 800. The coordinate information may be obtained by the electronic pen 820B reading minute position information formed on the screen 800, or the coordinate information may be received from the screen 800.

[0129] Then, based on the received coordinate information, the terminal device 600 generates image data of an image of the handwritten data input by the electronic pen 820B, and causes the image projection device 700A to project the image of the handwritten data.

[0130] Furthermore, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by combining the background image projected by the image projection device 700A and the image of the handwritten data input by the electronic pen 820B.

[0131] As described above, each of the above-described embodiments can be applied to various system configurations.

[0132] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0133] For example, in the present embodiment, input of handwritten data has been mainly described, but the drawing object processing unit 308 can perform character recognition of the handwritten data. The display device 2 may determine whether or not a space is present and may expand a frame at the timing of character recognition.

[0134] Also, in this embodiment, a display device that can be used as an electronic whiteboard has been described, but the display device may be any device capable of displaying images, such as digital signage. Also, a projector may display display data instead of a display. In this case, in this embodiment, the coordinates of the pen are detected by a method in which the coordinates of the pen tip are detected by a touch panel, but the display device 2 may detect the coordinates of the pen tip by ultrasonic waves. The pen emits light and also emits ultrasonic waves, and the display device 2 calculates the distance based on the arrival time of the ultrasonic waves. The position of the pen can be identified based on the direction and distance. The projector draws (projects) the trajectory of the pen as a stroke.

[0135] In addition, although the present embodiment has been described as an example of an electronic whiteboard, any information processing device having a touch panel can be suitably applied. A device having the same function as an electronic whiteboard is also called an electronic whiteboard, an electronic information board, an interactive board, etc. Examples of information processing devices equipped with a touch panel include output devices such as PJ (Projector), digital signage, HUD (Head Up Display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, network home appliances, notebook PCs (Personal Computers), mobile phones, smartphones, tablet terminals, game consoles, PDAs, digital cameras, wearable PCs, desktop PCs, etc.

[0136] Moreover, the configuration examples in Fig. 5 and the like are divided according to main functions in order to facilitate understanding of the processing by the display device 2. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing by the display device 2 can be divided into more processing units according to the processing contents. Also, it can be divided so that one processing unit includes more processes.

[0137] Although the display device 2 has the functions of the present embodiment by itself in the above description, the functions of the display device 2 may be provided by a server. In this case, the display device 2 and the server communicate via a network. The display device 2 displays handwritten data and transmits it to the server, and the server provides functions such as character recognition and table functions.

[0138] In addition, in this embodiment, even if a threshold is exemplified as a comparison target, the threshold is not limited to the exemplified value. Therefore, in this embodiment, for all thresholds, the descriptions "less than threshold" and "below threshold" have the same meaning, and the descriptions "exceeding threshold" and "above threshold" have the same meaning. For example, the description "less than threshold" when the threshold is 11 has the same meaning as the description "below threshold" when the threshold is 10. Furthermore, the description "exceeding threshold" when the threshold is 10 has the same meaning as the description "below threshold" when the threshold is 11.

[0139] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0140] The stroke processing unit 306 is an example of a receiving unit, the character size calculation unit 307 is an example of a calculation unit, the drawing object processing unit 308 is an example of a control unit, and the UI image generation unit 309 is an example of a display control unit. [Explanation of symbols]

[0141] 2 Display device [Prior art documents] [Patent documents]

[0142] [Patent Document 1] JP 2015-191565 A

Claims

1. an accepting means for accepting input of characters handwritten by an input means within a frame surrounded by straight lines; a calculation means for calculating a size of the handwritten character; The space between the handwritten characters and the frame in the handwriting direction where the handwritten characters are added is If the handwriting direction is horizontal, the character size is equal to or smaller than a threshold based on the width of the character size. If the handwriting direction is vertical, the character size is equal to or smaller than a threshold based on the height of the character; control means for expanding the box to accommodate handwritten characters; A display device having the above configuration.

2. 2 . The display device according to claim 1 , wherein the calculation means calculates a maximum width or a maximum height of stroke data included in the handwritten character as the size of the character.

3. The calculation means performs character recognition on the handwritten characters, 2. The display device according to claim 1, wherein the maximum width or maximum height of a single character that has been subjected to character recognition is calculated as the size of the character.

4. The display device has a display equipped with a touch panel, the display accepts input of the character based on a position touched by the input means; 4. The display device according to claim 1, wherein the calculation means calculates the size of the characters at a timing when the input means is moved away from the display.

5. 4. The display device according to claim 1, wherein the control means expands the frame in a handwritten direction.

6. 6. The display device according to claim 1, wherein the control means returns the expanded frame to its original state when handwriting within the frame is completed.

7. When another object already exists in the horizontal handwriting direction, the control means 7. The display device according to claim 1, wherein the frame is extended in a downward direction perpendicular to a handwriting direction.

8. 4. The display device according to claim 1, wherein the control means expands the frame in both a horizontal handwriting direction and a vertical downward direction relative to the handwriting direction.

9. an accepting means for accepting input of characters handwritten by an input means within a frame surrounded by straight lines; a calculation means for calculating a size of the handwritten character; The space between the handwritten characters and the frame in the handwriting direction where the handwritten characters are added is If the handwriting direction is horizontal, the character size is equal to or smaller than a threshold based on the width of the character size. If the handwriting direction is vertical, the character size is equal to or smaller than a threshold based on the height of the character; A display control means for displaying an expansion button; a control means for expanding the frame in response to pressing of the expansion button; A display device having the above configuration.

10. When the control means expands the frame in response to the pressing of the expansion button, the display control means displays a shrink button for returning the expanded frame to its original state, 10. The display device according to claim 9, wherein the control means returns the expanded frame to its original state in response to pressing of the shrink button.

11. a step of accepting an input of characters handwritten by an input means within a frame surrounded by straight lines; A calculation means calculates a size of the handwritten character; The space between the handwritten characters and the frame in the handwriting direction where the handwritten characters are added is If the handwriting direction is horizontal, the character size is equal to or smaller than a threshold based on the width of the character size. If the handwriting direction is vertical, the character size is equal to or smaller than a threshold based on the height of the character; a control means for expanding the box to accommodate a handwritten character; A display method having the following structure:

12. The display device, an accepting means for accepting input of characters handwritten by an input means within a frame surrounded by straight lines; a calculation means for calculating a size of the handwritten character; The space between the handwritten characters and the frame in the handwriting direction where the handwritten characters are added is If the handwriting direction is horizontal, the character size is equal to or smaller than a threshold based on the width of the character size. If the handwriting direction is vertical, the character size is equal to or smaller than a threshold based on the height of the character; control means for expanding said box to accommodate handwritten characters; A program to function as a

Images