Display device, display method, program

Abstract

To provide a display device with which it is possible to expand a frame for characters to fit in.SOLUTION: Provided is a display device comprising: acceptance means for accepting input of characters handwritten by input means 291 in a frame 402 enclosed by a straight line; calculation means for calculating the size of the handwritten characters; and control means for expanding the frame for the handwritten characters to fit in, on the basis of the size of the characters calculated by the calculation means and a space between the frame and the handwritten characters in the handwriting direction in which the handwritten characters are added on.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a display device, a display method, and a program.

Background Art

[0002] There is known a display device that uses handwriting recognition technology to convert handwritten data into characters and display them on a display. A display device equipped with a relatively large touch panel is installed in a conference room or a public facility, and is used by multiple users as an electronic blackboard or the like.

[0003] On the display device, a user can write characters within a frame such as a figure or a table. When the frame is too small and the characters cannot fit within the frame, there is known a technique for fitting the characters into the frame (see, for example, Patent Document 1). Patent Document 1 discloses a technique for automatically reducing the characters to an appropriate size so that the entire character string fits within the frame when the handwritten characters protrude.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the conventional technology has a problem that the frame cannot be expanded. If the characters are reduced without changing the size of the frame, the characters may become difficult to see.

[0005] In view of the above problems, an object of the present invention is to provide a display device capable of expanding a frame so that characters can be entered.

Means for Solving the Problems

[0006] In view of the above problems, the present invention provides reception means for receiving input of characters handwritten by an input means within a frame, and when a space between the frame in the handwriting direction in which the handwritten characters are added and the handwritten characters is less than or equal to a threshold value based on the size of the handwritten characters, The handwriting direction is horizontal. the frame is So that handwritten text can be included. the frame is , in the direction of handwriting, or perpendicularly downward with respect to the direction of handwritingThe present invention provides a display device having an expandable control means. [Effects of the Invention]

[0007] A display device can be provided that allows the frame to be expanded to accommodate text. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram illustrates how the size of the table's borders changes. [Figure 2] This diagram illustrates the general process by which a display device expands its frame. [Figure 3] This is an example of a schematic perspective view of a display device. [Figure 4] This is an example of a hardware configuration diagram for a display device. [Figure 5] This is an example of a functional block diagram that explains the functions of a display device by dividing them into blocks. [Figure 6] This is a diagram illustrating how to expand the frame. [Figure 7] This diagram explains how to determine the size of a single character. [Figure 8] This is an example of a flowchart illustrating the procedure by which a display device expands a frame based on handwritten data written on the frame. [Figure 9] This diagram illustrates the extension of the lower edge of the frame. [Figure 10] This diagram illustrates the extension of the right and bottom edges of the frame. [Figure 11] This diagram illustrates how to remove spaces after handwriting is complete. [Figure 12] This diagram illustrates the frame that expands in response to the press of the expand button. [Figure 13] This diagram illustrates how the frame shrinks when the minimize button is pressed. [Figure 14] This is an example of a flowchart illustrating the procedure for a display device to expand its frame in response to the press of an expansion button. [Figure 15] This figure shows another example of a display device configuration. [Figure 16] It is a diagram showing another configuration example of the display device. [Figure 17] It is a diagram showing another configuration example of the display device. [Figure 18] It is a diagram showing another configuration example of the display device.

Embodiment for Carrying out the Invention

[0009] Hereinafter, as an example of an embodiment for carrying out the present invention, a display device and a display method performed by the display device will be described with reference to the drawings.

Example

[0010] <Comparative Example> First, a comparative example suitable for explaining the display device 2 of the present embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram for explaining how the size of the frame 402 of the table 401 is changed. Note that it should be noted that the comparative example is not a prior art.

[0011] As shown in FIG. 1(a), the table 401 has four frames 402. When the characters cannot fit into the frame 402 where the user is writing by the input means 291, the display device 2 expands the frame 402 in accordance with the stroke data 403 being written.

[0012] As shown in FIG. 1(b), as the coordinates of the stroke data during handwriting move to the right, the display device 2 expands the frame 402 to the right.

[0013] However, the method of expanding frame 402 according to the coordinates of handwritten stroke data can be difficult for the user to use because there is always insufficient space between the right edge of the handwritten data and frame 402. For example, if the user wants to write the next stroke data to the right of the right edge of the stroke data (such as when writing from the right edge, like "イ", or when adding punctuation marks or diacritics), there may not be enough space. This can be seen in Figure 1(c), where there is no space to start writing the next character when the user stops writing. In this case, the user needs to manually change the size of frame 402 before writing.

[0014] <Outline of the display device's operation> Figure 2 is a diagram illustrating the general process by which the display device 2 expands the frame 402. As shown in Figure 2(a), there is no longer any space to write characters at the right edge of the frame 402. When the tip of the input means 291 moves away from the display, the display device 2 determines how much space is available to write, and if it determines that there is not enough space, it expands the frame 402 to create space for about one character, as shown in Figure 2(b).

[0015] For example, the display device 2 calculates the character size of one character from the stroke data already handwritten in the frame 402. If the space at the right edge in the handwriting direction (the space between the frame in the handwriting direction and the right edge of the last handwritten character) becomes smaller than the character size, the display device 2 expands the frame 402 to the right in the handwriting direction by at least one character's worth. In this way, the user can handwrite the next character in a space 405 that is larger than one character's worth.

[0016] <About Terminology> The input method can be any means that allows handwriting by specifying coordinates on a touch panel. Examples include a pen, a person's finger or hand, or a rod-shaped object.

[0017] A stroke is a series of operations performed by a user, involving pressing an input device against the display, moving it continuously, and then lifting it from the display. Stroke data is information displayed on the screen based on the trajectory of coordinates entered by the input device. Stroke data may be interpolated as appropriate. Handwritten data is data that contains one or more stroke data. Handwritten input indicates that handwritten data is entered by the user.

[0018] The items displayed on the screen based on stroke data are called objects. While "object" generally means a target or subject, in this embodiment it refers to the object to be displayed, etc. The string converted from handwritten data using character recognition may include not only text data, but also data displayed based on user actions, such as stamps that appear as fixed characters or marks like "completed," shapes like circles and stars, and straight lines.

[0019] A string is one or more character codes (fonts) converted from handwritten data through character recognition. A string can consist of one or more characters that can be handled by a computer. These characters include numbers, letters, and symbols. A string is also called text data.

[0020] A frame is an outline enclosed by at least four straight lines. Examples of frames include each cell that makes up a table, and a rectangle, which is a type of geometric shape.

[0021] Space refers to areas, margins, or whitespace. In this embodiment, the space between the text and the frame is called space.

[0022] Expansion means making something larger by broadening its scope. Reduction means making something smaller by shrinking it.

[0023] <Example Configuration> Figure 3 shows an example of a schematic perspective view of the display device 2. The display device 2 is being used by a user. In this embodiment, the user can write on the display using their hand or an input means 291 such as an electronic pen 2500.

[0024] Although the display device 2 in Figure 3 is positioned horizontally, it can also be positioned vertically. Users can switch between horizontal and vertical orientations by rotating the display device 2 around the center of the display.

[0025] <Example Hardware Configuration> Figure 4 shows an example of the hardware configuration of the display device 2. The display device 2 in this embodiment is equipped with a CPU (Central Processing Unit) 201, ROM (Read Only Memory) 202, RAM (Random Access Memory) 203, SSD (Solid State Drive) 204, network controller 205, and external device connection I / F 206 (Interface), and is a shared terminal for sharing information among multiple users.

[0026] Of these, the CPU 201 controls the operation of the entire display device 2. The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and the IPL (Initial Program Loader). The RAM 203 is used as the work area for the CPU 201.

[0027] SSD204 stores various data, such as programs for display device 2. This program may be an application program that runs on an information processing device equipped with a general-purpose OS (such as Windows®, Mac OS®, Android®, iOS®, etc.).

[0028] The network controller 205 controls communication with the network. The external device connection interface 206 controls communication with the USB (Universal Serial Bus) memory 2600 and external devices (camera 2900, speaker 2800, microphone 2700).

[0029] The display device 2 also includes a capture device 211, a GPU 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication unit 219, and an antenna 219a for the short-range communication unit 219.

[0030] Of these, the capture device 211 displays video information as a still image or video on the PC 10's display. The GPU (Graphics Processing Unit) 212 is a semiconductor chip specializing in graphics. The display controller 213 controls and manages screen display in order to output the image from the GPU 212 to the display 220, etc.

[0031] The contact sensor 214 detects when the electronic pen 2500 or the user's hand H comes into contact with the display 220. When the electronic pen 2500 and the hand H are not distinguished, it is referred to as the input means 291.

[0032] The sensor controller 215 controls the processing of the contact sensor 214. The contact sensor 214 performs coordinate input and coordinate detection using an infrared blocking method. This method of coordinate input and coordinate detection may involve two light-emitting and receiving devices installed at both upper ends of the display 220 emitting multiple infrared rays parallel to the display 220, which are reflected by reflective members provided around the display 220, and the light received by the light-receiving element is received when it returns along the same optical path as the light emitted by the light-receiving element.

[0033] The contact sensor 214 outputs the infrared ID emitted by two light-receiving devices that are blocked by an object to the sensor controller 215, which then identifies the coordinate position of the object's contact. The electronic pen controller 216 communicates with the electronic pen 2500 to determine whether the pen tip or the pen end has touched the display 220. The short-range communication unit 219 is a communication circuit such as NFC or Bluetooth®.

[0034] Furthermore, the display device 2 is equipped with a bus line 210. The bus line 210 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 201.

[0035] Furthermore, the contact sensor 214 is not limited to the infrared blocking method. Various detection methods may be used, such as a capacitive touch panel that identifies the contact position by detecting changes in capacitance, a resistive touch panel that identifies the contact position by voltage changes between two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by contact between an object and the display. In addition, the electronic pen controller 216 may be configured to determine whether or not there is touch not only at the tip and end of the electronic pen 2500, but also at the part of the electronic pen 2500 held by the user, or other parts of the electronic pen.

[0036] <About the features> Figure 5 is a diagram showing an example of a functional block diagram that explains the functions of the display device 2 in a block-like manner. Each functional unit of the display device 2 shown in Figure 5 is a function or means realized by any of the components shown in Figure 4 operating according to instructions from the CPU 201 that follow a program deployed from the SSD 204 onto the RAM 203.

[0037] The display device 2 includes a contact detection unit 301 that detects when the input means 291 comes into contact with the display 220, and a coordinate detection unit 302 that detects the coordinates of the contact position. The contact detection unit 301 mainly transmits a signal that the tip or end of the electronic pen 2500 has been pressed. The contact detection unit 301 detects contact based on this signal.

[0038] When contact is detected by the contact detection unit 301, the coordinate detection unit 302 detects the coordinate information, and an event indicating contact is notified to the event distribution unit 303.

[0039] The event distribution unit 303 determines whether the contact is an operation of a UI control, a gesture operation, or a stroke drawing, and distributes it to the appropriate processing unit. An operation of a UI control refers to a user pressing a button or icon displayed as an operation menu. UI stands for User Interface. Gesture operations refer to selecting, moving, expanding or shrinking handwritten data, etc.

[0040] If contact with input means 291 is determined to be an operation of a UI control, an event is sent to the operation processing unit 304, which then executes processing in response to the user's request. The result of the processing is sent to the image processing unit 313 (described later) to be reflected in the screen display.

[0041] If contact with input means 291 is determined to be a gesture operation, an event is sent to gesture processing unit 305, which determines the operation associated with the gesture performed by the user and instructs operation processing unit 304 to execute the corresponding operation. Operation processing unit 304 executes the process instructed by gesture processing unit, similar to the case of UI control operations.

[0042] If contact by the input means 291 is determined to be a stroke drawing, an event is sent to the stroke processing unit 306. The stroke processing unit 306 draws handwritten data based on the coordinate points of the stroke data, deletes drawn handwritten data, or edits drawn handwritten data.

[0043] The character size calculation unit 307 calculates the size (width and height) of one stroke data or the size (width and height) of a single character in the string drawn by the stroke processing unit 306. The method for calculating the character size will be described later.

[0044] The drawing object processing unit 308 is responsible for processing the conversion and editing of drawing objects such as lines, shapes, tables, and images. For example, the drawing object processing unit 308 converts stroke data into lines, shapes, or tables, or converts it into text data using the OCR (Optical Character Recognition) function. It also performs scaling (expanding and shrinking) and transformation of drawing objects such as lines, shapes, tables, or images. The expansion and shrinking of the frame 402, described later, is performed by this drawing object processing unit 308. The drawing object processing unit 308 expands the frame according to the result of comparing the calculated character size with the space between the frame in the handwriting direction and the handwritten character.

[0045] Stroke data output by the stroke processing unit 306 and drawing objects output by the drawing object processing unit 308 are drawn on pages managed by the page processing unit 310. The page processing unit 310 manages pages, including adding and deleting pages and switching the displayed page. Information about each page, such as stroke data and drawing objects, is stored in the page data storage unit 311.

[0046] The UI image generation unit 309 generates operation menus to be displayed on the screen based on information output from the page processing unit 310 (which menus to display) and user operations. The generated operation menus are output to the display unit 312 and displayed on the display of the display device 2.

[0047] [Table 1] Table 1 schematically shows the object information stored in the page data storage unit 311. The object information is information that manages the various objects displayed by the display device 2.

[0048] • An object ID is identification information that identifies an object on the screen. An object refers to all kinds of displayed items on the screen.

[0049] The "Type" refers to the type of object. Examples include tables (rows or columns), handwritten data, characters, shapes, images, etc. A table represents a table object. Handwritten data is stroke data (a sequence of coordinate points). Characters are strings of characters (character codes) converted from handwritten data. Strings of characters are sometimes also called text data. Shapes are geometric shapes such as triangles and squares converted from handwritten data. Images are image data such as JPEG, PNG, and TIFF imported from PCs or the internet. The data for each object is managed in association with an object ID.

[0050] The coordinates indicate the position of the starting point of an object relative to a predetermined origin on the screen of the display device 2. The starting point of an object is, for example, the top-left vertex of the bounding rectangle of a drawing object. The coordinates are expressed, for example, in pixels of the display.

[0051] • The size is the dimensions (width and height) of each object. The endpoint position may also be recorded.

[0052] [Table 2] Table 2 shows detailed information about the tables. Table objects are managed by associating object IDs with the tables. Each table has a frame, which is defined by rows and columns. A frame is a region of the table separated by rows and columns. Detailed information includes coordinates and size for each frame. As shown in Table 1, the position of each object, such as stroke data, is known, so it is possible to identify which frame each object is contained within.

[0053] <Example of expanding the table frame> Referring to Figure 6, the method for expanding frame 402 will be explained. As shown in Figure 6(a), the user hand-wrote "Whitebo" in frame 402 of table 401 and lifted the tip of the input means 291 away from the display. There is almost no space between the right edge of the handwritten data "Whitebo" (the rightmost coordinate of the "-") and frame 402 in the direction of handwriting (to the right).

[0054] The drawing object processing unit 308 expands the handwriting-direction frame 402 so that at least one character's worth of space is created when there is less space than one character's worth. The size of one character is determined according to the stroke data already handwritten in the frame 402 or the size of one character.

[0055] Figure 6(b) shows the expanded frame 402. A space 405 is generated at the right edge of the handwritten data. The expansion of frame 402 creates a space 405 which is the sum of a space determined to be smaller than one character and a space equal to one character. Therefore, the user can handwrite the next character in space 405.

[0056] The handwriting direction is determined based on the bounding rectangle of the entire handwriting data ("whiteboard"). The drawing object processing unit 308 determines that the writing is horizontal if the aspect ratio of the bounding rectangle of the entire handwriting data within the frame is wider, and vertical if it is taller. Alternatively, the drawing object processing unit 308 may monitor the width and height of the entire handwriting data within the frame and determine that the greater the change, the greater the handwriting direction.

[0057] When deciding whether to write from right to left, left to right, top to bottom, or bottom to top, you can choose the direction in which the coordinates of the handwritten data increase.

[0058] Figure 7 illustrates the method for determining the size of a single character. The character size calculation unit 307 determines the width and height of a single character by using the largest width and height among the already handwritten stroke data. That is, the character size calculation unit 307 finds the bounding rectangle of each stroke data and extracts the maximum width and height. When expanding the frame 402 horizontally, the drawing object processing unit 308 expands the frame 402 by at least the maximum width. When expanding the frame 402 vertically downward with respect to the handwriting direction, the drawing object processing unit 308 expands the frame 402 by at least the maximum height. Note that the drawing object processing unit 308 may use the minimum value, average value, or median value instead of the maximum value.

[0059] Furthermore, the character size calculation unit 307 may focus on a single character rather than just one stroke data. For example, the drawing object processing unit 308 converts handwritten data into characters in real time using OCR processing. The character size calculation unit 307 finds the bounding rectangle of each character and extracts the maximum width and height. The character size calculation unit 307 sets the maximum width or maximum height of a single character to the size of one character. Although various algorithms have been devised for recognition methods, details are omitted in this embodiment as known technologies can be used.

[0060] <Processing procedure for the display device to expand its frame> Figure 8 is an example of a flowchart illustrating the procedure by which the display device 2 expands frame 402 based on handwritten data written in frame 402. This flowchart assumes that the user is writing to an area enclosed by frame 402, such as a table or a figure. If the writing is not to an area enclosed by frame 402, the display device 2 does not need to perform the processing in this flowchart, or the subsequent processing may be prevented by determining NO in step S3 described later.

[0061] When a user writes by hand, the coordinate detection unit 302 detects the coordinates that the input means 291 has contacted. The event distribution unit 303 determines that it is a stroke drawing event based on the coordinate point sequence, and the stroke processing unit 306 displays the stroke data on the display (S1).

[0062] When the user lifts the tip of the input means 291 from the display, the character size calculation unit 307 calculates the character size of a single character (S2).

[0063] Furthermore, the timing at which the character size calculation unit 307 calculates the character size of a single character does not necessarily have to be when the tip of the input means 291 is lifted from the display. The character size calculation unit 307 may calculate the character size of a single character in real time while the user is writing by hand.

[0064] Next, the drawing object processing unit 308 determines whether there is space to write the next character within the frame 402 (S3). In this embodiment, the character size of one character calculated in step S2 is used as a threshold for determining whether there is space or not in the drawing object processing unit 308. The drawing object processing unit 308 determines that there is no space if the space at the right edge of the frame 402 (the distance from the right edge of the rightmost character to the frame) is smaller than the character size of one character.

[0065] Furthermore, the threshold used by the drawing object processing unit 308 to determine the presence or absence of a space does not need to match the size of a single character; it may use a value such as the size of a single character plus a margin. By taking the margin into account, the drawing object processing unit 308 can make a space larger than the size of a single character. The margin may be a fixed value, or it may be determined by the drawing object processing unit 308 according to the character size, such as 10% of the character size.

[0066] Furthermore, the threshold for determining the presence or absence of a space may be two characters or more. From the user's perspective, two or more characters of space will be reserved, making it easier to concentrate on handwriting. In addition, the user may be able to set the threshold (one character, two characters, etc.) for the drawing object processing unit 308 to determine the presence or absence of a space.

[0067] Furthermore, in this embodiment, since the handwriting direction is from left to right, the space at the right edge is used for determination. However, if the user writes from right to left, it is better to use the space at the left edge. Similarly, if the user writes from top to bottom, it is better to use the space at the bottom edge. If the user writes from bottom to top, it is better to use the space at the top edge.

[0068] Furthermore, even if the handwriting direction is to the right, if there is already other handwritten data to the right, the drawing object processing unit 308 may expand downwards (see Figure 9). Since the coordinates of each drawing object are stored in the page data storage unit 311, the drawing object processing unit 308 determines whether the bounding rectangle of the drawing object overlaps with the frame. In this case, when determining the presence or absence of space in step S3, the space at the bottom edge as well as the right edge is also considered. If there is a space of one character or more, expansion is not necessary.

[0069] If the space at the right edge in step S3 is greater than the threshold, the drawing object processing unit 308 does nothing and returns to step S1.

[0070] If the space at the right edge in step S3 is less than or equal to a threshold, the drawing object processing unit 308 expands the frame 402 (S4). Expanding the frame 402 means creating at least one character's worth of space at the right edge of the frame 402. In the case of a table, the drawing object processing unit 308 creates space by widening the column width, and in the case of a shape, by widening the width of the shape.

[0071] The drawing object processing unit 308 may also expand the frame to create space for one character, taking into account the space that exists before the frame expansion. In other words, the space after expansion should be for one character. The drawing object processing unit 308 expands the frame by subtracting the space that exists before the frame expansion from the size of one character calculated in step S2.

[0072] Furthermore, the drawing object processing unit 308 may expand the frame 402 to provide space for two or three characters, rather than just one character. This reduces the frequency of expanding the frame 402 and lowers the load on the display device 2. Also, from the user's perspective, the size of the frame 402 does not change frequently, making it easier to concentrate on handwriting. Additionally, the user may be able to set the size (one character, two characters, etc.) that the drawing object processing unit 308 expands at once.

[0073] Next, the drawing object processing unit 308 determines whether the user has finished writing (S5). For example, if a certain amount of time has elapsed since the last writing, the drawing object processing unit 308 determines that the writing has finished. This method of determination is just one example; as another example, in a display device 2 that has a writing mode and other modes, the end of writing may be determined when the user exits writing mode by pressing a button or the like.

[0074] If it is determined in step S5 that handwriting is still in progress, the process returns to step S1, and the display device 2 continues processing.

[0075] In step S5, if it is determined that the handwriting is finished, the drawing object processing unit 308 shrinks the frame 402 that was expanded in step S4 back to its pre-expansion size (S6). That is, the drawing object processing unit 308 shrinks the frame 402 in the opposite direction by the amount it was expanded in step S4. This prevents the rightmost space that was automatically created by the expansion of the frame 402 from remaining unnecessary, and ensures that the size of the frame 402 becomes appropriate to accommodate the text.

[0076] <Expansion of the bottom edge of the frame> As shown in Figure 9, the drawing object processing unit 308 may extend the frame 402 to create space at the bottom edge (perpendicular downwards with respect to the handwriting direction) rather than at the right edge of the frame 402. Figure 9 illustrates the extension of the bottom edge of the frame.

[0077] Figure 9(a) is the same as Figure 6(a). There is almost no space from the right edge of the handwritten data "Whiteboard" to frame 402 in the handwriting direction (rightward).

[0078] However, there are cases where the drawing object processing unit 308 cannot create space at the right edge of the handwritten data. For example, other handwritten data 409 or text strings may already be displayed. The coordinates of the other handwritten data 409, etc., are stored in the page data storage unit 311.

[0079] In such cases, the drawing object processing unit 308 expands the frame 402 so that a space of at least one character height is created at the bottom edge of the handwritten data. In other words, the drawing object processing unit 308 increases the height of the frame 402 by at least one character by expanding the bottom edge of the frame 402. The drawing object processing unit 308 identifies the coordinates of the lowest stroke data among all the stroke data within the frame and expands the frame 402 by at least one character height from that point. For example, if the bottom edge of the character "ト" in "ホワイトボー" is the coordinate of the lowest stroke data, the drawing object processing unit 308 expands the frame by one character height relative to the bottom edge of "ト".

[0080] In Figure 9(b), a space 406 is generated at the bottom of the handwritten data. Therefore, the user can handwrite the next character in the space 406 at the bottom of the handwritten data.

[0081] Furthermore, as shown in Figure 10, the drawing object processing unit 308 may extend the frame 402 to create space at both the right and bottom edges (perpendicular downwards with respect to the handwriting direction). Figure 10 illustrates the extension of the right and bottom edges of the frame.

[0082] Figure 10(a) is the same as Figure 6(a). There is almost no space from the right edge of the handwritten data "Whitebo" to frame 402 in the handwriting direction (rightward).

[0083] The drawing object processing unit 308 expands the frame 402 so that at least one character's worth of space is created at the rightmost and bottommost edges of the handwritten data. In other words, the drawing object processing unit 308 increases the width and row height of the frame 402 by at least one character's worth.

[0084] In Figure 10(b), a space 405 is generated at the right edge of the handwritten data, and a space 406 is generated at the bottom edge. Therefore, the user can handwrite the next character after the space 405 at the right edge or the space 406 at the bottom edge.

[0085] The extension method shown in Figure 10 can accommodate cases where the user wants to write on a new line. If the user writes only to the right, the drawing object processing unit 308 should delete the space at the bottom after the user finishes writing.

[0086] <Deleting spaces after handwriting is finished> Figure 11 illustrates the deletion of space after handwriting is complete. When it is determined that handwriting has finished, the drawing object processing unit 308 shrinks the expanded frame 402 back to its original size.

[0087] Figure 11(a) shows frame 402 in its expanded state. Figure 11(b) shows frame 402 reduced to its original size. This allows frame 402 to be resized to an appropriate size for accommodating handwritten data.

[0088] Furthermore, the drawing object processing unit 308 may reduce the frame 402 to a size that accommodates the handwritten data, regardless of whether the frame has been expanded or the amount of expansion of the frame 402. In this case, the drawing object processing unit 308 only needs to reduce the frame 402 to the right edge of the handwritten data.

[0089] Furthermore, as shown in Figure 11(b), if the size of the frame 402 is reduced to accommodate the handwritten data, the user can expand the frame using the expansion button described in Example 2.

[0090] Furthermore, as explained in Figure 9, instead of the drawing object processing unit 308 automatically shrinking the expanded frame 402, the frame 402 may be shrunk by pressing the shrink button described in Example 2. In this case, space remains at the right edge, allowing the user to leave space and add handwritten notes.

[0091] <Main effects> In this embodiment, the display device 2 expands the frame 402 so that there is at least enough space for one character when the space at the right edge becomes smaller than the character size, allowing the user to handwrite the next character in a space larger than one character. [Examples]

[0092] In this embodiment, we will describe a display device 2 that expands the frame 402 when the user presses the expansion button 410 of the frame 402.

[0093] In this embodiment, we will use the hardware configuration diagram shown in Figure 4 and the functional block diagram shown in Figure 5, which were described in the above embodiment, as references.

[0094] <Example of an extension button> Figure 12 illustrates the frame 402 that expands in response to the press of the expansion button 410. Figure 12(a) is the same as Figure 6(a). In this embodiment, the UI image generation unit 309 displays the expansion button 410 when the space at the right edge becomes smaller than the width of one character. When the expansion button 410 is pressed, the drawing object processing unit 308 expands the right edge of the frame 402 so that space is created at the right edge of the handwritten data (Figure 12(b)). The expansion method may be the same as in Embodiment 1.

[0095] Frame 402 is only expanded when the expansion button 410 is pressed, allowing the user to expand frame 402 at their own discretion.

[0096] The UI image generation unit 309 may display the lower extension button 410 as an extension button 410, or it may display both the right extension button 410 and the lower extension button 410.

[0097] As shown in Figure 13, pressing the reduce button 411 allows the drawing object processing unit 308 to reduce the size of the frame 402.

[0098] Figure 13 illustrates the frame 402 that is reduced in size when the reduce button 411 is pressed. Figure 13(a) is the same as Figure 12(b), but shows the state in which a space 405 is generated at the right edge of the handwritten data.

[0099] In this embodiment, the UI image generation unit 309 displays a minimize button 411 when the frame 402 is expanded. When the minimize button 411 is pressed, the drawing object processing unit 308 reduces the frame 402 to its size before expansion. The reduction method may be the same as in Embodiment 1.

[0100] If the frame 402 is extended to the bottom, the UI image generation unit 309 may display a minimize button 411 to return the bottom extension to its original size. The UI image generation unit 309 may also display both a minimize button 411 to return the bottom extension to its original size and a minimize button 411 to return the right extension to its original size.

[0101] <Processing procedure for the display device to expand its frame> Figure 14 is an example of a flowchart illustrating the procedure by which the display device 2 expands the frame 402 in response to the pressing of the expansion button 410. Note that the explanation of Figure 14 mainly focuses on the differences from Figure 8.

[0102] Steps S11, S12, and S13 are the same processes as steps S1, S2, and S3 in Figure 8, respectively.

[0103] In step S13, if it is determined that the space within frame 402 is below a threshold, the UI image generation unit 309 displays an expansion button 410 for expanding frame 402 (S14).

[0104] The drawing object processing unit 308 determines whether the user has pressed the expansion button 410 (S15). The event distribution unit 303 determines that the coordinates of the input means 291 detected by the coordinate detection unit 302 are superimposed on the expansion button 410, and therefore distributes the UI operation event to the operation processing unit 304. The operation processing unit 304 determines that the expansion button 410 has been pressed and requests the drawing object processing unit 308 to expand the frame 402.

[0105] If it is determined in step S15 that the expansion button 410 has been pressed, the drawing object processing unit 308 expands the frame 402 (S16). The frame expansion process in step S16 may be the same as in S4 in Figure 8. The drawing object processing unit 308 may expand the frame 402 only to the right (handwriting direction), or it may expand the frame 402 downwards (perpendicular downwards relative to the handwriting direction).

[0106] Furthermore, the UI image generation unit 309 erases the expansion button 410 and displays a reduction button 411 for reducing the size of the frame 402 (S17).

[0107] The method for determining whether the user has finished handwriting in step S18 is the same as in step S5 in Figure 8. If it is determined in step S18 that handwriting is still ongoing, the process returns to step S11 and continues.

[0108] If it is determined in step S18 that the handwriting has finished, the drawing object processing unit 308 determines whether the user has pressed the reduce button 411 to reduce the size of the frame 402 (S19). The event distribution unit 303 determines that the coordinates of the input means 291 detected by the coordinate detection unit 302 are superimposed on the reduce button 411, and therefore distributes the UI operation event to the operation processing unit 304. The operation processing unit 304 determines that the reduce button 411 has been pressed and requests the drawing object processing unit 308 to reduce the size of the frame 402.

[0109] If it is determined in step S19 that the minimize button 411 has not been pressed, the drawing object processing unit 308 terminates processing without doing anything. In this case, the minimize button 411 may remain displayed or may disappear after a certain period of time.

[0110] In step S19, if it is determined that the reduce button 411 has been pressed, the drawing object processing unit 308 performs a reduction operation on the frame 402 (S20). The reduction operation of the frame 402 in step S20 is the same as the reduction operation in step S6 in Figure 8, and is the process of returning the frame 402 that was expanded in step S1 to its original size.

[0111] Note that in Figure 14, the pressing of the expand button 410 or the minimize button 411 is determined in steps S15 and S19 to illustrate the processing flow. However, the drawing object processing unit 308 may be configured to asynchronously determine whether the user has pressed the expand button 410 or the minimize button 411, and to immediately expand or minimize the frame 402 at the moment the button is pressed.

[0112] <Main effects> According to the display device 2 of this embodiment, the frame can be expanded when the user presses the expand button 410, and the frame can be returned to its previous size when the user presses the shrink button 411. [Examples]

[0113] In the following embodiment, another configuration example of the display device 2 will be described.

[0114] <<Another example of a display device configuration 1>> Although the display device 2 of this embodiment is described as having a large touch panel, the display device 2 is not limited to having a touch panel.

[0115] Figure 15 shows another example of the display device 2 configuration. In Figure 15, a projector 451 is installed on the top edge of a regular whiteboard 452. This projector 451 corresponds to the display device 2. A regular whiteboard 452 is not a flat panel display integrated with a touch panel, but a whiteboard on which the user directly writes with a marker. Note that the whiteboard can also be a blackboard; any flat surface large enough to project an image will suffice.

[0116] The projector 451 has an ultra-short-throw optical system and can project a low-distortion image onto the whiteboard 452 from a distance of about 10 cm. This image may be transmitted from a PC connected wirelessly or via a wired connection, or it may be stored in the projector 451's memory.

[0117] The user writes on the whiteboard 452 using a dedicated electronic pen 2501. The electronic pen 2501 has a light-emitting part, for example at its tip, which switches on and emits light when the user presses it against the whiteboard 452 to write. The wavelength of the light is near-infrared or infrared, so it is invisible to the user's eye. The projector 451 has a camera that captures an image of the light-emitting part and analyzes the image to determine the direction of the electronic pen 2501. In addition, the electronic pen 2501 emits sound waves along with the light, and the projector 451 calculates the distance based on the arrival time of the sound waves. The projector 451 can determine the position of the electronic pen 2501 based on its direction and distance. The handwritten data is drawn (projected) at the position of the electronic pen 2501.

[0118] The projector 451 projects the menu 454, so when a user presses a button with the electronic pen 2501, the projector 451 identifies the pressed button based on the position of the electronic pen 2501 and the ON signal of the switch. For example, when the save button 455 is pressed, the handwritten data (sequence of coordinate points) written by the user is saved by the projector 451. The projector 451 saves the handwritten information to a predetermined server 453 or USB memory 2600, etc. The handwritten information is saved page by page. Since it is saved as coordinates and not as image data, the user can re-edit it. However, in this embodiment, since operation commands can be called up by handwriting, the menu 454 does not need to be displayed. [Examples]

[0119] <<Another example of a display device configuration 2>> Figure 16 shows another example configuration of the display device 2. In the example in Figure 16, the display device 2 includes a terminal device 600, an image projection device 700A, and a pen motion detection device 810.

[0120] The terminal device 600 is connected by wires to the image projection device 700A and the pen motion detection device 810. The image projection device 700A projects the image data input by the terminal device 600 onto the screen 800.

[0121] The pen motion detection device 810 communicates with the electronic pen 820 and detects the operation of the electronic pen 820 in the vicinity of the screen 800. Specifically, the electronic pen 820 detects coordinate information indicating the point it is pointing to on the screen 800 (the detection method may be the same as in Figure 15) and transmits it to the terminal device 600.

[0122] The terminal device 600 generates image data of the handwritten data input by the electronic pen 820 based on coordinate information received from the pen motion detection device 810, and causes the image projection device 700A to draw the image of the handwritten data on the screen 800.

[0123] Furthermore, the terminal device 600 generates superimposed image data, which is a composite image obtained by combining the background image projected onto the image projection device 700A with the image of the handwritten data input by the electronic pen 820. [Examples]

[0124] <<Another example of a display device configuration 3>> Figure 17 shows an example configuration of the display device 2. In the example in Figure 17, the display device 2 includes a terminal device 600, a display 800A, and a pen motion detection device 810A.

[0125] The pen motion detection device 810A is positioned near the display 800A and detects coordinate information indicating the point pointed to by the electronic pen 820A on the display 800A (the detection method may be the same as in Figure 15), and transmits it to the terminal device 600. In the example of Figure 17, the electronic pen 820A may be charged by the terminal device 600 via a USB connector.

[0126] The terminal device 600 generates image data of the handwritten data input by the electronic pen 820A based on the coordinate information received from the pen motion detection device 810A, and displays it on the display 800A. [Examples]

[0127] <<Another example of a display device configuration 4>> Figure 18 shows an example configuration of the display device 2. In the example shown in Figure 18, the display device 2 includes a terminal device 600 and an image projection device 700A.

[0128] The terminal device 600 communicates wirelessly with the electronic pen 820B (using Bluetooth®, etc.) to receive coordinate information of a point indicated by the electronic pen 820B on the screen 800. The coordinate information may be obtained by the electronic pen 820B reading minute positional information formed on the screen 800, or by receiving coordinate information from the screen 800.

[0129] Then, the terminal device 600 generates image data of the handwritten data input by the electronic pen 820B based on the received coordinate information, and projects the image of the handwritten data onto the image projection device 700A.

[0130] Furthermore, the terminal device 600 generates superimposed image data, which is a composite image of the background image projected onto the image projection device 700A and the handwritten data image input by the electronic pen 820B.

[0131] As described above, each of the embodiments described can be applied to various system configurations.

[0132] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0133] For example, although this embodiment mainly describes the input of handwritten data, the drawing object processing unit 308 can recognize handwritten data as characters. The display device 2 may also check for the presence or absence of spaces or expand the frame at the time of character recognition.

[0134] Furthermore, although this embodiment describes a display device that can be used as an electronic whiteboard, the display device only needs to be able to display images, and could be, for example, a digital signage device. Alternatively, a projector could be used to display the data instead of a display. In this case, although the coordinates of the pen tip were detected by detecting the coordinates of the pen tip on a touch panel in this embodiment, the display device 2 may detect the coordinates of the pen tip using ultrasound. The pen emits ultrasound along with light emission, and the display device 2 calculates the distance based on the arrival time of the ultrasound. The position of the pen can be determined by its direction and distance. The projector draws (projects) the trajectory of the pen as a stroke.

[0135] Furthermore, although an electronic blackboard was described as an example in this embodiment, any information processing device having a touch panel can be suitably applied. Devices having similar functions to an electronic blackboard are also called electronic whiteboards, electronic information boards, interactive boards, etc. Examples of information processing devices equipped with a touch panel include projectors (PJ), output devices such as digital signage, head-up displays (HUD), industrial machinery, imaging devices, sound collection devices, medical equipment, networked home appliances, personal computers (notebook PCs), mobile phones, smartphones, tablet terminals, game consoles, PDAs, digital cameras, wearable PCs, or desktop PCs.

[0136] Furthermore, the configuration examples shown in Figure 5 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the display device 2. The present invention is not limited by the way the processing units are divided or their names. The processing of the display device 2 can be further divided into many more processing units depending on the processing content. It can also be divided so that one processing unit includes even more processing.

[0137] Furthermore, although the description has focused on the case where the display device 2 alone possesses the functions of this embodiment, the server may also provide the functions of the display device 2. In this case, the display device 2 and the server communicate via a network. The display device 2 displays the handwritten data and transmits it to the server, and the server provides functions such as character recognition and table functions.

[0138] Furthermore, even if a threshold is given as an example for comparison in this embodiment, the threshold is not limited to the example value. For this reason, in this embodiment, for all thresholds, the descriptions "less than the threshold" and "less than or equal to the threshold" have the same meaning, and the descriptions "greater than the threshold" and "greater than or equal to the threshold" have the same meaning. For example, if the threshold is 11, the description "less than the threshold" has the same meaning as if the threshold is 10 and therefore less than or equal to the threshold. Also, if the threshold is 10, the description "greater than the threshold" has the same meaning as if the threshold is 11 and therefore greater than or equal to the threshold.

[0139] Furthermore, each function of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0140] Note that the stroke processing unit 306 is an example of a receiving means. The character size calculation unit 307 is an example of a calculation means. The drawing object processing unit 308 is an example of a control means. The UI image generation unit 309 is an example of a display control means. [Explanation of Symbols]

[0141] 2 Display device [Prior art documents] [Patent Documents]

[0142] [Patent Document 1] Japanese Patent Publication No. 2015-191565

Claims

1. A receiving means that accepts input of characters handwritten within a frame using an input means, with the handwriting direction being horizontal. A control means expands the frame in the handwriting direction, or vertically downward with respect to the handwriting direction, to accommodate the handwritten characters if the space between the handwritten character and the frame in the handwriting direction, is less than or equal to a threshold based on the size of the handwritten character, A display device.

2. The system further includes a calculation means for calculating the size of the handwritten characters, The aforementioned display device has a display equipped with a touch panel, The display accepts the input of the characters based on the position touched by the input means. The display device according to claim 1, characterized in that the calculation means calculates the size of the characters at the timing when the input means moves away from the display.

3. The display device according to claim 1 or 2, characterized in that the control means returns the expanded frame to its original state when handwriting within the frame is completed.

4. The control means, when another object already exists in the direction of horizontal handwriting, The display device according to any one of claims 1 to 3, characterized in that the frame is extended in a direction perpendicular to the handwriting direction.

5. The display device according to any one of claims 1 to 4, characterized in that the control means expands the frame in both the horizontal handwriting direction and the vertical downward direction relative to the handwriting direction.

6. A receiving means that accepts input of characters handwritten within a frame using an input means, with the handwriting direction being horizontal. A display control means that displays an expansion button when the space between the handwritten character and the frame in the handwriting direction where the handwritten character is added is less than or equal to a threshold based on the size of the handwritten character, A control means that, in response to the pressing of the expansion button, expands the frame in the direction of handwriting, or perpendicularly downward with respect to the direction of handwriting, so that handwritten characters can be placed inside. A display device.

7. When the control means expands the frame in response to the press of the expansion button, the display control means displays a shrink button to return the expanded frame to its original size. The display device according to claim 6, characterized in that the control means returns the expanded frame to its original state in response to the pressing of the reduce button.

8. The reception method includes a step of accepting input of characters handwritten within a frame using an input method, with the handwriting direction being horizontal, The control means, if the space between the frame in the handwriting direction on which the handwritten characters are added and the handwritten characters is less than or equal to a threshold based on the size of the handwritten characters, expands the frame in the handwriting direction or perpendicularly downward with respect to the handwriting direction to accommodate the handwritten characters. A method of displaying the information.

9. Display device, A receiving means that accepts input of characters handwritten within a frame using an input means, with the handwriting direction being horizontal. If the space between the handwritten character and the frame in the handwriting direction to which the handwritten character is added is less than or equal to a threshold based on the size of the handwritten character, a control means expands the frame in the handwriting direction or vertically downward with respect to the handwriting direction to accommodate the handwritten character. A program designed to function as such.

Images