Display device, method for display, and program

Abstract

To provide a display device that can display drawing data including a stored drawing component on the basis of the drawing component.SOLUTION: The display device includes: reception means for receiving an input of stroke data; storage means for storing plural pieces of stroke data divided according to a predetermined condition as a drawing component and storing drawing data including at least one drawing component; and display control means for displaying the drawing data including the stored drawing component on the basis of the input drawing component and the drawing component stored by the storage means when the reception means receives an input of the drawing component.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a display device, a display method, and a program. [Background technology]

[0002] There are known display devices such as electronic whiteboards on which a user draws strokes using an input means such as a dedicated electronic pen or a finger on a touch panel display, and handwritten data formed by several strokes is displayed. Display devices equipped with relatively large touch panels are placed in conference rooms, public facilities, etc., and are used as electronic whiteboards by multiple users.

[0003] There is known a technology for supporting handwritten input (see, for example, Patent Document 1). Patent Document 1 discloses a device that accumulates handwritten characters as stroke data, searches for stroke data that has common features with a new input, and displays the data as candidates. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has a problem in that drawing parts are not saved. In other words, in Patent Document 1, handwritten input drawing data is not divided into drawing parts and saved, so in the case of a figure with a lot of drawing data, similar candidates cannot be displayed unless a lot of handwritten drawings are made.

[0005] An object of the present invention is to provide a display device capable of displaying drawing data including a drawing component based on the stored drawing component. [Means for solving the problem]

[0006] In consideration of the above problems, the present invention provides a display device having a receiving means for receiving input of stroke data, a storage means for storing a plurality of the stroke data classified according to predetermined conditions as drawing components and for storing drawing data including one or more of the drawing components, and a display control means for, when the receiving means receives input of the drawing component, displaying the drawing data including the stored drawing components based on the input drawing components and the drawing components stored by the storage means. Effect of the Invention

[0007] It is possible to provide a display device capable of displaying drawing data including the drawing components based on the stored drawing components. [Brief description of the drawings]

[0008] [Figure 1] 11A to 11C are diagrams illustrating a flow in which the display device of the present embodiment displays an overall diagram that was handwritten by a user in the past. [Diagram 2] FIG. 1 is a schematic perspective view of a display device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device. [Figure 4] FIG. 2 is a diagram showing an example of a functional block diagram for explaining functions of the display device by dividing them into blocks. [Diagram 5] FIG. 2 is a diagram illustrating a relationship between an overall view and drawing components. [Figure 6] 13 is an example of an image diagram for explaining a method of dividing drawing components; [Figure 7] 11A and 11B are diagrams illustrating an example of corners detected in a drawing component on an electronic whiteboard on which a user has handwritten text; [Figure 8] 11 is an example of a diagram illustrating corners detected in a drawing part of a notebook PC handwritten by a user. [Figure 9] 1 is a diagram showing an example of a drawing component of an electronic whiteboard and a drawing component of a notebook PC. [Figure 10] This is an example of an overall view including the drawing components of an electronic whiteboard and a notebook PC. [Figure 11] FIG. 2 illustrates an example of a drawing part of a printer. [Figure 12] 1A and 1B are diagrams illustrating examples of drawing components of a printer and a notebook PC. [Figure 13] This is an example of an overall view including the drawing components of a printer and a notebook PC. [Figure 14] FIG. 13 is a diagram showing an example of a list dialog display. [Figure 15] FIG. 13 is an example of a sequence diagram illustrating the saving of drawing components and an overall view based on handwritten data handwritten by a user. [Figure 16] 11 is a sequence diagram showing an example of a procedure for displaying an overall view having drawing components similar to a drawing component handwritten by a user on a display device; [Figure 17] FIG. 17 is an example of a flowchart illustrating the flow of FIG. 16. [Figure 18] FIG. 13 is a diagram showing an example of a functional block diagram for explaining functions of a display device by dividing them into blocks (Example 2). [Figure 19] FIG. 13 is a sequence diagram illustrating an example of saving of drawing components and an overall view based on handwritten data handwritten by a user (Example 2); [Figure 20] 11 is a sequence diagram showing an example of a procedure in which the display device displays an overall diagram having drawing components similar to a diagram handwritten by a user (Example 2); [Figure 21] FIG. 13 is a diagram showing another configuration example of the display device. [Figure 22] FIG. 13 is a diagram showing another configuration example of the display device. [Figure 23] FIG. 13 is a diagram showing another configuration example of the display device. [Figure 24] FIG. 13 is a diagram showing another configuration example of the display device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A display device and a display method performed by the display device will be described below with reference to examples as embodiments of the present invention. EXAMPLES

[0010] <Comparative technologies> First, a comparative technique will be described for reference in explaining the display device of this embodiment. In a display device, even if a user frequently uses a figure, the user must handwrite the same figure every time. For example, there is a method in which the user saves a figure that he or she frequently uses and reuses the figure by selecting it from a saved list, but this method is troublesome in terms of the operation required for selection and the need to save the figure in advance.

[0011] <Outline of the display device of this embodiment> Therefore, in the display device of this embodiment, when a user handwrites a part of a diagram that is similar to a part of an overall diagram that the user handwrote in the past, the display device displays the saved overall diagram. The user can display the overall diagram handwritten in a previous meeting by selecting the overall diagram. This eliminates the need for the user to perform operations such as saving the overall diagram in advance or opening a list of saved overall diagrams from a menu.

[0012] In this embodiment, a diagram is an arbitrary shape formed by handwritten data. Specific examples of diagrams vary depending on what the user handwrites. An overall diagram is a diagram having multiple diagram components, and diagram components are multiple handwritten data categorized under specified conditions. Diagram components may have meaningful shapes, or may have unclear meanings depending on the category. Hereinafter, diagram components are referred to as "drawing components."

[0013] 1 is a diagram for explaining a flow in which the display device of this embodiment displays an overall diagram handwritten by a user in the past. Fig. 1(a) shows an overall diagram 121 handwritten by a user in a first meeting. When saving the overall diagram 121, the display device also saves drawing components 121a to 121e that the overall diagram 121 has.

[0014] As shown in Fig. 1(b), in the second meeting, the user starts to handwrite a drawing component 120 that is part of the overall drawing that was handwritten in the first meeting. This drawing component 120 has the shape of a PC.

[0015] The display device searches by comparing the drawing component 120 in Fig. 1(b) with the drawing components 121a to 121e saved in the first conference. The drawing component 120 is determined to be similar to the drawing component 121a. Therefore, the display device displays the overall picture 121 having the drawing component 121a of the first conference, which is similar to the drawing component 120 handwritten in the second conference, as a candidate (Fig. 1(c)). The user can select whether or not to display the overall picture 121 handwritten in the first conference.

[0016] When the user inputs an operation to display the overall diagram handwritten in the first meeting, the display device deletes the incomplete drawing component 120 and displays the overall diagram 121 (FIG. 1(d)).

[0017] In this way, the display device of this embodiment stores the overall diagram and the drawing components that the overall diagram has. After that, when the user handwrites a drawing component, the drawing component is compared with the stored drawing components, and if there is a drawing component that has a high similarity, the overall diagram that includes that drawing component is displayed. Therefore, if the user handwrites a part of the overall diagram, the user can call up and reuse the overall diagram that was handwritten in the past.

[0018] <Terminology> The input means may be any means capable of specifying coordinates on a touch panel and writing by hand, such as a pen, a human finger or hand, or a rod-shaped member.

[0019] A stroke refers to a series of operations in which a user presses an input means against a display, moves it continuously, and then releases it from the display. Stroke data is information displayed on the display based on a trajectory of coordinates input by the input means. Stroke data may be appropriately interpolated. Handwritten data is data having one or more stroke data. Handwritten input refers to handwritten data being input by a user.

[0020] An object that is displayed on a display based on stroke data is called an object. Object means a target, but in this embodiment, it means a display target. The character string converted from handwritten data through character recognition may include not only text data, but also data displayed based on user operations, such as fixed characters such as "Done," stamps displayed as marks, shapes such as circles and stars, and straight lines.

[0021] A character string is one or more character codes (fonts) that are converted from handwritten data by character recognition. A character string can be one or more characters that are handled by a computer. Characters include numbers, alphabets, and symbols. A character string is also called text data.

[0022] The whole drawing data refers to the whole of the handwritten data displayed on the screen. It does not matter whether it has meaning as a figure. The screen refers to the whole display or a defined display range. One screen may be called a page. In this embodiment, the whole drawing data is called the whole picture.

[0023] The drawing components are a plurality of stroke data that are divided into a whole drawing based on a predetermined condition, such as whether the next stroke data is to be input at a position that is a predetermined distance or more away from the last input stroke data, or whether the next stroke data is not to be input within a certain time period after the last input stroke data.

[0024] <Configuration example> 2 shows an example of a schematic perspective view of the display device 2. A user (user) uses the display device 2. In the display device 2 of this embodiment, the user can handwrite on the display 220 with an input means 291 such as a hand or an electronic pen 2500.

[0025] 2 is oriented horizontally, but it can also be oriented vertically. The user can switch between horizontal and vertical orientations by rotating the display device 2 around the center of the display 220.

[0026] <Hardware configuration example> 3 is a diagram showing an example of a hardware configuration of the display device 2. The display device 2 of this embodiment includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, a network controller 205, and an external device connection I / F 206 (Interface), and is a shared terminal for sharing information among a plurality of users.

[0027] Of these, the CPU 201 controls the overall operation of the display device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201.

[0028] The SSD 204 stores various data such as programs for the display device 2. The programs may be application programs that run on an information processing device equipped with a general-purpose OS (Windows (registered trademark), Mac OS (registered trademark), Android (registered trademark), iOS (registered trademark), etc.).

[0029] A network controller 205 controls communication with other devices via a network. An external device connection I / F 206 controls communication with a USB (Universal Serial Bus) memory 2600 and external devices (a camera 2400, a speaker 2300, and a microphone 2200).

[0030] The display device 2 also includes a capture device 211, a GPU 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication unit 219, and an antenna 219a of the short-range communication unit 219.

[0031] Of these, the capture device 211 displays video information as still images or moving images on a display 220 of the PC 10. The GPU (Graphics Processing Unit) 212 is a semiconductor chip that specializes in graphics. The display controller 213 controls and manages screen display to output an output image from the GPU 212 to the display 220 etc.

[0032] The contact sensor 214 detects that the electronic pen 2500, the user's hand H, or the like has come into contact with the display 220. When the electronic pen 2500 and the hand H are not distinguished from each other, they are referred to as an input means 291.

[0033] The sensor controller 215 controls the processing of the contact sensor 214. The contact sensor 214 inputs and detects coordinates by an infrared blocking method. The method of inputting and detecting coordinates may be a method in which two light receiving and emitting devices installed at both ends of the upper side of the display 220 emit a plurality of infrared rays parallel to the display 220, and receive light that is reflected by a reflecting member installed around the display 220 and returns along the same optical path as the optical path of the light emitted by the light receiving element.

[0034] The contact sensor 214 outputs the ID of the infrared rays radiated by the two light receiving and emitting devices that are blocked by the object to the sensor controller 215, and the sensor controller 215 specifies the coordinate position that is the contact position of the object. The electronic pen controller 216 determines whether the pen tip or the pen tail has touched the display 220 by communicating with the electronic pen 2500 via Bluetooth (registered trademark). The short-range communication unit 219 is a communication circuit such as NFC or Bluetooth (registered trademark).

[0035] Furthermore, the display device 2 includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 201.

[0036] The contact sensor 214 is not limited to an infrared blocking type, and may use various detection means such as a capacitive touch panel that identifies the contact position by detecting a change in capacitance, a resistive film touch panel that identifies the contact position by a voltage change between two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by a contacting object touching the display unit. The electronic pen controller 216 may determine the presence or absence of a touch not only on the tip and end of the electronic pen 2500, but also on the part of the electronic pen 2500 that the user holds and other parts of the electronic pen.

[0037] <About the function> Fig. 4 is a diagram showing an example of a functional block diagram for explaining functions of the display device 2 in blocks. Each functional unit of the display device 2 shown in Fig. 4 is a function or means realized by any of the components shown in Fig. 4 operating in response to an instruction from the CPU 201 in accordance with a program loaded from the SSD 204 onto the RAM 203.

[0038] The display device 2 includes a contact position detection unit 21, a drawing data generation unit 22, a drawing drawing management unit 23, a drawing part management unit 24, a drawing comparison unit 25, a display control unit 26, a conference data storage unit 27, and a network communication unit .

[0039] The contact position detection unit 21 detects the coordinate points of the position where the input means 291 touches the contact sensor 214, and accepts handwritten input by the user. Handwritten input is input using a locus of coordinates where the input means 291 touches the contact sensor 214. The contact position detection unit 21 may obtain the sequence of coordinate points as discrete values. Coordinates between the discrete values ​​are calculated by interpolation.

[0040] The drawing data generating unit 22 generates stroke data from the sequence of coordinate points detected by the contact position detecting unit 21. The drawing data generating unit 22 also generates objects such as circles, rectangles, speech bubbles, and arrows from the sequence of coordinate points found by the contact position detecting unit 21. For example, the user handwrites original shapes such as circles, rectangles, speech bubbles, and arrows by operating the input means 291. The drawing data generating unit 22 generates objects similar to the handwritten data.

[0041] The drawing management unit 23 stores data of an overall drawing handwritten by a user as a file in the overall drawing data storage unit 31, and reads out the overall drawing stored in the overall drawing data storage unit 31. The drawing management unit 23 manages the correspondence between the overall drawing and the drawing components that compose it.

[0042] The drawing component management unit 24 stores the drawing components in the component data storage unit 32 and reads out the stored drawing components from the component data storage unit 32 .

[0043] The drawing comparison unit 25 compares handwritten data handwritten by the user (if the handwritten data is similar to a saved drawing component, the handwritten data is the drawing component) with the drawing component and judges whether or not there is a drawing component similar to the handwritten data.

[0044] The display control unit 26 controls the display of handwritten data, character strings, figures, images, buttons and icons for operations, etc. on the display 220. The conference data storage unit 27 stores material data, files, etc. used in the conference in a storage medium such as the SSD 204.

[0045] The network communication unit 28 is connected to a network such as a LAN or the Internet, and transmits and receives data to and from other devices via the network.

[0046] <Overall view and drawing parts> 5 shows a schematic diagram of the relationship between the overall diagram and the drawing parts. When the conference is over, the drawing diagram management unit 23 stores the overall diagram in the overall diagram data storage unit 31. Before the conference is over, the drawing part management unit 24 classifies the overall diagram into drawing parts according to a predetermined condition and stores them in the part data storage unit 32. The drawing part management unit 24 may store the drawing parts when the conference is over.

[0047] As shown in Figure 5, the ID of the user (diagram creator) is associated with the overall diagram created by that user and saved. Also, the drawing parts included in the overall diagram are associated with the overall diagram and saved.

[0048] In FIG. 5, the overall diagram is associated with the ID of the user (diagram creator), but the user may choose whether or not to associate the overall diagram with the user.

[0049] There are various possible methods for dividing drawing components. One example is a method for dividing input stroke data using at least one of time and space.

[0050] 6 is an image diagram of a method of dividing drawing components. When the tip of input means 291 such as electronic pen 2500 touches display 220 to write, and the time without next contact after the tip is removed from display 220 exceeds a certain time, drawing component management unit 24 saves a plurality of stroke data handwritten up to that point as one drawing component. Alternatively, when the distance between two stroke data is a predetermined value or more, drawing component management unit 24 may determine that the two stroke data are divided into different drawing components.

[0051] However, since handwritten data such as characters and arrows may be included in any overall diagram, if the data is saved as a drawing component, an unintended overall diagram may be displayed every time the user handwrites characters or arrows. For this reason, it is preferable that the drawing component management unit 24 does not save as a drawing component when it determines that the data is a character or an arrow by image recognition.

[0052] <Details of drawing parts> Next, the rendering component will be described in detail with reference to Fig. 7. First, the position of a pixel on the display 220 is expressed in display coordinates with the pixel position at the upper left corner of the display area of ​​the display 220 as the origin, with the rightward direction being the positive direction of the X axis and the downward direction being the positive direction of the Y axis. Display coordinates = (number of pixels in the X-axis direction from the pixel at the origin, number of pixels in the Y-axis direction from the pixel at the origin) It is.

[0053] The process of saving stroke data of a drawing component of an electronic whiteboard drawn by a user on the display 220 of the display device 2 will be described. During a meeting or the like, a user hand-drawn a drawing component 101 of an electronic whiteboard on the display 220 of the display device 2. Fig. 7(a) shows the direction and writing order of the strokes. Stroke 2-1 and stroke 2-2 show that the input means 291 is moving while being in contact with the display 220.

[0054] The drawing part management unit 24 stores the start and end coordinates of the stroke, and calculates the distance between the stroke and other strokes. The distance is the length of the line segment that minimizes the distance between the two strokes (between two stroke data).

[0055] Then, if this distance is less than a predetermined value and the angle between the directions of the two strokes is greater than or equal to a predetermined value, the drawing component management unit 24 determines the midpoint between the two strokes (the midpoint of the line segment with the smallest distance) as a corner, and stores the coordinates of this corner in the component data storage unit 32. An angle of greater than or equal to a predetermined value may be a value at which it is determined that the user has added an angle rather than a straight line, for example, 10 to 20 degrees or more. In other words, the part where the two strokes form an angle is a corner.

[0056] For corners formed in a single stroke, such as corner B, the coordinate point where the direction of the stroke changes is assumed to be a corner, and the angle made by the stroke up to this corner and the stroke after the corner (the angle made by the direction of the stroke data before and after the point where the direction of the stroke data changes) is calculated. If the angle is equal to or greater than a threshold, the point where the change occurs is considered to be a corner.

[0057] Fig. 7(b) shows corners A to L detected from the drawing component 101 in Fig. 7(a). The corners A to L are stored as corner coordinate data in the form of a table (array) shown in Table 1. The corner coordinate data records the coordinates of the corners.

[0058] Moreover, the drawing component management unit 24 obtains the distance between each of the adjacent corners and stores it as corner distance data in a table (array) form as shown in Table 2. The corner distance data records the distance between the corners.

[0059] [Table 1]

[0060] [Table 2] The target corner pairs in Table 2 are extracted according to the following priority rules: That is, the drawing object management unit 24 preferentially extracts pairs that form a horizontal or vertical line and that form the end points of a stroke. The difference in the Y coordinate value is within a specified range and it is the start point or end point of the stroke. ⇒(A, B)(C, D)(C, E)(E, F)(F, I)(I, J)(G, H)(K, L) pairs The difference in the X-coordinate value is within a specified range and it is the start point or end point of the stroke. ⇒(A, C)(B, D)(E, G)(F, H)(I, K)(J, L) pairs

[0061] Then, the drawing component management unit 24 sorts the corners so that the Y coordinate values ​​of the corners are in ascending order (small → large), and if the difference between the Y coordinate values ​​of multiple corners is within a predetermined range, the X coordinate values ​​of the corners are in ascending order (small → large). In other words, the drawing component management unit 24 sorts the corners from the top left to the bottom right. In Table 2, the corner pairs sorted in this way are registered in order.

[0062] If the difference between the Y coordinate values ​​of three or more corners is within a certain range, the pair with the largest distance between the corners is given top priority. In Table 2, the corner pairs are arranged according to this rule.

[0063] 8 is a diagram for explaining corners detected in a notebook PC drawing component 102 handwritten by a user. The user handwritten the notebook PC drawing component 102 on the display of the display device 2. Fig. 8(a) shows the direction and stroke order of strokes. Stroke 2-1 and stroke 2-2 show that the input means 291 is moving while being in contact with the display 220.

[0064] The drawing component management unit 24 stores the start and end coordinates of a stroke, and calculates the distance between the start or end coordinates of a stroke and the start or end coordinates of another stroke. If this distance is equal to or less than a predetermined value and the angle between the directions of the two strokes is equal to or greater than a predetermined value, the drawing component management unit 24 detects the midpoint of the two coordinates whose distance is equal to or less than the predetermined value as a corner. The drawing component management unit 24 stores the coordinates of this corner in the component data storage unit 32.

[0065] 8(b) shows corners O to T detected from the drawing component 102 in FIG. 8(a). The corners O to T are stored as corner coordinate data shown in Table 3.

[0066] Moreover, the drawing component management unit 24 obtains the distance between each of the adjacent corners and stores it as corner-to-corner distance data as shown in Table 4. The method of obtaining Table 4 is the same as that of Table 2.

[0067] [Table 3]

[0068] [Table 4]

[0069] Next, as shown in Fig. 9, assume that the user hand-draws the drawing component 102 of the notebook PC in Fig. 8 (Fig. 9(b)) next to the drawing component 101 of the electronic whiteboard shown in Fig. 7 (Fig. 9(a)). In Fig. 9, the drawing component 101 of the electronic whiteboard and the drawing component 102 of the notebook PC are hand-drawn on one display. Points A to L and points O to T indicate corners.

[0070] When a user handwrites a first stroke 103 (point O is the start point and point Q is the end point) to draw the drawing component 102 of the notebook PC, the drawing component management unit 24 obtains the distance between the start point (point O) of the stroke 103 and the corner (point B) closest to point O among the corners of the previously drawn drawing component 101 of the electronic whiteboard. The drawing component management unit 24 also obtains the distance between the end point (point Q) of the stroke 103 and the corner (point D) closest to point Q among the corners of the previously drawn drawing component 101 of the electronic whiteboard. If both distances are equal to or greater than a predetermined value (i.e., the distance between the stroke data of the drawing component 101 and the stroke data of the drawing component 102 is equal to or greater than a predetermined value), the drawing component management unit 24 determines that the drawing component 101 of the electronic whiteboard and the drawing component 102 of the notebook PC are different drawing components. In other words, the drawing component management unit 24 distinguishes the drawing components 101 and 102 from multiple stroke data.

[0071] The drawing component management unit 24 stores stroke data (drawing attribute data such as coordinate point sequence, line thickness, color, etc. input from the contact position detection unit 21) of the drawing component 101 of the electronic whiteboard as a file in the component data storage unit 32. The file name is, for example, "fig_parts_1.data." The drawing component management unit 24 also assigns a drawing component ID=1 to the drawing component 101 of the electronic whiteboard.

[0072] The drawing part management unit 24 generates a drawing part management table that associates this drawing part ID, the corner coordinate data shown in Table 1, the corner distance data shown in Table 2, and the stroke data file ("fig_parts_1.data") of the drawing part 101 of the electronic whiteboard (see Table 5).

[0073] The drawing part management unit 24 stores this drawing part management table as a file in the part data storage unit 32. The file name of this drawing part management file is, for example, "fig_parts_table_1.data."

[0074] In addition, even if the next stroke is not handwritten within a certain time after the last stroke 104 (starting point is point K and ending point is point L) of the drawing component 101 on the electronic whiteboard is handwritten, the drawing component management unit 24 judges the part up to the last stroke 104 as one drawing component 101.

[0075] 9, the user has finished handwriting the drawing component 102 of the notebook PC. After the user has handwritten the last stroke 105 (starting point is point S and ending point is point T), if the user draws the next stroke at a position that is a predetermined distance or more away from the corner of the drawing component 102, or if the next stroke is not handwritten within a certain time, the drawing component management unit 24 determines that the stroke up to the last stroke 105 constitutes one drawing component 102.

[0076] Then, the drawing component management unit 24 stores the stroke data (drawing attribute data such as coordinate points, line thickness, color, etc. input from the contact position detection unit 21) of the drawing component 102 of the notebook PC as a file in the component data storage unit 32. The file name is, for example, "fig_parts_2.data."

[0077] The drawing component management unit 24 also assigns a drawing component ID of 2 to the drawing component 102 of the notebook PC. The drawing component management unit 24 generates a drawing component management table that associates the drawing component ID, the corner coordinate data shown in Table 3, the corner distance data shown in Table 4, and the stroke data file ("fig_parts_2.data") of the drawing component 102 of the notebook PC (see Table 5).

[0078] The drawing component management unit 24 adds the drawing component management table of the drawing component 102 to the drawing component management table (file name: “fig_parts_table_1.data”) generated for the drawing component 101, and overwrites and saves it in the component data storage unit 32.

[0079] The generated drawing object management table is shown in Table 5. As described above, the drawing object management table has the drawing object ID, the stroke data file name of the drawing object, the corner coordinate data, and the corner distance data for all the drawing objects in one screen (page).

[0080] [Table 5]

[0081] <Save the whole picture> Next, a method for saving the overall view will be described with reference to Fig. 10 etc. Fig. 10 shows a line 106 connecting a drawing component 101 on the electronic whiteboard and a drawing component 102 on the notebook PC. After a predetermined time has elapsed since the user hand-drawn the drawing component 102 on the notebook PC, the user hand-draws the line 106 connecting the drawing component 101 on the electronic whiteboard and the drawing component 102 on the notebook PC. Point M indicates the start point and point N indicates the end point.

[0082] After that, if the next stroke is not handwritten within a certain time, the drawing component management unit 24 judges the line 106 as one drawing component, but if there is only one stroke, the drawing component management unit 24 does not execute the file saving process. This is because one stroke is included in all drawing components, and many saved drawing components match the search.

[0083] Next, the user presses a save button for the overall diagram displayed on the display 220 of the display device 2. Saving the overall diagram is a button that causes the display device 2 to save the displayed objects (stroke data, images, figures, etc.) on a page-by-page basis, with the entire display being regarded as one page. Therefore, it also includes a button for displaying a new page. Note that only the stroke data may be saved, excluding images and figures.

[0084] The drawn figure management unit 23 saves all the stroke data shown in Fig. 10 (drawing attribute data such as coordinate point sequence, line thickness, color, etc. input from the contact position detection unit 21) as a file in the overall drawing data storage unit 31. This file name is, for example, "figure_1.data". The drawn figure management unit 23 then assigns a drawn figure ID (drawing figure ID = 1001) to this overall drawing, and generates a drawn figure management table that associates this drawn figure ID, the drawing parts management file name "fig_parts_table_1.data", and the overall drawing stroke data file name "figure_1.data".

[0085] The drawing figure management section 23 stores the data of the drawing figure management table as a file in the overall drawing data storage section 31. The file name of this drawing figure management file is, for example, "figure_table.data." The drawing figure management table is shown in Table 6.

[0086] [Table 6]

[0087] <Printer drawing parts and notebook PC drawing parts> Next, for example, in a meeting regarding a printer, the user first hand-draws a drawing component of the printer on the display 220 of the display device 2.

[0088] Fig. 11 shows a drawing component 111 of a printer. Points a to k in Fig. 11 indicate the corners of the drawing component 111, and the drawing component management unit 24 stores the corner coordinate data shown in Table 7 in the component data storage unit 32. The drawing component management unit 24 also obtains the distance between each corner and stores the corner distance data shown in Table 8 in the component data storage unit 32. Table 7 may be obtained in the same way as Tables 1 and 3, and Table 8 may be obtained in the same way as Tables 2 and 4.

[0089] [Table 7]

[0090] [Table 8]

[0091] As shown in Fig. 12, the user then hand-draws a notebook PC drawing element 112 next to the printer drawing element 111. The notebook PC drawing element 112 in Fig. 12(b) is similar to the one shown in Fig. 9(b).

[0092] When the user draws the first stroke 107 (starting at point O and ending at point Q) to draw the laptop PC drawing component 112, the drawing component management unit 24 calculates the distance between the starting point (point O) of the stroke 107 and the corner (point f) that is closest to point O among the corners of the printer drawing component 111 that was drawn earlier. The drawing component management unit 24 also calculates the distance between the ending point (point Q) of the stroke 107 and the corner (point i) that is closest to point Q among the corners of the printer drawing component 111. If both distances are equal to or greater than a predetermined value, the drawing component management unit 24 determines that the printer drawing component 111 and the laptop PC drawing component 112 are different drawings.

[0093] Then, the drawing component management unit 24 stores the stroke data of the printer's drawing component 111 (drawing attribute data such as coordinate point sequence, line thickness, color, etc. input from the contact position detection unit 21) in the component data storage unit 32 as a file. The file name is, for example, "fig_parts_3.data". The drawing component management unit 24 also assigns a drawing component ID=3 to the printer's drawing component 111. The drawing component management unit 24 generates a drawing component management table that associates the drawing component ID, the corner coordinate data shown in Table 7, the corner distance data shown in Table 8, and the stroke data file name "fig_parts_3.data" of the printer's drawing component 111.

[0094] The drawing part management unit 24 stores the data of the drawing part management table as a file in the part data storage unit 32. The file name of this drawing part management file is, for example, "fig_parts_table_2.data" (see Table 9).

[0095] Next, the user handwrites the drawing component 112 of the notebook PC. If the user draws the next stroke at a position that is a predetermined distance or more away from the corner of the drawing component 112 after drawing the last stroke 105 (starting point is point S and ending point is point T), or if the next stroke is not drawn by hand within a certain time, the drawing image management unit 23 determines that strokes 107 to 105 are one drawing component.

[0096] Then, the drawing component management unit 24 stores the stroke data (drawing attribute data such as coordinate point sequence, line thickness, color, etc. input from the contact position detection unit 21) of the drawing component 112 of the notebook PC as a file in the component data storage unit 32. The file name is, for example, "fig_parts_4.data". The drawing component management unit 24 also assigns a drawing component ID=4 to the drawing component 112 of the notebook PC. The drawing component management unit 24 generates a drawing component management table that associates the drawing component ID, the corner coordinate data shown in Table 3, the corner distance data shown in Table 4, and the stroke data file ("fig_parts_4.data") of the drawing component 112 of the notebook PC (see Table 9).

[0097] The drawing part management unit 24 adds the data of this drawing part management table to the previously generated drawing part management file (file name: "fig_parts_table_2.data") and overwrites and saves it in the part data storage unit 32. This drawing part management table is shown in Table 9.

[0098] [Table 9]

[0099] <Save the whole picture> 13, after a predetermined time has elapsed since the user hand-drawn the drawing component 112 of the notebook PC, the user hand-draws a line 108 connecting the drawing component 111 of the electronic whiteboard and the drawing component 112 of the notebook PC. Point m indicates the start point and point n indicates the end point. If the next stroke is not hand-drawn within a certain time after that, this line 108 is determined to be one drawing, but the drawn drawing management unit 23 does not execute a file saving process for the one stroke.

[0100] Then, the user presses a save button for the overall drawing displayed on the display 220 of the display device 2. The drawn drawing management unit 23 saves all the stroke data shown in Fig. 13 (drawing attribute data such as coordinate point sequence, line thickness, color, etc. input from the contact position detection unit 21) as a file in the overall drawing data storage unit 31. The file name is, for example, "figure_2.data". The drawn drawing management unit 23 then assigns a drawn drawing ID of 1002 to the overall drawing in Fig. 13. The drawn drawing management unit 23 associates the drawn drawing ID, the file name of the drawn parts management file "fig_parts_table_2.data", and the stroke data file name of the overall drawing "figure_2.data", and adds them to the already existing drawn drawing management table.

[0101] The drawing figure management unit 23 overwrites and saves the data in the drawing figure management table in the drawing figure management file (file name: "figure_table.data") of Table 6. The overwritten drawing figure management table is shown in Table 10.

[0102] [Table 10] 10 and the two drawing components 101 and 102 included in the overall drawing are associated with each other by a drawing diagram ID of 1001 in Table 10. Also, the overall drawing in FIG. 13 and the two drawing components 111 and 112 included in the overall drawing are associated with each other by a drawing diagram ID of 1002 in Table 10.

[0103] When the conference is ended (when the user presses the end conference button), the drawing diagram management unit 23 may execute the above-mentioned overall diagram saving process.

[0104] Furthermore, if the display device 2 has a page switching function, when the user performs a page switching operation, the drawn image management section 23 may execute the above-mentioned overall image saving process.

[0105] <Search for drawing parts> Next, when the user starts drawing a diagram on the display 220 of the display device 2, the drawing comparison unit 25 compares the drawing components being handwritten with the stored drawing components to determine similarity, and displays an overall diagram having similar drawing components. The process of searching for drawing components and displaying the overall diagram will be described below.

[0106] When the user handwrites another drawing part, the drawing part management unit 24 assigns a new drawing part ID (drawing part ID=5) and associates it with stroke data, corner coordinate data, and corner distance data in the same manner as described above. That is, the drawing part management unit 24 creates one row of the drawing part management table shown in Tables 5 and 9.

[0107] Then, the drawing part management unit 24 reads out the data of the drawing drawing management file (file name: "figure_table.data") stored in the overall drawing data storage unit 31 into the RAM 203. Then, the drawing part management unit 24 further reads out the data of all the drawing part management files (data of the drawing part management table) stored in the part data storage unit 32 into the RAM 203, using the file names of the drawing part management files included in the drawing drawing management file.

[0108] Next, the drawing component management unit 24 requests the drawing comparison unit 25 to compare the corner coordinate data and corner distance data of the drawing component 5 (drawing component ID=5) with the data in the drawing component management table read into the RAM 203.

[0109] Using the corner coordinate data and the corner distance data, the drawing comparison unit 25 judges the similarity between this drawing component 5 (drawing component ID=5) and the drawing components stored in the component data storage unit 32. For example, the drawing comparison unit 25 first narrows down the comparison targets based on whether the numbers of elements in the corner distance data match. The drawing comparison unit 25 may narrow down the comparison targets based on whether the numbers of elements are close, rather than just matching completely.

[0110] Next, the drawing comparison unit 25 regards the pair of corners listed in the corner distance data as the start point and end point of a vector, and calculates the cosine similarity for each vector of the two drawing components in the order listed in the corner distance data.

[0111] Cosine similarity is an index that expresses how similar two vectors are. Cosine similarity expresses the closeness of the angle between the vectors, so just like the cosine of the trigonometric function, if it is closer to 1, they are similar, and if it is closer to 0, they are dissimilar.

[0112] The rendering comparison unit 25 determines that there is similarity if the cosine similarity of all vectors in the corner distance data is equal to or greater than a threshold value (e.g., 0.7). Alternatively, the rendering comparison unit 25 may calculate the average of the cosine similarities between all corner pairs in the corner distance data and determine that there is similarity if the average value is equal to or greater than a threshold value (e.g., 0.7).

[0113] An example of determining similarity based on cosine similarity will be described below.

[0114] Tables 11 and 12 below show the data for the head parts of the corner coordinate data and corner distance data for the saved drawing component 1 (drawing component ID = 1) (the head parts of Tables 1 and 2). Tables 13 and 14 show the data for the head parts of the corner coordinate data and corner distance data for the drawing components currently handwritten by the user.

[0115] [Table 11]

[0116] [Table 12]

[0117] [Table 13]

[0118] [Table 14]

[0119] The drawing comparison unit 25 compares drawing component 5 (drawing component ID = 5) handwritten by the user with drawing component 1 (drawing component ID = 1) using Tables 11 to 14. Note that in the following, vectors are represented by enclosing two corners in brackets [ ].

[0120] The vector of the first corner pair of drawing part 1 is [AB]=(x2-x1, y2-y1), The second corner pair vector is [AC]=(x3-x1,y3-y1). The vector of the first corner pair of drawing component 5 is [αβ]=(x101-x100, y101-y100), The vector of the second corner pair is [αγ]=(x102-x100,y102-y100).

[0121] The cosine similarity between the first corner pair is given by equation (1). Similarly, the cosine similarity between the second corner pair is given by equation (2).

[0122]

number

[0123]

number

[0124] The drawing comparison unit 25 compares the drawing component 5 with drawing components (drawing component IDs=1 to 4). If the drawing component 5 is a drawing component of a notebook PC, it can be determined that the drawing component 5 is similar to the drawing components with drawing component IDs=2 and 4.

[0125] The drawing comparison unit 25 notifies the drawing component management unit 24 of the drawing component IDs 2 and 4 that it has determined to have similarity. The drawing component management unit 24 notifies the drawing drawing management unit 23 of the file name of the drawing component management file having the notified drawing component IDs.

[0126] The drawing data management unit 23 reads out the overall drawing stroke data files ("figure_1.data", "figure_2.data") associated with this file name in the drawing data management table, and the drawing data generation unit 22 generates drawing lines of the strokes according to drawing attributes such as line thickness, color, etc. The drawing data generation unit 22 then reduces this overall drawing and places it in the list dialog 130, and the display control unit 26 displays this list dialog 130 on the display 220.

[0127] When the user handwrites a drawing component on a notebook PC, the two overall views of Figures 10 and 13 match the search, so the display control unit 26 displays the two overall views in the list dialog 130. An example of this display is shown in Figure 14.

[0128] Fig. 14(a) shows a display example of the list dialog 130. In Fig. 14(a), a handwritten notebook PC drawing component 140 and the list dialog 130 are currently displayed. In addition, the list dialog 130 has two overall views 131 and 132 that have drawing components similar to the notebook PC drawing component 140.

[0129] When the user selects the overall view 131 of the electronic whiteboard and notebook PC displayed in the list dialog 130, the drawing management unit 23 closes the list dialog 130 and displays the overall view 131 of the electronic whiteboard and notebook PC (drawing ID=1001) (Figure 14(b)).

[0130] <Operation procedure of display device 2> FIG. 15 is an example of a sequence diagram illustrating the saving of drawing components and the overall view based on handwritten data handwritten by the user.

[0131] S1: When the user moves the tip of the input means 291 while in contact with the display 220, the contact position detection unit 21 accepts the input of stroke data.

[0132] S2: The contact position detection unit 21 notifies the drawing data generation unit 22 of the detected coordinate point sequence.

[0133] S3: The drawing data generating unit 22 generates handwritten data (stroke data) by interpolating or smoothing the sequence of coordinate points. The drawing data generating unit 22 notifies the display control unit 26 of the generated stroke data. The display control unit 26 displays the stroke data on the display 220.

[0134] S4: The drawing data generating unit 22 also notifies the drawing part managing unit 24 of the stroke data.

[0135] S5: The drawing object management unit 24 generates corner coordinate data in which the coordinates of the corners are registered, and corner-to-corner distance data.

[0136] S6: The drawing component management unit 24 generates a drawing component management table when the next stroke is handwritten at a position that is a predetermined distance or more away from the corner of the handwritten drawing component, or when the next stroke is not handwritten within a certain time after the last stroke is handwritten (when a predetermined condition is satisfied). The drawing component management unit 24 determines that the stroke data received by the contact position detection unit 21 from the last time the predetermined condition is satisfied until the next time the predetermined condition is satisfied, is one drawing component.

[0137] In the drawing component management table, the drawing component ID, the corner coordinate data, the corner distance data, and the stroke data file name of the drawing component are associated with each other. The drawing component management unit 24 stores this drawing component management table in the component data storage unit 32 as a file.

[0138] S7: The user presses the save button for the overall view displayed on the display 220 of the display device 2.

[0139] S8: The contact position detection unit 21 notifies the drawing management unit 23 that the save button has been pressed.

[0140] S9: The drawn drawing management unit 23 saves all stroke data of the display 220 as a file in the overall drawing data storage unit 31. The drawn drawing management unit 23 assigns a drawn drawing ID to the overall drawing, associates the drawn drawing ID, the file name of the drawing part management file, and the stroke data file name of the overall drawing, and saves them in the drawn drawing management file (if a drawn drawing management file already exists, it is overwritten).

[0141] FIG. 16 is an example of a sequence diagram showing a procedure in which the display device 2 displays an overall diagram having drawing components similar to a drawing component handwritten by the user.

[0142] S11 to S16: S11 to S16 may be similar to steps S1 to S6 in FIG.

[0143] S17: The drawing component management unit 24 requests the drawing comparison unit 25 to compare the drawing components stored in step S16 with the drawing components stored in the component data storage unit 32. The drawing comparison unit 25 returns to the drawing component management unit 24 the drawing component IDs of drawing components similar to the drawing components stored in step S16, for example, based on cosine similarity.

[0144] S18: The drawing object management unit 24 notifies the drawing object management unit 23 of the file name of the drawing object management file of the drawing object ID determined to be similar.

[0145] S19, S20: The drawing drawing management unit 23 reads out from the overall drawing data storage unit 31 a drawing drawing management file (overall drawing) having the file name of the drawing part management file.

[0146] S21: The drawing image management unit 23 notifies the drawing data generation unit 22 of the read drawing image management file.

[0147] S22: The drawing data generating unit 22 generates an overall view using the stroke data included in the drawing management file, and requests the display control unit 26 to display the list dialog 130 including the overall view. In response to this, the display control unit 26 displays on the display 220 the list dialog 130 including the overall view in a reduced state such as a thumbnail.

[0148] S23: The user presses the overall view using the input means 291.

[0149] S24: The contact position detection unit 21 identifies the selected overall view based on the coordinates of the contact with the tip of the input means 291. The contact position detection unit 21 notifies the drawing data generation unit 22 of this overall view.

[0150] S25: The drawing data generating unit 22 requests the display control unit 26 to display the overall view.

[0151] FIG. 17 is an example of a flow chart illustrating the flow of FIG.

[0152] The drawing component management unit 24 determines that the user has finished drawing a drawing component when the next stroke is handwritten at a position that is a predetermined distance or more away from the corner of the handwritten drawing component, or when the next stroke is not handwritten within a certain time after the last stroke is handwritten (S101).

[0153] Next, the drawing comparison unit 25 compares the drawing components that the user has completed drawing with the drawing components of the overall view stored in the overall view data storage unit 31 (S102).

[0154] If there is a drawing component in the overall view that is similar to the drawing component that the user has just finished drawing (Yes in S103), the display control unit 26 displays the list dialog 130 on the display 220 (S104). The list dialog 130 includes the overall view including the similar drawing components in a reduced state, such as a thumbnail.

[0155] Then, when the user presses the overall view with the input means 291 (Yes in S105), the contact position detection unit 21 accepts the selected overall view based on the coordinates where the tip of the input means 291 touches, and the display control unit 26 displays the overall view on the display 220 (S106).

[0156] <Major Effects> The display device 2 of this embodiment stores the overall diagram and the drawing parts that the overall diagram has. After that, when the user handwrites a drawing part, the drawing part is compared with the stored drawing parts, and if there is a drawing part with a high similarity, the overall diagram with the drawing part is displayed. Therefore, if the user handwrites a part of the overall diagram, the user can call up and reuse the overall diagram that was handwritten in the past. EXAMPLES

[0157] In the first embodiment, the similarity between the user's handwritten drawing and all the saved overall drawings is compared. However, if the number of saved overall drawings increases, it takes time to make the comparison, so it is preferable to narrow down the search target. Also, if the contents of the overall drawing are highly confidential, the user who saved the overall drawing may want to restrict the people who can access the overall drawing.

[0158] In this embodiment, a display device 2 that realizes this by using a user ID to search for only overview maps created by that user in the past will be described.

[0159] <About the function> 18 is an example of a functional block diagram showing functions of the display device 2 of this embodiment in the form of blocks. In the explanation of FIG. 18, differences from FIG. 4 will be mainly explained.

[0160] The display device 2 of this embodiment has a login management unit 29. The login management unit 29 performs authentication by querying an authentication database configured in the SSD 204 based on the user ID held in the user's IC card received by an IC card reader (corresponding to the short-range communication unit 219 in FIG. 3) via NFC. The drawing image management unit 23 and the drawing component management unit 24 store the drawing components and the overall image in association with the authenticated user ID.

[0161] The login management unit 29 may perform authentication based on the user ID written in the IC card, but may also perform authentication based on the user ID and password entered by the user using a hardware keyboard or a software keyboard displayed on a touch panel. Also, the login management unit 29 may perform biometric authentication, and the authentication format is not important.

[0162] Furthermore, the display device 2 may inquire about the user ID not only to the built-in SSD of the display device 2 but also to a different authentication server.

[0163] <Operation procedure> Fig. 19 is an example of a sequence diagram for explaining the saving of drawing components and an overall view based on handwritten data handwritten by a user. Note that the explanation of Fig. 19 will mainly focus on the differences from Fig. 15.

[0164] S31: The user holds the IC card over the IC card reader.

[0165] S32: The IC card reader (short-distance communication unit 219) receives the user ID from the IC card. The login management unit 29 acquires the user ID from the short-distance communication unit 219.

[0166] S33: The login management unit 29 performs user authentication based on the user ID.

[0167] S34: If the authentication is successful, the login management unit 29 notifies the drawing component management unit 24 of the user ID.

[0168] The subsequent processing is the same as that in FIG. 15, but in step S40, the drawing component management unit 24 stores the drawing component management table as a file in the component data storage unit 32 in association with the user ID.

[0169] As an example of a case where the user ID is 0001 and there are two drawing components, the drawing component management table is as shown in Table 15.

[0170] [Table 15] Compared with Table 5 or Table 9, the drawing object management table of Table 15 has an additional item of user ID. The drawing comparison unit 25 can search for drawing objects associated with the same user ID as the user ID of the currently logged-in user.

[0171] 20 is an example of a sequence diagram showing a procedure for displaying an overall diagram having drawing components similar to a diagram handwritten by a user on the display device 2. In the explanation of FIG. 20, the differences from FIG. 16 will be mainly explained.

[0172] S51 to S54: The login procedure may be the same as in FIG.

[0173] S55 to S60: Steps S55 to S60 may be similar to steps S35 to S40 in FIG.

[0174] S61: The drawing component management unit 24 requests the drawing comparison unit 25 to compare only drawing components associated with the user ID of the logged-in user with a drawing handwritten by the user.

[0175] In this way, the drawing comparison unit 25 can compare only the drawing components associated with the user ID of the logged-in user with the drawing components handwritten by the user. The display device 2 can reduce the comparison time, and can limit the users who can view the overall map even if the contents of the overall map are highly confidential.

[0176] The subsequent processing may be the same as that in FIG.

[0177] <Major Effects> In addition to the effects of the first embodiment, the display device 2 of the present embodiment can reduce the time it takes for the display device 2 to search for drawing components, and can limit the users who can view the overall diagram even if the contents of the overall diagram are highly confidential.

[0178] The display device 2 may search for drawing parts of users who belong to the same group, such as a department or team, instead of searching for drawing parts based on a user ID. Alternatively, a user may assign a security level to a drawing part, and the display device may search for drawing parts whose security level is equal to or lower than a certain level. EXAMPLES

[0179] In the following embodiment, another configuration example of the display device 2 will be described.

[0180] <<Another configuration example 1 of the display device>> Although the display device 2 of the present embodiment has been described as having a large touch panel, the display device 2 is not limited to having a touch panel.

[0181] Fig. 21 is a diagram showing another example of the configuration of the display device 2. In Fig. 21, a projector 411 is installed on the upper side of a screen 413. This projector 411 corresponds to the display device 2. The screen may be a blackboard, and may be any flat surface large enough to project an image.

[0182] The projector 411 has an ultra-short focal length optical system and can project an image with little distortion from a distance of about 10 cm onto the screen 413. This image may be transmitted from a PC connected wirelessly or via a wire, or may be stored in the projector 411.

[0183] The user writes by hand on the screen 413 using a dedicated electronic pen 2501. The electronic pen 2501 has a light-emitting part, for example at the tip, that is switched on and emits light when the user presses it against the screen 413 to write by hand. The wavelength of the light is near-infrared or infrared, so it is invisible to the user's eyes. The projector 411 has a camera that captures the light-emitting part, analyzes the image, and identifies the direction of the electronic pen 2501. The electronic pen 2501 also emits sound waves along with the light emission, and the projector 411 calculates the distance based on the arrival time of the sound waves. The projector 411 can identify the position of the electronic pen 2501 based on the direction and distance. The handwritten data is drawn (projected) at the position of the electronic pen 2501.

[0184] Since the projector 411 projects the menu 430, when the user presses a button with the electronic pen 2501, the projector 411 identifies the pressed button by the position of the electronic pen 2501 and the ON signal of the switch. For example, when the save button 431 is pressed, data handwritten by the user (sequence of coordinate points) is saved in the projector 411. The projector 411 saves the handwritten information in a predetermined server 412, USB memory 2600, or the like. EXAMPLES

[0185] <<Another configuration example 2 of the display device>> 22 is a diagram showing another example of the configuration of the display device 2. In the example of FIG.

[0186] The terminal device 600 is connected by wire to the image projection device 700A and the pen operation detection device 810. The image projection device 700A projects image data input by the terminal device 600 onto the screen 800.

[0187] The pen motion detection device 810 communicates with the electronic pen 820 and detects the motion of the electronic pen 820 in the vicinity of the screen 800. Specifically, the pen motion detection device 810 detects coordinate information indicating a point indicated by the electronic pen 820 on the screen 800 (the detection method may be the same as that in FIG. 21 ), and transmits the coordinate information to the terminal device 600.

[0188] The terminal device 600 generates image data of handwritten data input by the electronic pen 820 based on the coordinate information received from the pen operation detection device 810, and causes the image projection device 700A to draw an image of the handwritten data on the screen 800.

[0189] Furthermore, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by combining the background image projected by the image projection device 700A and the image of the handwritten data input by the electronic pen 820. EXAMPLES

[0190] <<Another configuration example 3 of the display device>> Fig. 23 is a diagram showing a configuration example of the display device 2. In the example of Fig. 23, the display device 2 includes a terminal device 600, a display 800A, and a pen motion detection device 810A.

[0191] The pen operation detection device 810A is disposed near the display 800A, detects coordinate information indicating a point indicated by the electronic pen 820A on the display 800A (the detection method may be the same as in FIG. 21), and transmits the coordinate information to the terminal device 600. In the example of FIG. 23, the electronic pen 820A may be charged by the terminal device 600 via a USB connector.

[0192] Based on the coordinate information received from the pen operation detection device 810A, the terminal device 600 generates image data of the handwritten data input by the electronic pen 820A, and displays the image data on the display 800A. EXAMPLES

[0193] <<Another Configuration Example 4 of the Display Device>> Fig. 24 is a diagram showing a configuration example of the display device 2. In the example of Fig. 24, the display device 2 includes a terminal device 600 and an image projection device 700A.

[0194] The terminal device 600 performs wireless communication (Bluetooth (registered trademark) or the like) with the electronic pen 820B to receive coordinate information of a point indicated by the electronic pen 820B on the screen 800. The coordinate information may be obtained by the electronic pen 820B reading minute position information formed on the screen 800, or the coordinate information may be received from the screen 800.

[0195] Then, the terminal device 600 generates image data of the handwritten data input by the electronic pen 820B based on the received coordinate information, and causes the image projection device 700A to project an image of the handwritten data.

[0196] Furthermore, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by combining the background image projected by the image projection device 700A and the image of the handwritten data input by the electronic pen 820B.

[0197] As described above, each of the above-described embodiments can be applied to various system configurations.

[0198] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0199] For example, in this embodiment, the drawing comparison unit 25 evaluates the similarity between the vectors of the pair of corners of the drawing component using cosine similarity, but the similarity may be obtained by pattern matching of the image data of the drawing component.

[0200] Also, in this embodiment, a display device that can be used as an electronic whiteboard has been described, but the display device may be any device capable of displaying images, such as digital signage. Also, display data from a projector may be displayed instead of a display. In this case, in this embodiment, the coordinates of the pen are detected by a method in which the coordinates of the pen tip are detected by a touch panel, but the display device 2 may detect the coordinates of the pen tip by ultrasonic waves. The pen emits light and also emits ultrasonic waves, and the display device 2 calculates the distance based on the arrival time of the ultrasonic waves. The display device 2 can identify the position of the pen based on the direction and distance. The projector draws (projects) the trajectory of the pen as a stroke.

[0201] In addition, although the present embodiment has been described as an example of an electronic whiteboard, any information processing device having a touch panel can be suitably applied. A device having the same function as an electronic whiteboard is also called an electronic whiteboard, an electronic information board, an interactive board, etc. Examples of information processing devices equipped with a touch panel include output devices such as PJ (Projector), digital signage, HUD (Head Up Display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, network home appliances, notebook PCs (Personal Computers), mobile phones, smartphones, tablet terminals, game consoles, PDAs, digital cameras, wearable PCs, desktop PCs, etc.

[0202] Moreover, the configuration example in FIG. 4 and the like is divided according to main functions in order to facilitate understanding of the processing by the display device 2. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing by the display device 2 can be divided into more processing units according to the processing contents. Also, it can be divided so that one processing unit includes more processes.

[0203] Although the display device 2 has the functions of the present embodiment by itself in the above description, the functions of the display device 2 may be provided by a server. In this case, the display device 2 and the server communicate via a network. The display device 2 displays handwritten data and transmits it to the server, and the server provides functions such as character recognition and table functions.

[0204] Furthermore, in this embodiment, even if a threshold value is exemplified as a comparison target, the threshold value is not limited to the exemplified value. Therefore, in this embodiment, for all threshold values, the terms "less than threshold value" and "below threshold value" have the same meaning, and the terms "exceeding threshold value" and "above threshold value" have the same meaning. For example, the term "less than threshold value" when the threshold value is 11 has the same meaning as the term "below threshold value" when the threshold value is 10. Furthermore, the term "exceeding threshold value" when the threshold value is 10 has the same meaning as the term "below threshold value" when the threshold value is 11.

[0205] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0206] The contact position detection unit 21 is an example of a receiving unit. The login management unit 29 is an example of an authentication unit. The drawing part management unit 24 is an example of a storage unit. The display control unit 26 is an example of a display control unit. The drawing comparison unit 25 is an example of a similarity calculation unit. [Explanation of symbols]

[0207] 2 Display device [Prior art documents] [Patent documents]

[0208] [Patent Document 1] JP 2007-265171 A

Claims

1. A receiving means for receiving input of stroke data; a storage means for storing a plurality of the stroke data classified according to a predetermined condition as drawing components and for storing drawing data including one or more of the drawing components; a display control means for displaying the drawing data including the stored drawing component based on the input drawing component and the drawing component stored by the storage means when the receiving means receives an input of the drawing component; A display device having the above configuration.

2. a similarity calculation means for calculating a similarity between the drawing component input received by the receiving means and the drawing component stored by the storage means, 2. The display device according to claim 1, wherein the display control means displays the drawing data including the drawing component whose similarity is determined to be equal to or greater than a threshold value.

3. the display control means displays a list of the drawing data including the drawing components stored based on the drawing components input by the receiving means and the drawing components stored by the storage means; 3. The display device according to claim 1, wherein the drawing data selected in the list of drawing data is displayed.

4. The predetermined condition is whether the input of the next stroke data is received at a position that is a predetermined distance or more away from the last stroke data received by the receiving means, or When the reception means does not receive the next stroke data within a certain period of time after the last stroke data is received, 4. The display device according to claim 1, wherein the storage means determines that stroke data received by the receiving means from when the predetermined condition is last satisfied until the next time the predetermined condition is satisfied, is one of the drawing components.

5. said storage means detects, when a distance between two stroke data possessed by said drawing component is equal to or smaller than a predetermined value and an angle formed by the directions of said two stroke data is equal to or larger than a threshold value, a midpoint of a line segment at which the distance between said two stroke data is the smallest as a corner, or detects, when an angle formed by directions of stroke data before and after a point at which a direction of stroke data changes is equal to or larger than a threshold value, said point at which a direction of stroke data changes as a corner, and stores coordinates of said corner of said drawing component in association with said drawing data; 3. The display device according to claim 2, wherein the similarity calculation means calculates a similarity between the drawing component whose input has been accepted by the accepting means and the drawing component stored by the storing means, based on the coordinates of the corner.

6. the similarity calculation means calculates a cosine similarity between coordinates of the adjacent corners of the drawing component whose input has been accepted by the accepting means and coordinates of the adjacent corners of the drawing component whose input has been stored by the storing means; The display device according to claim 5, characterized in that the cosine similarity is compared with a threshold value to determine whether the drawing component whose input has been accepted by the accepting means is similar to the drawing component stored by the storing means.

7. having an authentication means for authenticating a user; the storage means stores the drawing components and the drawing data in association with the user; A display device as described in any one of claims 1 to 6, wherein when the receiving means receives input of the drawing part by the user, the display means displays the drawing data including the stored drawing part based on the input drawing part and the drawing part stored by the storage means in correspondence with the user.

8. A step of receiving an input of stroke data by a receiving means; a step of storing the plurality of stroke data classified according to a predetermined condition as drawing parts, and storing drawing data including one or more of the drawing parts; a step of, when the receiving means receives an input of the drawing component, a display control means displays the drawing data including the stored drawing component based on the input drawing component and the drawing component stored by the storage means; A display method having the following structure:

9. The display device, A receiving means for receiving input of stroke data; a storage means for storing a plurality of the stroke data classified according to a predetermined condition as drawing components and for storing drawing data including one or more of the drawing components; a display control means for displaying the drawing data including the stored drawing component based on the input drawing component and the drawing component stored by the storage means when the receiving means receives an input of the drawing component; A program to function as a

Images