Display device, program, display method, display system

Abstract

To provide a display device that can display a display region with a specific object remaining in the display region when the display region has been moved.SOLUTION: The present invention relates to a display device for displaying a display region as a part of an input-possible region where handwriting is possible. The display device includes: selection receiving means for receiving a selection of an object displayed in the display region; object management means for moving the object in response to an operation of moving the display region; and display control means for displaying the moved object in the moved display region.SELECTED DRAWING: Figure 3

Description

Technical Field

[0004] , , , , , , , , ,

[0005]

[0001] The present invention relates to a display device, a program, a display method, and a display system.

Background Art

[0002] There is known a display device such as an electronic blackboard that displays handwritten data formed by several strokes when a user draws a stroke on a touch panel type display with an input means such as a dedicated electronic pen or a finger. A display device equipped with a relatively large touch panel is arranged in a conference room or a public facility and used as an electronic blackboard by a plurality of users.

[0003] Techniques for effectively utilizing the limited display area of a display device have been devised (for example, see Patent Document 1). Patent Document 1 discloses a navigation device that allows a user to move an object to be displayed in a display area while an operation menu is displayed in the display area.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the conventional technology has a problem that when the display area is moved, a specific object cannot be left and displayed in the display area. For example, in the case of a display device having an inputable area wider than the display area, the user can move the display area and write by hand in the space created at the destination. However, when there are areas with handwritten data such as an agenda and areas with detailed contents of the agenda that are separated from each other, only either the handwritten data or the detailed contents of the agenda can be displayed in the display area, and the user has to alternately display the two areas.

[0005] In view of the above problems, an object of the present invention is to provide a display device that can leave and display a specific object in the display area when the display area is moved.

Means for Solving the Problems

[0006] In view of the above problems, the present invention is This is the entire range in which the user can hand-draw objects. Inputable area Our some It is a domain. A display device that displays a display area, A display control means for displaying the display area located at a first position within the inputable area on a screen, The aforementioned display area Display control means that displays one or more objects handwritten by the user within the display area of ​​the screen, and when one or more of the displayed objects are moved in response to an operation to move the display area, the display control means that the position of the display area after the move is the same as the display area before the move. Displayed can A selection acceptance means that accepts the selection of an object, The display control means has, The aforementioned display area From the first position to the second position move Let Depending on the operation, The display area is moved to the second position and displayed, and the display device further, in accordance with the operation, selected The aforementioned object The position of the input-enabled area is moved to a position corresponding to the same position as the display area before the movement of the display area after moving from the first position to the second position. move Let It has object management means, The display control means causes the moved object to be displayed in the same position as the display area before the movement, after the movement from the first position to the second position. to special To be a sign To provide a display device. [Effects of the Invention]

[0007] This device can provide a display that allows specific objects to remain in the display area even when the display area is moved. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram illustrates the relationship between the input area and the display area. [Figure 2] This is a diagram illustrating the relationships between layers. [Figure 3] This figure shows an example of how objects pinned to a pinned layer are displayed. [Figure 4] This is an example of a schematic perspective view of a display device. [Figure 5] This figure shows an example of the hardware configuration of a display device. [Figure 6] This figure shows an example of a functional block diagram that explains the functions of a display device by dividing them into blocks. [Figure 7] This diagram illustrates the coordinates of objects managed by the drawing object management unit. [Figure 8] This diagram illustrates the layers used by the display control unit for display. [Figure 9]A diagram showing an example of the display of an object fixed to the pinning layer. [Figure 10] An example of a flowchart showing the procedure for the display device to display an object fixed to the display area (Example 1). [Figure 11] A diagram showing an example of the display of an object fixed to the pinning layer. [Figure 12] An example of a flowchart showing the procedure for the display device to display an object fixed to the display area (Example 2). [Figure 13] A diagram showing an example of the display of an object fixed to the pinning layer. [Figure 14] An example of a flowchart showing the procedure for the display device to display an object fixed to the display area (Example 3). [Figure 15] A diagram for explaining the menu layer. [Figure 16] A diagram showing an example of the display of an object fixed to the pinning layer by pressing the pinning button. [Figure 17] An example of a flowchart showing the procedure for the display device to display an object fixed to the display area (Example 4). [Figure 18] A diagram showing an example of the display of an object fixed to the pinning layer by being surrounded by a stroke. [Figure 19] An example of a flowchart showing the procedure for the display device to display an object fixed to the display area (Example 5). [Figure 20] A diagram for explaining an example of the display of an object at the end of pinning. [Figure 21] An example of a flowchart showing the procedure for the display device to release an object fixed to the display area (Example 6). [Figure 22] A diagram showing an example of the display of an object fixed to the pinning layer (Example 7). [Figure 23]This is an example of a flowchart illustrating the procedure for a display device to display an object fixed within its display area (Example 7). [Figure 24] This figure shows an example of how objects fixed to a pinned layer are displayed (Example 8). [Figure 25] This is an example of a flowchart illustrating the procedure for a display device to display an object fixed within its display area (Example 8). [Figure 26] This figure shows an example of how objects fixed to a pinned layer are displayed (Example 9). [Figure 27] This is an example of a flowchart illustrating the procedure for a display device to display an object fixed within its display area (Example 9). [Figure 28] This is an example of a system configuration diagram for a display system. [Figure 29] This is an example of a functional block diagram that shows the functions of a display device in a block-like manner. [Figure 30] This is an example of an object that a user at site A has handwritten on a display device. [Figure 31] This figure shows the display area where the pin button is displayed. [Figure 32] This diagram shows the display area created when the user moves the "list of agenda items". [Figure 33] This is an example of a sequence diagram illustrating the procedure for sharing object pinning operations between display devices 2A and 2B. [Figure 34] This figure shows a display area where a string of characters related to the timetable (object D) has been handwritten. [Figure 35] This is an example of a system configuration diagram for a display system. [Figure 36] This figure shows the hardware configuration of an example server device. [Figure 37] This is an example of a functional block diagram that explains the functions of a display device by dividing them into blocks. [Figure 38] This is an example of a functional block diagram that explains the functions of a server device by dividing them into blocks. [Figure 39A] This is an example of a sequence diagram illustrating the procedure for display devices 2A and 2B to conduct a remote conference via a server device. [Figure 39B] This is an example of a sequence diagram illustrating the procedure for display devices 2A and 2B to conduct a remote conference via a server device. [Figure 40] This figure shows an example of a message displayed on the display device's screen. [Figure 41] This figure shows another example of a display device configuration. [Figure 42] This figure shows another example of a display device configuration. [Figure 43] This figure shows another example of a display device configuration. [Figure 44] This figure shows another example of a display device configuration. [Modes for carrying out the invention]

[0009] Hereinafter, as an example of an embodiment for carrying out the present invention, a display device and a display method in the display device will be described with reference to examples. [Examples]

[0010] <Regarding comparative technologies> First, to explain the display device of this embodiment, we will describe some comparative technologies that may be helpful. A display device having an input area for handwriting input that is larger than the display area displays a portion of the input area in the display area according to the user's operation.

[0011] Figure 1 illustrates the relationship between the input area 52 and the display area 51. Figure 1(a) shows an example of the input area 52 of a display device. The input area 52 is the entire area in which the user can handwrite or place objects. Some display devices have an infinite input area 52. The display area 51 is a portion of the input area 52 that the display device is currently displaying on the screen. If the entire screen of the display is the display area 51, then the display area 51 is the same as the size of the display.

[0012] Figure 1(c) shows the display area 51 in Figure 1(a) as it is actually displayed on the screen.

[0013] When the user moves the display area 51 from the state shown in Figure 1(a) to the state shown in Figure 1(b) using a swipe gesture, the display will show the result as shown in Figure 1(d). Swiping refers to the action of sliding a finger up, down, left, or right while keeping it pressed down on the touch panel.

[0014] Thus, since only a portion of the input area 52 is displayed in the display area 51, swiping to display an object that the user wants to refer to may interrupt the discussion.

[0015] <Outline of the display device in this embodiment> Therefore, in this embodiment, a transparent layer (pinning layer) is used for pinning objects.

[0016] Figure 2 illustrates the relationship between layers. The display device has a drawing layer 54 and a pinning layer 53, starting from the back in the line of sight. Drawing layer 54 is the layer that contains all objects. • Pinned layer 53 is always positioned in front of the viewer's line of sight, ahead of drawing layer 54. Pinned layer 53 is a layer fixed to the display area 51. This means that even if the display area 51 moves, objects pinned to pinned layer 53 will always be visible.

[0017] Drawing layer 54 represents each pixel using RGB values. Pinned layer 53 temporarily stores objects from drawing layer 54, and pixels other than objects are transparent (value is zero). Objects in pinned layer 53 overwrite the pixels in drawing layer 54 and are displayed in display area 51.

[0018] For example, the following actions allow a user to pin an object. (i) The user long-presses the object they want to pin using an input method and then presses the pin button that appears. (ii) When a user handwrites a designated mark and then encloses an area with the handwritten data, the enclosed area can be pinned.

[0019] Users can also unpin objects and areas they have previously pinned.

[0020] Refer to Figure 3 to explain how to pin objects. Figure 3 shows an example of how objects pinned to pinning layer 53 are displayed.

[0021] Figure 3(a) shows objects A and B in the input area 52. The area shown by the dotted line is the display area 51. The coordinates of the top-left corner of the display area 51 are set as the origin (0,0), the top-left coordinates of the object are set as the starting coordinates, and the bottom-right coordinates are set as the ending coordinates.

[0022] When the user long-presses object B, the display device selects the table and displays a pin button 9 for pinning (Figure 3(b)).

[0023] When the user presses the pin button 9, the display device deletes object B from drawing layer 54 and adds object B to pin layer 53.

[0024] Next, the user hand-writes object C in the space to write meeting minutes about the agenda item "Equipment Reservation" (Figure 3(c)).

[0025] The user attempts to write meeting minutes about the agenda item "Determining the Timetable," but there is no space, so they swipe the display area 51 to the upper left. Object B's coordinates do not change relative to the display area 51, so its display position remains unchanged. In other words, in the moved display area 51, object B is displayed at the same position as object B was in the display area 51 when the selection of object B was accepted.

[0026] When the display area 51 is moved to the destination of the input area 52, space becomes available to the right of object B (Figure 3(d)). Therefore, the user can handwrite meeting minutes about the agenda item "Determining the Timetable".

[0027] Thus, the display device of this embodiment can always display the pinned object in the display area 51.

[0028] <About Terminology> The input method can be any means that allows handwriting by specifying coordinates on a touch panel. Examples include an electronic pen, a person's finger or hand, or a rod-shaped object.

[0029] A stroke is a series of operations performed by a user, involving pressing an input device against the display, moving it continuously, and then lifting it from the display. Stroke data is information displayed on the screen based on the trajectory of coordinates entered by the input device. Stroke data may be interpolated as appropriate. Handwritten data is data that contains one or more stroke data. Handwritten input indicates that handwritten data is entered by the user.

[0030] The strings converted from handwritten data using character recognition may include not only text data but also data displayed based on user actions, such as stamps that appear as fixed characters or marks like "completed," shapes like circles, rectangles, lines, and stars, speech bubbles, and sticky notes. A string is one or more characters handled by a computer. The actual entity of a string is a character code. Characters include numbers, alphabets, and symbols. A string is also called text data.

[0031] The items displayed on the screen are called objects. While "object" generally means a target or subject, in this embodiment it refers to the object being displayed. In this embodiment, handwritten data, text strings, shapes (circles, rectangles, arrows, etc.), stamps, speech bubbles, sticky notes, and images are also considered objects.

[0032] Display components are buttons and icons that appear on the screen. For example, the pin button 9 is a display component.

[0033] <Example Configuration> Figure 4 shows an example of a schematic perspective view of the display device 2. The display device 2 is being used by a user. In this embodiment, the user can write on the display 220 using their hand or an input means 291 such as an electronic pen 2500.

[0034] Although the display device 2 in Figure 4 is positioned horizontally, it can also be positioned vertically. The user can switch between horizontal and vertical orientation by rotating the display device 2 around the center of the display 220.

[0035] <Example Hardware Configuration> Figure 5 shows an example of the hardware configuration of the display device 2. The display device 2 in this embodiment is equipped with a CPU (Central Processing Unit) 201, ROM (Read Only Memory) 202, RAM (Random Access Memory) 203, SSD (Solid State Drive) 204, a network controller 205, and an external device connection I / F 206 (Interface), and is a shared terminal for sharing information among multiple users.

[0036] Of these, the CPU 201 controls the operation of the entire display device 2. The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and the IPL (Initial Program Loader). The RAM 203 is used as the work area for the CPU 201.

[0037] SSD204 stores various data, such as programs for display device 2. This program may be an application that runs on an information processing device equipped with a general-purpose OS (such as Windows®, Mac OS®, Android®, iOS®, etc.). In this case, display device 2 is normally used as a general-purpose information processing terminal, but when a user runs an installed application program, it can function like a dedicated display device, allowing the user to write on it, etc.

[0038] The network controller 205 controls communication with the network. The external device connection interface 206 controls communication with the USB (Universal Serial Bus) memory 2600 and external devices (camera 2400, speaker 2300, microphone 2200).

[0039] The display device 2 also includes a capture device 211, a GPU 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication unit 219, and an antenna 219a for the short-range communication unit 219.

[0040] Of these, the capture device 211 displays video information as a still image or video on the PC 10's display. The GPU (Graphics Processing Unit) 212 is a semiconductor chip specializing in graphics. The display controller 213 controls and manages screen display in order to output the image from the GPU 212 to the display 220, etc.

[0041] The contact sensor 214 detects when the electronic pen 2500 or the user's hand H comes into contact with the display 220. When the electronic pen 2500 and the hand H are not distinguished, it is referred to as the input means 291.

[0042] The sensor controller 215 controls the processing of the contact sensor 214. The contact sensor 214 performs coordinate input and coordinate detection using an infrared blocking method. This method of coordinate input and coordinate detection may involve two light-emitting and receiving devices installed at both upper ends of the display 220 emitting multiple infrared rays parallel to the display 220, which are reflected by reflective members provided around the display 220, and the light received by the light-receiving element is received when it returns along the same optical path as the light emitted by the light-receiving element.

[0043] The contact sensor 214 outputs the infrared ID emitted by two light-receiving devices that are blocked by an object to the sensor controller 215, which then identifies the coordinate position of the object's contact. The electronic pen controller 216 communicates with the electronic pen 2500 to determine whether the pen tip or the pen end has touched the display 220. The short-range communication unit 219 is a communication circuit such as NFC or Bluetooth®.

[0044] Furthermore, the display device 2 is equipped with a bus line 210. The bus line 210 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 201.

[0045] Furthermore, the contact sensor 214 is not limited to an infrared blocking type; it may also be a capacitive type touch panel that identifies the contact position by detecting changes in capacitance. The contact sensor 214 may also be a resistive type touch panel that identifies the contact position by voltage changes between two opposing resistive films. The contact sensor 214 may also be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by contact between an object and the display unit. The contact sensor 214 may use any of these various detection means. In addition, the electronic pen controller 216 may be configured to determine whether there is a touch not only on the tip and end of the electronic pen 2500, but also on the part of the electronic pen 2500 that the user holds and other parts of the electronic pen.

[0046] <About the features> Figure 6 is a diagram showing an example of a functional block diagram that explains the functions of the display device 2 in a block-like manner. The display device 2 shown in Figure 6 includes a drawing object management unit 11, a contact position detection unit 12, a drawing data generation unit 13, a display control unit 14, a swipe operation detection unit 15, a network communication unit 16, a symbol recognition unit 17, and a conference data storage unit 39.

[0047] Each function of the display device 2 is a function or means realized by any of the components shown in Figure 5 operating according to instructions from the CPU 201 in accordance with an application deployed on the RAM 203.

[0048] The contact position detection unit 12 detects the coordinates of the position where the input means 291 (such as a finger or electronic pen) makes contact with the touch panel. The contact position detection unit 12 determines that the electronic pen 2500 has made contact if the number of phototransistors (width of light blocking) that cannot detect light from an LED or other light-emitting element is less than or equal to a predetermined number, and determines that another object (such as a finger) has made contact if the number is greater than the predetermined number.

[0049] The drawing data generation unit 13 generates a stroke from the sequence of coordinate points detected by the contact position detection unit 12 when the user writes by hand. The drawing data generation unit 13 also generates shapes such as circles and rectangles from the shape of the stroke.

[0050] The drawing object management unit 11 manages the stroke area, string area, shape area such as circles and rectangles, or table object area generated by the drawing data generation unit 13 as objects. The drawing object management unit 11 assigns an object ID to each of these areas and also manages the display position (pixel position on the display 220) of each drawing object.

[0051] The display control unit 14 controls the display of these objects, menus, etc., on the display 220.

[0052] The symbol recognition unit 17 recognizes symbols such as "+" that the user has handwritten by comparing them with pre-registered symbol data. The symbol recognition unit 17 may use pattern matching or machine learning results.

[0053] The swipe operation detection unit 15 detects a swipe operation based on the coordinate point sequence detected by the contact position detection unit 12 when the input means 291 detected by the contact position detection unit 12 is not an electronic pen. Note that swiping with an electronic pen 2500 is also possible; in this embodiment, the user swipes using the input means 291.

[0054] The meeting data storage unit 39 stores materials, objects, etc., used in the meeting in the SSD 204.

[0055] The network communication unit 16 connects to a network such as a LAN or the Internet and transmits and receives data with other devices via the network.

[0056] <Object coordinates> Referring to Figure 7, the coordinates of objects managed by the drawing object management unit 11 will be explained. The coordinates of objects in the display area 51, such as a screen, are expressed using display coordinates, with the pixel position at the upper left corner of the display area 51 of the display 220 of the display device 2 as the origin, and the right direction being the positive X-axis and the downward direction being the positive Y-axis. That is, Position (coordinates) = (number of pixels in the X-axis direction from the origin pixel, number of pixels in the Y-axis direction from the origin pixel) That is the case.

[0057] The display area 51 is the area currently displayed on the display 220 within the input-enabled area 52 handled by the display device 2. The coordinates of the display area 51 itself within the input-enabled area 52 are based on the pixel position at the upper left corner of the initial display area 51 (part of the display 220) as the origin. Furthermore, the coordinates are expressed with the right direction as the positive X-axis, the left direction as the negative X-axis, the down direction as the positive Y-axis, and the up direction as the negative Y-axis.

[0058] Furthermore, the drawing object management unit 11 manages the object's area using a rectangle that circumsects the object, using the pixel position at the upper left corner of this rectangular area as the starting coordinate and the pixel position at the lower right corner of the rectangular area as the ending coordinate.

[0059] The coordinate spacing of the touch panel is the same as the coordinate spacing of display 220, and the coordinates of both are assumed to be identical.

[0060] <Relationship between layer structure and display control unit> Referring to Figure 8, the relationship between the layer structure and the display control unit 14 will be explained. Figure 8 is a diagram illustrating the layers used by the display control unit 14 for display. The display control unit 14 can display objects using two layers. 1. Drawing layer 54 2. Pinned layer 53 Both layers consist of data that represents each pixel using RGB values. When an object is created, it is first added to drawing layer 54.

[0061] Pinned layer 53 is a transparent layer to which objects are pinned, and it is always located above drawing layer 54. Objects from drawing layer 54 are temporarily copied to pinned layer 53, but pixels other than the objects remain transparent.

[0062] The display control unit 14 performs an OR operation on the pixels of the drawing layer 54 and the pinning layer 53 and displays them on the display 220. In other words, the display control unit 14 combines the pixels of the drawing layer 54 and the pinning layer 53 and displays them. The display control unit 14 may also simply prioritize displaying the pixels of the pinning layer 53, which is closer to the viewer. The size of the drawing layer 54 matches the input area 52 handled by the display device.

[0063] <Example of display using pinned layers> Refer to Figure 9 to explain how to display objects using the pinned layer 53. Figure 9 shows an example of how to display objects fixed to the pinned layer 53.

[0064] As shown in Figure 9(a), the user hand-wrote "〇× Seminar" (Object A) and a list of agenda items (Object B). The user also hand-wrote "Equipment Reservation" and an equipment list ("• Microphone", "• Cable", "• Monitor", "• Laser Pointer") as meeting minutes for the agenda item "Equipment Reservation". Let's call these meeting minutes Object C.

[0065] Object C, consisting of "equipment reservation," "microphone," "cable," "monitor," and "laser pointer," is actually a single object. However, in this embodiment, for explanatory purposes, it is collectively referred to as a single object C.

[0066] In this embodiment, the drawing object management unit 11 determines that input for one object is complete when no stroke input is received for a predetermined time, or when a stroke is entered at a coordinate more than a predetermined distance away. In addition, objects may be classified according to their type, and in this embodiment, objects are classified in a predetermined manner.

[0067] Assuming the top left corner of the display area 51 is the origin (0,0), and the coordinates of the top left corner of an object are the starting coordinates and the coordinates of the bottom right corner are the ending coordinates, the starting coordinates, ending coordinates, and layer attributes of objects A, B, and C are shown in Table 1(a).

[0068] [Table 1] Table 1 is an example of an object table in which object information is registered. The object table is maintained by the display device 2. The object table may reside on a network. The object ID is identification information that identifies the object. The start and end coordinates are as shown in Figure 7. The layer attribute indicates which layer each object is on.

[0069] If a user writes a stroke before a predetermined time has elapsed since the previous stroke, the drawing object management unit 11 manages those strokes as a single object. Strokes written after a predetermined time has elapsed since the previous stroke are managed as separate objects. In the case of table objects, the drawing object management unit 11 may consider the strokes within the outer frame area as a single object.

[0070] When the next agenda item on the "Agenda List" (Object B) is to be discussed, the user needs space to write notes for this item, so they clear the right side of the "Agenda List" (Object B). First, the user long-presses the "Agenda List" (Object B). The drawing object management unit 11 determines which object the contact point overlaps with. Since the contact point was inside the "Agenda List" (Object B), the drawing object management unit 11 selects the "Agenda List" (Object B) and displays a pin button 9 (an example of the first display component) for pinning.

[0071] Figure 9(b) shows object B in the table with the pin button 9 displayed. When the user presses the pin button 9 with the input means 291, the drawing object management unit 11 copies the pixel data corresponding to the "Agenda List" (object B) in the drawing layer 54 to the pin layer 53 and deletes the pixel data corresponding to the "Agenda List" (object B) in the drawing layer 54. The start and end coordinates of objects A to C in this state are shown in Table 1(b).

[0072] The user then swipes the input means 291 from any position on the display 220 to any position. When the swipe operation detection unit 15 detects that a swipe is being performed, the display control unit 14 repeatedly subtracts the amount of movement in the x-axis direction from the x-coordinate of objects A and C on the drawing layer 54, and the amount of movement in the y-axis direction from the y-coordinate of objects A and C, from the x-coordinate of the drawing layer 54, at predetermined intervals.

[0073] The display control unit 14 updates the display of the object at predetermined intervals. When the user stops swiping, the starting coordinates of objects A and C in the drawing layer 54 become (Sxa-dx, Sya-dy) and (Sxc-dx, Syc-dy), respectively.

[0074] The display control unit 14 performs an OR operation on each pixel of the pinned layer 53 and the drawing layer 54 corresponding to the display area 51 and displays it on the display 220. Since object B is on the pinned layer 53, the coordinates of object B do not change (the display position does not change), and the new display area 51 is displayed.

[0075] Figure 9(c) shows the new display area 51 created by moving objects A and C. In this way, the user was able to create a space to write notes for the next agenda item on the "Agenda List" (object B) to the right of the "Agenda List" (object B). The starting and ending coordinates of objects A to C in the state shown in Figure 9(c) are shown in Table 1(c).

[0076] The method for ending the pinning of object B will be explained later.

[0077] <Operating Procedure> Figure 10 is an example of a flowchart illustrating the procedure by which the display device 2 displays an object fixed in the display area.

[0078] The contact position detection unit 12 determines whether or not the stroke was handwritten (S1).

[0079] If the stroke was handwritten (Yes in S1), determine whether a predetermined amount of time has elapsed since the stroke was entered (S2).

[0080] The drawing object management unit 11 continues to add strokes to the temporary object until a predetermined time has elapsed (S3).

[0081] After a predetermined time has elapsed, the drawing object management unit 11 determines that the multiple strokes contained in the temporary object are a single object and registers it in the object table (S4).

[0082] Next, the drawing object management unit 11 determines whether the contact position detection unit 12 has detected that the object has been pressed and held down (S5).

[0083] If the contact position detection unit 12 detects that an object has been pressed and held (Yes in S5), the display control unit 14 displays the pinning button 9 around the object that was pressed and held (S6).

[0084] Next, the drawing object management unit 11 determines whether the contact position detection unit 12 has detected that the pinning button 9 has been pressed (S7).

[0085] The drawing object management unit 11 copies the pixel data of the object pressed and held on the pinned layer 53 and deletes the said pixel data from the drawing layer 54 (S8). Note that the deletion of the pixel data from the drawing layer 54 is not required, and the drawing object management unit 11 may indicate that the pixel data from the drawing layer 54 has been copied at half brightness.

[0086] Next, the drawing object management unit 11 determines whether the contact position detection unit 12 has detected that the display area has been swiped (S9). Note that it is sufficient to detect that a swipe has been made at any position on the display area, and it is also acceptable to detect that an object on the display area has been swiped.

[0087] If a swipe is performed, at predetermined intervals (S10), the display control unit 14 repeatedly subtracts the amount of movement in the x-axis direction from the x-coordinate of the object in the drawing layer 54, and the amount of movement in the y-axis direction from the y-coordinate (S11). In this way, while accepting an operation to move the display area 51, the display control unit 14 can maintain the position of the object in the display area 51 at the time the object selection was accepted.

[0088] Furthermore, the display control unit 14 performs an OR operation on the pixel data of the drawing layer 54 and the pixel data of the pinning layer 53 and displays it on the display 220 (S12).

[0089] When the swipe is finished (when the input means 291 leaves the display 220), the process in Figure 10 ends. In order to allow the user to move the input means 291 away from the display 220 and swipe again, the contact position detection unit 12 may detect the end of the swipe if no contact of the input means 291 is detected for a certain period of time after the input means 291 leaves the display 220.

[0090] As described above, according to this embodiment, the display device 2 can always display the pinned object in the display area 51, and the user can write in the new space around the object. [Examples]

[0091] This embodiment describes a display device 2 that can be selected by the user by drawing a stroke around an object.

[0092] In this embodiment, we will explain the process assuming that the hardware configuration diagram in Figure 5 and the functional block diagram in Figure 6, as described in the above embodiment, can be used as reference.

[0093] <Example of display using pinned layers> Refer to Figure 11 to explain how to display objects using the pinned layer 53. Figure 11 shows an example of how to display objects fixed to the pinned layer 53.

[0094] As shown in Figure 11(a), the user manually entered "〇× Seminar" (Object A), a list of agenda items (Object B), and a list of equipment reservations (Object C). The object table in this state is shown in Table 2(a).

[0095] [Table 2] The structure of the object table (item names) is the same as in Table 1.

[0096] Next, as shown in Figure 11(b), the user draws a predetermined mark 301 for pinning (in this case, a mark with a circle around a plus sign), and then surrounds the area to be pinned with a stroke 302. Mark 301 is just an example; any mark can be used.

[0097] If the symbol recognition unit 17 determines that the mark 301 drawn by the user is a "plus sign enclosed in a circle", the drawing object management unit 11 monitors the area enclosed by the stroke 302.

[0098] The drawing object management unit 11 pins object B, which is located in the area enclosed by the stroke, to the pinning layer 53. That is, the drawing object management unit 11 copies the pixel data of object B in the drawing layer 54 to the pinning layer 53 and deletes the pixel data corresponding to object B in the drawing layer 54. The state of objects A, B, and C when object B is enclosed is shown in the object table in Table 2(b).

[0099] The user then swipes the display area (any object or any position). The processing of the display device 2 during swiping can be the same as in Example 1. The state of objects A, B, and C after moving the display area (any object or any position) is shown in the object table in Table 2(c).

[0100] Figure 11(c) shows the display area 51 after swiping. Since object B's coordinates do not change, its display position remains the same, and the display device 2 can display the new display area 51. The state of objects A, B, and C after swiping is shown in the object table in Table 2(c).

[0101] <Operating Procedure> Figure 12 is an example of a flowchart illustrating the procedure by which the display device 2 displays an object fixed in the display area. Note that the explanation of Figure 12 will primarily focus on the differences from Figure 10.

[0102] The symbol recognition unit 17 determines whether the user has handwritten a predetermined mark 301 for pinning based on the stroke generated by the drawing data generation unit 13 (S21).

[0103] If a predetermined pinning mark 301 is handwritten (Yes in S21), the drawing object management unit 11 determines, based on the stroke 302 generated by the drawing data generation unit 13, whether or not there is an object whose entirety is contained within the rectangle of the stroke 302 (S22, S23). In this determination, it is acceptable if a part of the object extends beyond the rectangle of the stroke 302. The drawing object management unit 11 determines that the object within the stroke 302 is the object to be moved.

[0104] The processing in steps S24 to S28 may be the same as in steps S8 to S12 in Figure 10.

[0105] According to this embodiment, the user can pin object B by hand-drawing a predetermined mark 301 and a stroke surrounding object B, thereby moving other objects and creating new space. Mark 301 is just an example; any mark may be used. Alternatively, object B may be pinned if it is surrounded by a stroke, even without mark 301. [Examples]

[0106] This embodiment describes a display device 2 that allows a user to select an object by handwriting a predetermined mark next to the object.

[0107] In this embodiment, we will explain the process assuming that the hardware configuration diagram in Figure 5 and the functional block diagram in Figure 6, as described in the above embodiment, can be used as reference.

[0108] <Example of display using pinned layers> Refer to Figure 13 to explain how to display objects using the pinned layer 53. Figure 13 shows an example of how to display objects fixed to the pinned layer 53.

[0109] As shown in Figure 13(a), the user hand-wrote "○× Seminar" (Object A), the agenda item "·Schedule Notification" (Object B), and "·Equipment Reservation" (Object C).

[0110] Next, as shown in Figure 13(b), the user hand-wrote an equipment list ("microphone", "cable", "monitor", "laser pointer") as meeting minutes for the agenda item "reserving equipment". Each of the equipment list items ("microphone", "cable", "monitor", "laser pointer") is a single object, but for the sake of explanation, they are collectively referred to as object D. Table 3(a) shows the object table for objects A to D when they were hand-written.

[0111] [Table 3] The structure (item names) of the object table in Table 3 can be the same as in Tables 1 and 2.

[0112] Next, as shown in Figure 13(c), the user hand-writes a designated mark 305 (in this case, a + mark) next to the "Schedule Notification" (object B) to which they want to pin. Mark 305 is just an example; any mark can be used.

[0113] The drawing object management unit 11 determines whether there are other objects within a distance m (within a threshold) from the mark 305. If the mark 305 and "Schedule Notification" (object B) and "Equipment Reservation" (object C) are within a distance m, the unit selects "Schedule Notification" (object B) and "Equipment Reservation" (object C), and the user can specify the object to be pinned by pressing object B or C with the input means 291.

[0114] Alternatively, the drawing object management unit 11 may identify objects B that are within a distance m from the mark 305 only in the horizontal direction.

[0115] When the symbol recognition unit 17 determines that the handwritten mark 305 by the user is a "+ mark", the drawing object management unit 11 copies the pixel data corresponding to "·Schedule Notification" (object B) in the drawing layer 54 to the pinning layer 53 and deletes the pixel data corresponding to "·Schedule Notification" (object B) in the drawing layer 54. The state of objects A to D when the handwritten mark 305 is determined to be a "+ mark" is shown in the object table in Table 3(b).

[0116] The user then swipes the display area (any object or any position). The processing of the display device 2 during swiping can be the same as in Example 1. The state of objects A to D after moving the display area (any object or any position) is shown in the object table in Table 3(c).

[0117] Figure 13(d) shows the display area 51 after swiping. Since object B's coordinates do not change, its display position remains the same, and the display device 2 can display the new display area 51.

[0118] <Operating Procedure> Figure 14 is an example of a flowchart illustrating the procedure by which the display device 2 displays an object fixed in the display area. Note that the explanation of Figure 14 will primarily focus on the differences from Figure 10.

[0119] The symbol recognition unit 17 determines whether the user has handwritten a predetermined mark 305 for pinning based on the stroke generated by the drawing data generation unit 13 (S31).

[0120] If mark 305 is drawn by hand, the drawing object management unit 11 determines whether or not an object exists within a distance m from mark 305 (S32). The drawing object management unit 11 compares the distance between the bounding rectangle of mark 305 and the nearest bounding rectangle of the object. The drawing object management unit 11 determines that any object within a distance m from mark 305 is an object to be fixed.

[0121] The processing in steps S33 to S37 may be the same as in steps S8 to S12 in Figure 10.

[0122] In this embodiment, the user can pin object B by handwriting a predetermined mark 305, move other objects, and create new space. [Examples]

[0123] This embodiment describes a display device 2 that allows the user to pin objects by operating an operation menu.

[0124] In this embodiment, we will explain the process assuming that the hardware configuration diagram in Figure 5 and the functional block diagram in Figure 6, as described in the above embodiment, can be used as reference.

[0125] <About Menu Layers> In this embodiment, the operation menu includes a pin button 9, and the user can pin an object by pressing the pin button 9.

[0126] Therefore, we will explain the menu layer 55 with reference to Figure 15. Like the other layers, the menu layer 55 represents each pixel using RGB. The menu layer 55 is the layer that displays buttons, icons, etc., for selecting functions, and is always located above the pinned layer 53. Also, the size of the menu layer 55 matches the display area 51.

[0127] The display control unit 14 performs an OR operation on the pixels of the drawing layer 54 and the pinning layer 53. Furthermore, for non-transparent pixels in the menu layer 55, it replaces the value of the pixel obtained by the OR operation with the value of the pixel in the menu layer 55 and displays it on the display 220. In other words, the display control unit 14 displays the menu layer 55 in the foreground.

[0128] <Example of display using pinned layers> Refer to Figure 16 to explain how to display objects using the pinned layer 53. Figure 16 shows an example of how to display an object that has been pinned to the pinned layer 53 by clicking the pinned icon 307.

[0129] As shown in Figure 16(a), the user hand-wrote "O× Seminar" (Object A) and "List of Agenda Items" (Object B). An operation menu 306 that accepts the selection of various functions is displayed on the right side of the display area 51. This operation menu 306 is formed in menu layer 55. The operation menu 306 has pen input, eraser, undo, redo, and pin icon 307 (these are just examples). The pin icon 307 is an icon for pinning objects. Table 4(a) shows the state of objects A and B when they were hand-wrote.

[0130] [Table 4] The structure (item names) of the object table in Table 4 can be the same as in Tables 1-3.

[0131] Next, as shown in Figure 16(b), the user presses the pin icon 307 from the operation menu 306. The drawing object management unit 11 enters a state where it is waiting for the selection of an object. If the user presses, for example, "List of Agenda Items" (object B) using the input means 291, the drawing object management unit 11 determines that "List of Agenda Items" (object B) is a target for pinning.

[0132] The drawing object management unit 11 copies the pixel data corresponding to the "Agenda List" (Object B) from the drawing layer 54 to the pinned layer 53, and deletes the pixel data corresponding to the "Agenda List" (Object B) from the drawing layer 54. The state of objects A and B when the "Agenda List" (Object B) is selected by the user is shown in the object table in Table 4(b).

[0133] The user then swipes the display area (any object or any position). The processing of the display device 2 during swiping can be the same as in Example 1. The state of objects A and B after moving the display area (any object or any position) is shown in the object table in Table 4(c).

[0134] Figure 16(c) shows the display area 51 after swiping. Since object B's coordinates do not change, its display position remains the same, and the display device 2 can display the new display area 51. Also, since the operation menu 306 is fixed to the display area 51, it is displayed in the same position.

[0135] <Operating Procedure> Figure 17 is an example of a flowchart illustrating the procedure by which the display device 2 displays an object fixed in the display area 51. Note that the explanation of Figure 17 will primarily focus on the differences from Figure 10.

[0136] The contact position detection unit 12 determines whether or not the user has pressed the pin icon 307 based on the contact position of the input means 291 (S41).

[0137] Next, the drawing object management unit 11 determines whether or not an object has been pressed based on the contact position of the input means 291 detected by the contact position detection unit 12 (S42). The drawing object management unit 11 determines that the pressed object is the object to be moved.

[0138] The processing in steps S43 to S47 may be the same as in steps S8 to S12 in Figure 10.

[0139] In this embodiment, the user can pin object B by pressing the pin icon 307 and selecting the object, thereby moving other objects and creating new space. [Examples]

[0140] In this embodiment, we will describe a display device 2 that allows the user to pin an object by operating the operation menu 306 and surrounding the object with a stroke.

[0141] In this embodiment, we will use the hardware configuration diagram in Figure 5 and the functional block diagram in Figure 6, which were described in the above embodiment, as references. Furthermore, the menu layer 55 is the same as in Embodiment 4.

[0142] <Example of display using pinned layers> Referring to Figure 18, the method of displaying an object using the pinning layer 53 will be explained. Figure 18 shows an example of displaying an object that is fixed to the pinning layer 53 by being enclosed in stroke 308.

[0143] As shown in Figure 18(a), the user hand-wrote "O× Seminar" (Object A) and "List of Agenda Items" (Object B). An operation menu 306 that accepts the selection of various functions is displayed on the right side of the display area 51. The operation menu 306 has pen input, eraser, undo, redo, and pin icon 307 (these are just examples). The pin icon 307 is an icon for pinning objects. Table 5(a) shows the state of objects A and B when they were hand-wrote.

[0144] [Table 5] The structure (item names) of the object table in Table 5 can be the same as in Tables 1-4.

[0145] Next, as shown in Figure 18(b), the user presses the pin icon 307 from the operation menu 306. The drawing object management unit 11 enters a state where it is waiting for an object to be selected. If the user, for example, draws a stroke 308 around the "List of Agenda Items" (object B), the drawing object management unit 11 determines that the "List of Agenda Items" (object B) is a target for pinning.

[0146] The drawing object management unit 11 copies the pixel data corresponding to the "Agenda List" (object B) from the drawing layer 54 to the pinned layer 53, and deletes the pixel data corresponding to the "Agenda List" (object B) from the drawing layer 54. The state of objects A and B when the "Agenda List" (object B) is selected by the user is shown in the object table in Table 5(b).

[0147] The user then swipes the display area (any object or any position). The processing of the display device 2 during swiping can be the same as in Example 1. The state of objects A and B after moving the display area (any object or any position) is shown in the object table in Table 5(c).

[0148] Figure 18(c) shows the display area 51 after swiping. Since object B's coordinates do not change, its display position remains the same, and the display device 2 can display the new display area 51. Also, since the operation menu 306 is fixed to the display area 51, it is displayed in the same position.

[0149] <Operating Procedure> Figure 19 is an example of a flowchart illustrating the procedure by which the display device 2 displays an object fixed in the display area 51. Note that the explanation of Figure 19 will primarily focus on the differences from Figure 10.

[0150] The contact position detection unit 12 determines whether or not the user has pressed the pin icon 307 based on the contact position of the input means 291 (S51).

[0151] If the pin icon 307 is pressed (Yes in S51), the drawing object management unit 11 determines, based on the stroke 308 generated by the drawing data generation unit 13, whether or not there is an object whose entirety is contained within the bounding rectangle of the stroke 308 (S52, S53). In this determination, it is acceptable if a part of the object extends beyond the rectangle of the stroke 308.

[0152] The processing in steps S54 to S58 can be the same as in steps S8 to S12 in Figure 10.

[0153] In this embodiment, the user can pin object B by pressing the pin icon 307 and enclosing the object with a stroke 308, thereby moving other objects and creating new space. [Examples]

[0154] This embodiment describes the termination of object pinning.

[0155] In this embodiment, we will explain the process assuming that the hardware configuration diagram in Figure 5 and the functional block diagram in Figure 6, as described in the above embodiment, can be used as reference.

[0156] <Example of display when pinning is complete> Figure 20 illustrates an example of how objects are displayed when pinning is complete. Figure 20(a) shows the arrangement of objects A to C as in Figure 9(c), with the user having handwritten a "timetable" (object D) in the space. Object D's "10:00~12:00 Instructor A," "13:00~15:00 Instructor B," and "15:30~17:30 Instructor C" are actually separate objects. However, in this embodiment, for explanatory purposes, they are collectively referred to as a single object D. The state of objects A to D when object D is added is shown in the object table in Table 6(a).

[0157] [Table 6] The structure (item names) of the object table in Table 6 can be the same as in Tables 1-5.

[0158] When the next agenda item on the "Agenda List" (Object B) is to be discussed, the user is required to provide a layout diagram of the venue. Since there is not enough space in the current display area 51, the user moves the "Agenda List" (Object B) from the display area 51 to the hidden area. Therefore, when the user long-presses any position on the "Agenda List" (Object B), the drawing object management unit 11 displays an unpin button 8 (an example of a second display component) to unpin the object, as shown in Figure 20(b). When the user presses the unpin button 8 with the input means 291, the drawing object management unit 11 copies the pixel data corresponding to the "Agenda List" (Object B) in the pinned layer 53 to the drawing layer 54 and deletes the pixel data corresponding to the "Agenda List" (Object B) in the pinned layer 53.

[0159] Next, when the user swipes the display area 51 to the left by a distance of dx and up by dy, objects A to D are subtracted from their x-coordinates by dx and from their y-coordinates by dy, and disappear from the display area 51 as shown in Figure 20(c). The state of objects A to D when the display area 51 is moved is shown in the object table in Table 6(b).

[0160] <Operating Procedure> Figure 21 is an example of a flowchart illustrating the procedure for the display device 2 to release an object fixed in the display area 51. Note that the explanation of Figure 21 will primarily focus on the differences from Figure 10.

[0161] The drawing object management unit 11 determines whether or not an object on the pinned layer has been pressed down based on the contact position of the input means 291 detected by the contact position detection unit 12 (S61).

[0162] If an object on a pinned layer is pressed and held, the display control unit 14 displays an unpin button 8 around the selected object (S62).

[0163] The drawing object management unit 11 determines whether or not the pin release button 8 has been pressed based on the contact position of the input means 291 detected by the contact position detection unit 12 (S63).

[0164] If the unpin button 8 is pressed (Yes in S63), the drawing object management unit 11 moves the objects from the pinned layer 53 to the drawing layer 54 (S64). The drawing object management unit 11 deletes the pixel data of the objects from the pinned layer 53.

[0165] The processing in steps S65 to S67 may be the same as in steps S9 to S11 in Figure 10.

[0166] According to this embodiment, the user can unpin an object and display a display area 51 that does not contain any objects. [Examples]

[0167] In this embodiment, we will describe a display device 2 that moves the display area 51 after the user moves the object to be pinned with a simple operation.

[0168] In this embodiment, we will explain the process assuming that the hardware configuration diagram in Figure 5 and the functional block diagram in Figure 6, as described in the above embodiment, can be used as reference.

[0169] <Example of display using pinned layers> Refer to Figure 22 to explain how to display objects using the pinned layer 53. Figure 22 shows an example of how to display objects fixed to the pinned layer 53.

[0170] As shown in Figure 22(a), the user hand-wrote "O× Seminar" (Object A) and "List of Agenda Items" (Object B). A function selection menu 306 is displayed on the right side of the display 220.

[0171] Assuming the top left corner of the display area 51 is the origin (0,0), and the coordinates of the top left corner of the object are the starting coordinates and the coordinates of the bottom right corner are the ending coordinates, the starting coordinates, ending coordinates, and layer attributes of objects A and B are shown in Table 7(a).

[0172] [Table 7]

[0173] When the user presses the pinning icon 307 from the operation menu 306 to pin an object, the display device 2 transitions to pinning mode. Pinning mode is an operation mode in which an object is pinned to the pinning layer 53 and made movable. Next, as shown in Figure 22(b), the user presses an arbitrary position (Px, Py) on the display 220 using the input means 291. The display control unit 14 displays the pinning button 9 at the pressed location (Figure 22(c)).

[0174] Next, as shown in Figure 22(c), the user presses an arbitrary position on the "Agenda List" (Object B) using the input means 291. The drawing object management unit 11 copies the pixel data corresponding to the "Agenda List" (Object B) from the drawing layer 54 to the pinning layer 53 so that the center coordinates of the "Agenda List" (Object B) are at the position of the pinning button 9, and then deletes the pixel data corresponding to the "Agenda List" (Object B) from the drawing layer 54 (Figure 22(d)). The starting and ending coordinates of objects A and B after this process are shown in Table 7(b). The drawing object management unit 11 may also move the "Agenda List" (Object B) so that its upper left corner is at the position of the pinning button 9.

[0175] Moving on to the next topic, and needing space to write notes for this topic, the user swipes the display (any object or position) to move dx to the left and dy up. The swipe operation detection unit 15 detects the swipe and, at predetermined intervals, subtracts the amount of movement in the x-axis direction from the x-coordinate of object A in the drawing layer 54, and the amount of movement in the y-axis direction from the y-coordinate. The display control unit 14 updates the display data at predetermined intervals. When the user stops swiping, the starting coordinates of object A in the drawing layer 54 become (Sxa-dx, Sya-dy). Then, the display control unit 14 performs an OR operation on the pixels of the drawing layer 54 and the pinned layer 53 corresponding to the display area 51 and displays it on the display 220. Note that the display control unit 14 displays the menu layer in the foreground, but it is hidden in Figure 22.

[0176] As a result, as shown in Figure 22(e), only object B remains in the same position without swiping because its coordinates do not change. The starting and ending coordinates of objects A and B after this process are shown in Table 7(c).

[0177] <Operating Procedure> Figure 23 is an example of a flowchart illustrating the procedure for displaying an object fixed in the display area of ​​the display device 2.

[0178] The contact position detection unit 12 determines whether or not the input means 291 for the pin icon 307 has been pressed (S71).

[0179] If the pin icon 307 is pressed (Yes in S71), the drawing object management unit 11 operates in pin mode. Subsequently, the contact position detection unit 12 determines whether or not it has detected a press of the input means 291 at any position (Px, Py) on the display 220 (S72).

[0180] If the contact position detection unit 12 detects a press at any position (Px, Py) on the display 220 (Yes in S72), the display control unit 14 displays the pin button 9 at the pressed location (S73).

[0181] Next, the contact position detection unit 12 determines whether or not it has detected the user selecting (pressing) any object (S74). Any position within the bounding rectangle of the object may be pressed.

[0182] If the contact position detection unit 12 detects the selection (press) of any object (Yes in S74), the drawing object management unit 11 copies the pixel data corresponding to the "Agenda List" (object B) from the drawing layer 54 to the pinning layer 53 so that the center coordinates of the selected object become the position of the pinning button 9, and deletes the pixel data corresponding to the "Agenda List" (object B) from the drawing layer 54 (S75).

[0183] The processing in steps S76 to S79 ​​may be the same as in steps S9 to S12 in Figure 10. Also, the method for ending the pinning of the object may be the same as in Example 6.

[0184] <Main effects> According to this embodiment, the user can move the object to be pinned with a simple operation and then move the display area 51, so the object can be moved in advance while taking into account the position of the margins. [Examples]

[0185] This embodiment describes a display device 2 that allows the user to move and align one or more pinned objects before moving the display area 51.

[0186] In this embodiment, we will explain the process assuming that the hardware configuration diagram in Figure 5 and the functional block diagram in Figure 6, as described in the above embodiment, can be used as reference.

[0187] <Example of display using pinned layers> Refer to Figure 24 to explain how to display objects using the pinned layer 53. Figure 24 shows an example of how to display objects fixed to the pinned layer 53.

[0188] As shown in Figure 24(a), the user hand-wrote "O× Seminar" (Object A) and "List of Agenda Items" (Object B). Next, the user operates the display device 2 to selection mode. Selection mode is an operating mode in which objects can be selected using the input means 291. When the user specifies a rectangular area with the input means 291, the display device 2 displays a pinning button 9 for pinning the object. The position of the pinning button 9 is not limited to the position shown in the figure and can be any position.

[0189] When the user presses the pin button 9, the display device 2 displays a pinning area 320 with two points in the rectangular area specified by the user as diagonal vertices (Figure 24(b)). With the top left of the display area 51 as the origin (0,0), and the coordinates of the top left of the object as the starting coordinate and the coordinates of the bottom right as the ending coordinate, the starting and ending coordinates of objects A and B and the pinning area, as well as the layer attributes, are shown in Table 8(a).

[0190] [Table 8]

[0191] Next, as shown in Figure 24(c), the user drags and drops the "Agenda List" (Object B) to pin it to the pinning area 320. The drawing object management unit 11 copies the pixel data corresponding to the "Agenda List" (Object B) in the drawing layer 54 to the pinning layer 53, so that the starting point of the "Agenda List" (Object B) (for example, the top-left vertex) becomes the starting point of the pinning area 320. The copied Object B is named "Object B'". Then, the drawing object management unit 11 deletes the pixel data corresponding to the "Agenda List" (Object B) in the drawing layer 54 (Figure 24(d)). In this way, the user can place an object in the pinning area 320 and then move the display area 51.

[0192] A user can move more than one object to the pinning area 320. The drawing object management unit 11 shrinks the objects so that all objects moved by the user to the pinning area 320 fit within the pinning area 320. The drawing object management unit 11 may also display each object as a thumbnail or an icon representing the object.

[0193] In the example shown in Figure 24(c), since the width of object B is greater than the width of the pinning area 320, the drawing object management unit 11 shrinks object B to fit within the pinning area 320 before moving it to the pinning area 320. The shrinking ratio α is "width of pinning area / width of object B". If the height of object B is greater than the height of the pinning area 320, the shrinking ratio α is the ratio of the heights. The drawing object management unit 11 multiplies the height and width of object B' by the shrinking ratio α, respectively. Objects A and B' are shown in Table 8(b).

[0194] The user then swipes at any position on the display 220 using the input means 291. When the swipe operation detection unit 15 detects that a swipe is being performed, it repeatedly subtracts the amount of movement in the x-axis direction from the x-coordinate of objects A and B on the drawing layer 54, and the amount of movement in the y-axis direction from the y-coordinate of objects A and B on the drawing layer 54, at predetermined intervals. The display control unit 14 updates the display data at predetermined intervals. When the user stops swiping, the starting coordinates of objects A and B on the drawing layer 54 become (Sxa-dx, Sya-dy) and (Sxb-dx, Syb-dy), respectively.

[0195] The display control unit 14 then performs an OR operation on each pixel of the pinned layer 53 and the drawing layer 54 and displays it on the display 220. As a result, only object B' remains in the same position without swiping because its coordinates do not change (Figure 24(e)). The starting and ending coordinates of objects A and B' after this process are shown in Table 8(c).

[0196] Object B' may remain reduced in the pinning area 320, or the display control unit 14 may automatically return it to its original position in the display area 51 through user operation or when the swipe is finished (Figure 24(f)).

[0197] If object B' remains reduced in the pinning area 320, when the user presses object B', the display control unit 14 can display a preview 321 of object B' at its original size (Figure 24(g)). The preview 321 is the same size as or larger than the original object B, allowing the user to check the contents of object B. The preview 321 is cleared after a certain period of time or by user action. From the state of the preview 321, the user may return object B to its original position in the display area 51.

[0198] If object B returns to its original position in the display area 51 due to user action or automatically, the drawing object management unit 11 copies the object from the pinned layer 53 to the drawing layer 54. The drawing object management unit 11 then deletes the object's pixel data from the pinned layer 53.

[0199] <Operating Procedure> Figure 25 is an example of a flowchart illustrating the procedure by which the display device 2 displays an object fixed in the display area 51.

[0200] The contact position detection unit 12 determines whether or not a rectangle has been handwritten in selection mode (S81). For example, if the rectangle has the start and end points of a straight line as diagonal vertices, the contact position detection unit 12 detects the input of a straight line (stroke) and detects that the rectangle has been handwritten. The contact position detection unit 12 may also perform pattern recognition to determine whether or not it is a rectangle.

[0201] When a rectangle is drawn by hand in selection mode (Yes in S81), the display control unit 14 displays the pin button 9 inside or on the rectangle (S82).

[0202] The contact position detection unit 12 determines whether or not the pinning button 9 has been pressed based on the input means 291 (S83).

[0203] If the pressing of the pin button 9 is not detected (No in S83), the display control unit 14 determines whether a predetermined time has elapsed (S84). If the predetermined time has elapsed (Yes in S84), the display control unit 14 erases the rectangle and the pin button 9 (S85).

[0204] If the pressing of the pin button 9 is detected within a predetermined time (Yes in S83), the drawing object management unit 11 determines the area specified by the rectangle as the pinning area 320 (S86). The area specified by the rectangle may be the same area as the rectangle, or it may be an area adjusted so as not to include other objects that originally existed in the rectangle.

[0205] The contact position detection unit 12 determines whether or not an arbitrary object has been dragged and dropped into the pinning area 320 (S87). That is, the contact position detection unit 12 determines whether or not the starting point of the drag by the input means 291 is an object and the ending point of the drop is within the pinning area 320.

[0206] If the determination in step S87 is Yes, the drawing object management unit 11 copies the pixel data corresponding to the object in the drawing layer 54 to the pinning layer 53 so that the starting point of the object becomes the starting point of the pinning area 320, and then deletes the pixel data of the object from the drawing layer 54 (S88). This starting point is the upper left vertex of the object and the pinning area 320, but it may also be the upper right vertex, the lower right vertex, or the lower left vertex.

[0207] The processing in steps S89 to S92 may be the same as in steps S9 to S12 in Figure 10.

[0208] <Main effects> In this embodiment, the user can move one or more objects to the pinning area 320, align them, and then move the display area 51. Also, since the objects are scaled down to fit within the pinning area 320, it is easier to create margins when moving large objects along with the display area 51. [Examples]

[0209] In this embodiment, we will describe a display device 2 that allows the user to move the display area 51 while pressing and holding an object.

[0210] In this embodiment, we will explain the process assuming that the hardware configuration diagram in Figure 5 and the functional block diagram in Figure 6, as described in the above embodiment, can be used as reference.

[0211] <Example of display using pinned layers> Refer to Figure 26 to explain how to display objects using the pinned layer 53. Figure 26 shows an example of how to display objects fixed to the pinned layer 53.

[0212] As shown in Figure 26(a), the user handwrites "〇× Seminar" (Object A) and "List of Agenda Items" (Object B) on the display 220. The user also handwrites "Equipment Reservation" and an equipment list ("Microphone", "Cable", "Monitor", "Laser Pointer") as minutes for the agenda item "Equipment Reservation". Let this memo be Object C (the term Object C is used for explanatory purposes only, and these memos are not grouped objects). With the top left of the display area 51 as the origin (0,0), and the top left coordinates of the objects as the starting coordinates and the bottom right coordinates as the ending coordinates, Table 9(a) shows the starting and ending coordinates and layer attributes of Objects A, B, and C.

[0213] [Table 9]

[0214] Moving on to the next agenda item, the user needs space to write notes for this item, so they clear the right side of the "Agenda List" (Object B). First, the user long-presses the "Agenda List" (anywhere on Object B) with their hand H (an example of the first input method). The drawing object management unit 11 determines which object the coordinates touched by hand H correspond to. Since these coordinates are within the bounding rectangle of the "Agenda List" (Object B), the drawing object management unit 11 copies the pixel data corresponding to the "Agenda List" (Object B) in the drawing layer 54 to the pinning layer 53, and deletes the pixel data corresponding to the "Agenda List" (Object B) in the drawing layer 54.

[0215] Furthermore, to notify the user that the "Agenda List" (Object B) is temporarily fixed, the display control unit 14 outlines the frame 323 of the "Agenda List" (Object B) (Figure 26(b)). Outlining means displaying a thick border around the outer edge of Object B. Any border that emphasizes Object B, such as a dotted line border or a blinking border, is acceptable.

[0216] The user keeps their hand H in contact with the display 220 and swipes at any point on the display 220 using the electronic pen 2500 (an example of a second input means).

[0217] When the swipe operation detection unit 15 detects a swipe, the display control unit 14 repeatedly subtracts the amount of movement in the x-axis direction from the x-coordinate of objects A and C on the drawing layer 54, and the amount of movement in the y-axis direction from the y-coordinate of objects A and C on the drawing layer 54, at predetermined intervals. The display control unit 14 updates the display data at predetermined intervals. When the user stops swiping, the starting coordinates of objects A and C on the drawing layer 54 become (Sxa-dx, Sya-dy) and (Sxc-dx, Syc-dy), respectively.

[0218] When the user releases their hand H from pressing and holding the "Agenda List" (object B) (when they remove their hand H from the display 220), the drawing object management unit 11 copies the pixel data corresponding to the "Agenda List" (object B) in the pinned layer 53 to the drawing layer 54 and deletes the pixel data corresponding to the "Agenda List" (object B) in the pinned layer 53. Then the display control unit 14 displays each pixel of the drawing layer 54 corresponding to the display area 51 on the display 220. As a result, since the coordinates of object B do not change, the display position remains the same, and a new area is displayed (Figure 26(c)). The starting and ending coordinates of objects A to C after this process are shown in Table 9(c).

[0219] The user can also long-press object B with the electronic pen 2500 and swipe with their finger H, or long-press object B with the electronic pen 2500 and swipe with the electronic pen 2500, or long-press object B with their finger H and swipe with their finger H.

[0220] Furthermore, although only one object B is fixed in Figure 26, the user can fix multiple objects to the pinning layer 53 by using multiple fingers, for example.

[0221] <Operating Procedure> Figure 27 is an example of a flowchart illustrating the procedure by which the display device 2 displays an object fixed in the display area 51.

[0222] The contact position detection unit 12 determines whether or not an arbitrary object has been pressed and held by the hand H (S201). The contact position detection unit 12 determines whether the coordinates of contact made by the input means 291 overlap with the object and whether contact has been made for a certain period of time or longer.

[0223] If an object is pressed and held (Yes in S201), the drawing object management unit 11 copies the pixel data corresponding to the "Agenda List" (object B) in the drawing layer 54 to the pinning layer 53 and deletes the pixel data corresponding to the "Agenda List" (object B) in the drawing layer 54. In addition, the display control unit 14 outlines the frame of the "Agenda List" (object B) (S202).

[0224] The swipe operation detection unit 15, having detected that the hand H has come into contact with the display 220, determines whether or not a swipe by the electronic pen 2500 has been detected at any position on the display 220 (S203).

[0225] When it is being swiped, at every predetermined time (S204), the display control unit 14 repeatedly performs a process of subtracting the movement amount in the x-axis direction of the swipe from the x coordinate of the object in the drawing layer 54 and the movement amount in the y-axis direction of the swipe from the y coordinate (S205). By doing so, while receiving an operation to move the display area 51, the display control unit 14 can maintain the position of the selected object in the display area 51.

[0226] The display control unit 14 performs an OR operation on each pixel of the pinning layer 53 and the drawing layer 54 corresponding to the display area 51 and displays it on the display 220 (S206).

[0227] Next, the contact position detection unit 12 determines whether or not a long press by the hand H is continuously detected (S207).

[0228] When the long press ends (Yes in S207), the drawing object management unit 11 copies the pixel data corresponding to the "agenda list" (object B) in the pinning layer 53 to the drawing layer 54, and deletes the pixel data corresponding to the "agenda list" (object B) in the pinning layer 53. Then, the display control unit 14 displays each pixel of the drawing layer 54 corresponding to the display area 51 on the display 220 (S208).

[0229] <Main effects> According to this embodiment, by using two input means 291 by the user, the display area 51 can be moved while an object is being long pressed. Since there is no need to press a predetermined button or icon, intuitive operation becomes possible.

Example

[0230] In this example, a display system that shares object pinning by communicating between display devices installed at two bases will be described.

[0231] <System configuration example> Figure 28 is an example of a system configuration diagram of the display system 900. In Figure 28, display device 2A (an example of the first display device) is installed at site A, and display device 2B (an example of the second display device) is installed at site B. Display devices 2A and 2B are connected to each other via a network N such as the Internet.

[0232] Display devices 2A and 2B can share their operations and each display the same object.

[0233] <About the features> Figure 29 is an example of a functional block diagram showing the functions of display devices 2A and 2B in a block-like manner. Note that the explanation of Figure 29 will primarily focus on the differences from Figure 6.

[0234] In this embodiment, the display devices 2A and 2B are newly equipped with a remote conferencing control unit 18. When a user at site A inputs an operation to display device 2A to conduct a remote conferencing with display device 2B at site B (for example, by selecting display device 2B from a list and entering an ID and password), the remote conferencing control unit 18 of display device 2A establishes a conferencing session with display device 2B at site B. The remote conferencing control unit 18 mutually sends and receives operation details via the session.

[0235] For example, a user at site A hand-wrote the object shown in Figure 30 onto display device 2A. The display area 51 of display device 2A contains handwritten text: "○× Seminar" (object A), "List of Agenda Items" (object B), and "String related to equipment reservation" (object C).

[0236] The drawing data generation unit 13 of the display device 2A displays this handwritten data (coordinate point sequence, line attribute data such as line thickness and color) on the display 220 via the display control unit 14, and also passes it to the remote conferencing control unit 18. The remote conferencing control unit 18 transmits this handwritten data to the display device 2B via the network communication unit 16.

[0237] When the remote conferencing control unit 18 of the display device 2B receives this handwritten data via the network communication unit 16, the display control unit 14 displays it on the display 220. In this way, the same object is displayed on the display 220 of the display device 2A and the display 220 of the display device 2B.

[0238] <Pinning objects in remote meetings> Next, similar to Example 1, when the drawing object management unit 11 detects that the user has long-pressed any location on the "Agenda List" (Object B), the drawing object management unit 11 displays a pin button 9 for pinning the "Agenda List" (Object B).

[0239] Figure 31 shows the display area 51 where the pinning button 9 is displayed. Figure 31 is the same as Figure 9(b).

[0240] When the user presses the pin button 9 with the input means 291, the drawing object management unit copies the pixel data of the "Agenda List" (object B) from the drawing layer 54 to the pinned layer 53 and deletes the pixel data from the drawing layer 54.

[0241] The drawing object management unit 11 then passes the pixel data and origin coordinates of the "list of agenda items" (object B) to the remote conference control unit 18, which includes this data in a pinning command and transmits it to the display device 2B via the network communication unit 16.

[0242] When the remote conferencing control unit 18 of the display device 2B receives a pinning command via the network communication unit 16, the drawing object management unit 11 stores the pixel data of the received "list of agenda items" (object B) from the position of the received starting coordinates on the pinning layer 53. Then, the drawing object management unit 11 of the display device 2B deletes the pixel data of the "list of agenda items" (object B) from the position of the received starting coordinates on the drawing layer 54. In this way, the "list of agenda items" is also pinned to the pinning layer 53 on the display device 2B. Note that if the display devices 2A and 2B share the same object ID, the remote conferencing control unit 18 may send the object ID of the "list of agenda items" (object B) instead of sending the pixel data of the "list of agenda items" (object B) itself.

[0243] When the next agenda item (determining the timetable) from the "Agenda List" (Object B) is to be discussed, the user of display device 2A moves an object other than the pinned "Agenda List" (Object B) by swiping to take notes on this agenda item.

[0244] Figure 32 shows the display area 51 that has been freed up by the user moving an object other than the "Agenda List" (Object B). Figure 32 is the same as Figure 9(c).

[0245] The swipe operation detection unit 15 of the display device 2A passes the start coordinates (Ssx, Ssy) and end coordinates (Sex, Sey) of the swipe operation to the remote conferencing control unit 18, which then includes this data in a swipe command and transmits it to the display device 2B via the network communication unit 16.

[0246] When the remote conferencing control unit 18 of the display device 2B receives a swipe command via the network communication unit 16, the drawing object management unit 11 moves the object on the drawing layer 54 by the X-axis direction (Sex - Ssx) and the Y-axis direction (Sey - Ssy).

[0247] Then, the display control unit 14 of the display device 2B performs an OR operation on the pixel data in the drawing layer 54 and the pixel data in the pinning layer 53, and displays the result on the display 220. Therefore, the display device 2B displays the same display area 51 (object) as shown in FIG. 32 on the display 220.

[0248] <Operation Procedure> FIG. 33 is an example of a sequence diagram showing a procedure for sharing the pinning operation of an object between the display devices 2A and 2B.

[0249] S101: The remote conference control unit 18 of the display device 2A transmits a connection request (ID, password, etc.) to the display device 2B via the network communication unit 16.

[0250] S102: When the authentication for the connection request is successful, the remote conference control unit 18 of the display device 2B transmits a connection response (connection OK) to the display device 2A.

[0251] S103: The user of the display device 2A writes by hand using the input means 291. The drawing data generation unit 13 generates handwritten data and writes it into the drawing layer 54. In addition to the handwritten data, there are also character-recognized character strings, graphics, images, etc. Hereinafter, these are simply referred to as handwritten data.

[0252] S104: The handwritten data written in the drawing layer 54 is transmitted by the remote conference control unit 18 to the display device 2B via the network communication unit 16 at regular intervals.

[0253] S105: The remote conference control unit 18 of the display device 2B receives the handwritten data via the network communication unit 16, and the drawing data generation unit 13 writes it into the drawing layer 54. The display control unit 14 uses the handwritten data written in the drawing layer 54 to display an object on the display 220.

[0254] S106-S111: During the meeting, display devices 2A and 2B repeat the process in steps S103-S105. Note that handwritten data can also be sent from display device 2B to display device 2A.

[0255] S112: The user long-presses the "Agenda List" (Object B) to display the pin button 9, and then presses the pin button 9 with the input means 291. The drawing object management unit 11 detects the press of the pin button 9 based on the contact position of the input means 291 detected by the contact position detection unit 12.

[0256] S113: The drawing object management unit 11 copies the pixel data of the "Agenda List" (object B) from the drawing layer 54 to the pinned layer 53, and then deletes the pixel data from the drawing layer 54.

[0257] S114: The remote conference control unit 18 transmits the pixel data and origin coordinates of the "list of agenda items" (object B) to the display device 2B via the network communication unit 16, including them in the pinning command.

[0258] S115: When the remote conference control unit 18 of the display device 2B receives a pinning command via the network communication unit 16, the drawing object management unit 11 writes the pixel data of the received "list of agenda items" (object B) to the pinning layer 53 from the position of the received starting coordinates. The drawing object management unit 11 of the display device 2B also deletes the pixel data of the "list of agenda items" (object B) from the position of the received starting coordinates for the drawing layer 54.

[0259] S116: Next, the user of display device 2A moves an object other than the pinned "Agenda List" (Object B) by swiping.

[0260] S117: The swipe operation detection unit 15 of the display device 2A passes the start coordinates (Ssx, Ssy) and end coordinates (Sex, Sey) of this swipe operation to the remote conferencing control unit 18, and the remote conferencing control unit 18 transmits the start coordinates (Ssx, Ssy) and end coordinates (Sex, Sey) to the display device 2B via the network communication unit 16, including them in the swipe command.

[0261] S118: When the remote conferencing control unit 18 of the display device 2B receives a swipe command via the network communication unit 16, the drawing object management unit 11 moves the object in the drawing layer 54 by the X axis (Sex - Ssx) and the Y axis (Sey - Ssy). The display control unit 14 of the display device 2B performs an OR operation on the pixel data in the drawing layer 54 and the pixel data in the pinning layer 53 and displays it on the display 220.

[0262] <Adding objects to space> Next, the user of display device 2A handwrites text related to the "timetable" (object D) in the space created to the right of the "list of agenda items" (object B).

[0263] Figure 34 shows the display area 51 on which the "timetable" (object D) is handwritten. The drawing data generation unit 13 of the display device 2A displays this handwritten data (coordinate point sequence, line attribute data such as line thickness and color) on the display 220 via the display control unit 14 and also passes it to the remote conferencing control unit 18. The remote conferencing control unit 18 transmits this handwritten data to the display device 2B via the network communication unit 16. When the display device 2B receives this handwritten data, it displays it on the display 220 via the display control unit 14.

[0264] <Main effects> As described above, the display system 900 of this embodiment allows two or more display devices 2 participating in a remote conference to share object pinning and space creation. [Examples]

[0265] In this embodiment, a display system 900 is described in which display devices installed at two locations communicate with each other via a server device 60 to share object pinning.

[0266] <Example System Configuration> Figure 35 is an example of a system configuration diagram of the display system 900. The explanation of Figure 35 mainly describes the differences from Figure 28. In Figure 35, a server device 60 is connected to the network. The operation of display devices 2A and 2B is centrally managed by the server device 60, and display devices 2A and 2B can each display the same object by receiving images from the server device 60.

[0267] <Example of server hardware configuration> Figure 36 shows the hardware configuration of an example of a server device 60 according to this embodiment. As shown in Figure 36, the server device 60 is built by a computer and includes a CPU 501, ROM 502, RAM 503, HD 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, DVD-RW (Digital Versatile Disk Rewritable) drive 514, and media I / F 516.

[0268] Of these, the CPU 501 controls the operation of the entire server device 60. The ROM 502 stores programs used to drive the CPU 501, such as IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using network N. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 36.

[0269] The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor. The DVD-RW drive 514 controls the reading or writing of various data to the DVD-RW 513, which is an example of a removable recording medium. Note that the DVD-RW may be a DVD-R or the like. The media interface 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0270] <About the features> Figure 37 is an example of a functional block diagram illustrating the functions of the display device 2 in this embodiment, divided into blocks. The functions of display devices 2A and 2B may be the same. The explanation of Figure 37 mainly describes the differences from Figure 29. The display device in Figure 37 has an image decoding unit 19. The image decoding unit 19 decodes image data received from the server device 60. In this embodiment, the display device 2 does not have a pinning layer 53. This is because the server device 60 generates the image for the display area 51.

[0271] Figure 38 is an example of a functional block diagram that explains the functions of the server device 60 in a block-like manner. The server device 60 includes a drawing object management unit 21, a conference data storage unit 22, a drawing data generation unit 23, a network communication unit 26, a remote conference control unit 28, an image encoding unit 29, and an authentication control unit 30. Each function of the server device 60 is a function or means realized by any of the components shown in Figure 36 operating according to instructions from the CPU 501 in accordance with a program deployed on the RAM 503.

[0272] The drawing object management unit 21, conference data storage unit 22, drawing data generation unit 23, network communication unit 26, and remote conference control unit 28 may have the same functions as the display device 2.

[0273] The authentication control unit 30 authenticates the display devices 2A and 2B that are participating in the same meeting. The image encoding unit 29 encodes the image data that the server device 60 transmits to the display device 2.

[0274] <Operating Procedure> Figures 39A and 39B are examples of sequence diagrams illustrating the procedure for remote conferencing between display devices 2A and 2B via server device 60.

[0275] S121: Before conducting a remote conference, the same conference ID is set on display device 2A at site A and display device 2B at site B. This conference ID can be set by the user operating display devices 2A and 2B, or it can be set by using the equipment reservation system and sending the conference ID from the equipment reservation system to display devices 2A and 2B.

[0276] When a user at site A performs a meeting participation operation on display device 2A, display device 2A accesses server device 60 and transmits its account information (device ID and password) and meeting ID. Upon receiving the account information and meeting ID, server device 60 verifies the device ID and password stored in meeting data storage unit 22 and permits the connection of display device 2A. Then, if the meeting application process for the meeting ID received from display device 2A is not running, remote meeting control unit 28 starts the meeting application process for that meeting ID.

[0277] The same applies when a user at site B performs a meeting participation operation on display device 2B. However, since the meeting application process for the meeting ID received from display device 2B is already running, the remote meeting control unit 28 will have display device 2B join the meeting for this meeting ID. The remote meeting control unit 28 of server device 60 sends a meeting start message to display devices 2A and 2B via the network communication unit 26.

[0278] S122, S123: When the display devices 2A and 2B receive this message, the display control unit 14 displays the whiteboard (the screen of the display area 51) on the display 220.

[0279] S124: Users at site A hand-wrote "○× Seminar" (object A), "List of Agenda Items" (object B), and "Equipment Reservation List" (object C) on the display area 51 of the display device 2A, as shown in Figure 30. The drawing data generation unit 13 of the display device 2A converts this handwritten data (coordinate point sequence, line attribute data such as line thickness and color) into pixel data for display, writes it to the drawing layer 54, and the display control unit 14 displays it on the display 220.

[0280] S125: The remote conferencing control unit 18 of the display device 2A also transmits this handwritten data (coordinate point sequence, line attribute data such as line thickness and color) to the server device 60 via the network communication unit 16. The drawing data generation unit 13 of the display device 2A transmits the handwritten data to the server device 60 via the network communication unit 16 whenever the handwritten data reaches a predetermined amount.

[0281] S126: The network communication unit 26 of the server device 60 receives handwritten data, and the drawing data generation unit 23 converts the received handwritten data into pixel data for display and stores this pixel data in the drawing layer 54.

[0282] S127: The remote conferencing control unit 28 of the server device 60 then transmits the pixel data and the starting coordinates of the rectangular area (the bounding rectangle of the handwritten data) to the display device 2B via the network communication unit 26.

[0283] S128: When the remote conferencing control unit 18 of the display device 2B receives pixel data and the starting coordinates of a rectangular area via the network communication unit 16, the drawing data generation unit 13 writes the received pixel data to the drawing layer 54 from the position of the received starting coordinates. The display control unit 14 displays the handwritten data on the drawing layer 54 on the display 220. In this way, the same handwritten data is displayed on the displays 220 of the display devices 2A and 2B.

[0284] From this point onward, through the processing in steps S129 to S138, the display devices 2A and 2B share each stroke. Therefore, objects A to C are displayed on the displays 220 of the display devices 2A and 2B.

[0285] S139: The remote conferencing control unit 18 of the display device 2A transmits the starting and ending coordinates of objects A to C to the server device 60 via the network communication unit 16. Note that the processing in step S139 (object classification) may be performed by the server device 60.

[0286] S140: The remote conferencing control unit 28 of the server device 60 receives the starting coordinates and ending coordinates via the network communication unit 26, and the drawing object management unit 21 registers these received coordinate data in the object table.

[0287] Next, the pinning operation will be described. In this embodiment, the pinning operation and screen swipe operation will be described as being performed on the display device 2B. They may also be performed on the display device 2A.

[0288] S141: In the display device 2B, the contact position detection unit 12 detects that the user has pressed and held an arbitrary position on the "list of agenda items" (object B) using the input means 291.

[0289] S142: The drawing object management unit 11 of the display device 2B refers to the object table managed by the display device 2B and determines whether the pressed coordinates are within the object's area. Since these coordinates are not within the object's area managed by the display device 2B, the remote conferencing control unit 18 of the display device 2B sends the pressed coordinate data to the server device 60 via the network communication unit 16, including it in the pinning request command. This coordinate data identifies the object selected by the user.

[0290] S143: When the remote conference control unit 28 of the server device 60 receives a pinning request command via the network communication unit 26, the drawing object management unit 21 refers to the object table to determine whether these coordinates are within the object's area. Since these coordinates are within the area of ​​the "Agenda List" (object B), the drawing data generation unit 23 of the server device 60 reads the graphic data for the pinning button 9 for pinning the "Agenda List" from the HD504, converts it into pixel data for display, and overwrites the upper right position of the "Agenda List" (object B) on the drawing layer 54 with this pixel data.

[0291] S144: The remote conferencing control unit 28 then transmits the pixel data of the pin button 9 and the starting coordinates of the rectangular area circumscribing the pin button 9 to the display device 2B via the network communication unit 26.

[0292] S145: When the remote conferencing control unit 18 of the display device 2B receives pixel data including the pin button 9 via the network communication unit 16, the drawing data generation unit 13 overwrites the pixel data with the received starting coordinates of the drawing layer 54, and the display control unit 14 displays it on the display 220. As a result, the display device 2B is in the state shown in Figure 31.

[0293] S146: The remote conferencing control unit 28 of the server device 60 sends a "Pinning Ready" message to the display device 2B via the network communication unit 26.

[0294] S147: When the remote conferencing control unit 18 of the display device 2B receives this message via the network communication unit 16, the drawing object management unit 11 enters pinning operation mode.

[0295] S148, S149: When the user of the display device 2B presses the pin button 9 with the input means 291, the remote conferencing control unit 18 sends a pin execution command to the server device 60 via the network communication unit 16.

[0296] S150: When the remote conference control unit 28 of the server device 60 receives this message via the network communication unit 26, the drawing object management unit 21 copies the pixel data of the "list of agenda items" (object B) from the drawing layer 54 to the pinning layer 53 and deletes the pixel data from the drawing layer 54. The drawing object management unit 21 of the server device 60 also deletes the pixel data of the rectangular area circumscribing the pinning button 9.

[0297] S151: Next, the remote conference control unit 28 of the server device 60 sends the coordinate data, which was included in the pinning request command received in step S142 and pressed and held on the display device 2B, to the display device 2A via the network communication unit 26, along with a pinning notification message.

[0298] S152: When the remote conferencing control unit 18 of the display device 2A receives a message via the network communication unit 16, the display control unit 14 displays a pin button 9 (the same as shown in Figure 31) on the display 220 for a predetermined time at the coordinates included in the message. The display control unit 14 of the display device 2A also displays the message "Pinned at remote location" on the display 220 for a predetermined time.

[0299] Figure 40 shows an example of message 309 displayed on the display 220 of the display device 2A. A pin mark 310 is displayed on object B. This allows the user of the display device 2A to know that the object has been pinned at another location.

[0300] S153: Return to Figure 39. When the next agenda item (determining the timetable) from the "Agenda List" (Object B) is to be discussed, the user of the display device 2B moves an object other than the pinned "Agenda List" (Object B) by swiping to take notes on this agenda item. Therefore, the display area 51 becomes the same state as in Figure 32.

[0301] S154: The swipe operation detection unit 15 of the display device 2B detects the start of this swipe operation, and when the distance traveled by the contact position of the input means 291 reaches a predetermined value, the remote conferencing control unit 18 transmits the start coordinates of the swipe operation (Sx0, Sy0) and the current contact position (Sx1, Sy1) to the server device 60 via the network communication unit 16, including them in the swipe command.

[0302] S155: When the remote conferencing control unit 28 of the server device 60 receives this command via the network communication unit 26, the drawing object management unit 21 moves the objects in the drawing layer 54 by the X-axis direction (Sx1 - Sx0) and the Y-axis direction (Sy1 - Sy0). Then, the drawing data generation unit 23 of the server device 60 performs an OR operation on the pixel data in the drawing layer 54 corresponding to the display area 51 and the pixel data in the pinning layer 53.

[0303] S156: The image encoding unit 29 of the server device 60 compresses this pixel data into, for example, PNG format, and the network communication unit 26 transmits it to the display devices 2A and 2B.

[0304] S157,158: The remote conferencing control unit 18 of the display devices 2A and 2B receives PNG data via the network communication unit 16, and the image decoding unit 19 converts it into pixel data for display. The display control unit 14 displays the pixel data on the display 220.

[0305] S159~S164: While the user of the display device 2B is performing a swipe operation, the same processing as in steps S153~S158 is executed.

[0306] S165, S166: When the user of the display device 2B finishes the swipe operation, the network communication unit 16 sends a swipe command to the server device 60. This is to transmit the last contact position and the current contact position.

[0307] S167~S170: When the user of the display device 2B finishes the swipe operation and sends a swipe command to the server device 60, the same process as in steps S155~S158 is executed.

[0308] In this way, the displays 220 of the display devices 2A and 2B show the objects located in the display area 51 of Figure 32.

[0309] <Main effects> As described above, the display system 900 of this embodiment allows two or more display devices 2 participating in a remote conference to share object pinning and space creation, even when a server device 60 is present. [Examples]

[0310] In the following embodiment, another configuration example of the display device 2 will be described.

[0311] <<Another example of a display device configuration 1>> Although the display device 2 of this embodiment is described as having a large touch panel, the display device 2 is not limited to having a touch panel.

[0312] Figure 41 shows another example of the display device 2 configuration. In Figure 41, a projector 411 is installed on the top edge of the screen 413. This projector 411 corresponds to the display device 2. The screen can be a blackboard or any flat surface large enough to project an image.

[0313] The projector 411 has an ultra-short-throw optical system and can project a low-distortion image onto the screen 413 from a distance of about 10 cm. This image may be transmitted from a PC connected wirelessly or via a wired connection, or it may be stored in the projector 411's memory.

[0314] The user writes on the screen 413 using a dedicated electronic pen 2501. The electronic pen 2501 has a light-emitting part, for example at its tip, which switches on and emits light when the user presses it against the screen 413 to write. The wavelength of the light is near-infrared or infrared, so it is invisible to the user's eye. The projector 411 has a camera that captures an image of the light-emitting part and analyzes the image to determine the direction of the electronic pen 2501. In addition, the electronic pen 2501 emits sound waves along with the light, and the projector 411 calculates the distance based on the arrival time of the sound waves. The projector 411 can determine the position of the electronic pen 2501 based on its direction and distance. The handwritten data is drawn (projected) at the position of the electronic pen 2501.

[0315] The projector 411 projects the menu 430, so when the user presses a button with the electronic pen 2501, the projector 411 identifies the pressed button based on the position of the electronic pen 2501 and the ON signal of the switch. For example, when the save button 431 is pressed, the data (sequence of coordinate points) that the user has handwritten is saved by the projector 411. The projector 411 saves the handwritten information to a predetermined server 412 or USB memory 2600, etc. [Examples]

[0316] <<Another example of a display device configuration 2>> Figure 42 shows another example of the display device 2 configuration. In the example in Figure 42, the display device 2 includes a terminal device 600, an image projection device 700A, and a pen motion detection device 810.

[0317] The terminal device 600 is connected by wires to the image projection device 700A and the pen motion detection device 810. The image projection device 700A projects the image data input by the terminal device 600 onto the screen 800.

[0318] The pen motion detection device 810 communicates with the electronic pen 820 and detects the movement of the electronic pen 820 in the vicinity of the screen 800. Specifically, the pen motion detection device 810 detects coordinate information indicating the point that the electronic pen 820 is pointing to on the screen 800 (the detection method may be the same as in Figure 41) and transmits it to the terminal device 600.

[0319] The terminal device 600 generates image data of the handwritten data input by the electronic pen 820 based on coordinate information received from the pen motion detection device 810, and causes the image projection device 700A to draw the image of the handwritten data on the screen 800.

[0320] Furthermore, the terminal device 600 generates superimposed image data, which is a composite image obtained by combining the background image projected onto the image projection device 700A with the image of the handwritten data input by the electronic pen 820. [Examples]

[0321] <<Another example of a display device configuration 3>> Figure 43 shows an example configuration of the display device 2. In the example in Figure 43, the display device 2 includes a terminal device 600, a display 800A, and a pen motion detection device 810A.

[0322] The pen motion detection device 810A is positioned near the display 800A and detects coordinate information indicating the point pointed to by the electronic pen 820A on the display 800A (the detection method may be the same as in Figure 41), and transmits it to the terminal device 600. In the example of Figure 43, the electronic pen 820A may be charged by the terminal device 600 via a USB connector.

[0323] The terminal device 600 generates image data of the handwritten data input by the electronic pen 820A based on the coordinate information received from the pen motion detection device 810, and displays it on the display 800A. [Examples]

[0324] <<Another example of a display device configuration 4>> Figure 44 shows an example configuration of the display device 2. In the example in Figure 44, the display device 2 includes a terminal device 600 and an image projection device 700A.

[0325] The terminal device 600 communicates wirelessly with the electronic pen 820B (using Bluetooth®, etc.) to receive coordinate information of a point indicated by the electronic pen 820B on the screen 800. The coordinate information may be obtained by the electronic pen 820B reading minute positional information formed on the screen 800, or by receiving coordinate information from the screen 800.

[0326] Then, the terminal device 600 generates image data of the handwritten data input by the electronic pen 820B based on the received coordinate information, and projects the image of the handwritten data onto the image projection device 700A.

[0327] Furthermore, the terminal device 600 generates superimposed image data, which is a composite image of the background image projected onto the image projection device 700A and the handwritten data image input by the electronic pen 820B.

[0328] As described above, each of the embodiments described can be applied to various system configurations.

[0329] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0330] String data is stored as character codes in the display device 2, and handwritten data is stored as coordinate data. Furthermore, the data can be stored on various storage media or on a network storage device, and later downloaded from the display device 2 for reuse. The display device 2 used for reuse can be any display device, including a general-purpose information processing device. Therefore, users can reproduce their handwritten content on different display devices 2 and continue meetings, etc.

[0331] For example, although an electronic blackboard was described as an example in this embodiment, an electronic blackboard may also be called an electronic whiteboard, electronic information board, etc. Furthermore, this embodiment can be suitably applied to any information processing device having a touch panel. Examples of information processing devices equipped with a touch panel include projectors (PJ), output devices such as digital signage, head-up displays (HUD), industrial machinery, imaging devices, sound collection devices, medical equipment, networked home appliances, personal computers (notebook PCs), mobile phones, smartphones, tablet terminals, game consoles, personal digital assistants (PDAs), digital cameras, wearable PCs, or desktop PCs.

[0332] Furthermore, in this embodiment, the display device 2 detects the pen's coordinates by detecting the pen tip's coordinates via a touch panel, but the pen tip's coordinates may also be detected by ultrasound. The pen emits ultrasound along with light emission, and the display device 2 calculates the distance based on the arrival time of the ultrasound. The display device 2 can determine the pen's position based on its direction and distance. The projector draws (projects) the pen's trajectory as stroke data.

[0333] Furthermore, the configuration examples shown in Figure 6 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the display device 2. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the display device 2 can be further divided into many more processing units depending on the processing content. It can also be divided so that one processing unit includes even more processing.

[0334] Furthermore, some of the processing performed by the display device 2 may be performed by a server connected to the display device 2 via a network.

[0335] Furthermore, each function of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0336] The drawing object management unit 11 is an example of selection receiving means. The display control unit 14 is an example of display control means. The drawing data generation unit 23 is an example of image generation means. The drawing object management unit 11 is an example of object management means. Furthermore, the display control unit 14 in the display devices 2A and 2B is an example of the first display control means or the second display control means. The network communication unit 16 is an example of transmission means. The network communication unit 16 in the display devices 2A and 2B is an example of the first transmission means, and the network communication unit 26 in the server device 60 is an example of the second transmission means. [Explanation of symbols]

[0337] 2 Display device [Prior art documents] [Patent Documents]

[0338] [Patent Document 1] Japanese Patent Publication No. 2001-108475

Claims

1. A display device that displays a display area which is a portion of the input area which is the entire range in which a user can handwrite an object, A display control means for displaying the display area located at a first position within the inputable area on a screen, the display control means for displaying one or more objects handwritten by a user within the display area on the screen, The system includes a selection receiving means that accepts the selection of one or more of the displayed objects to be displayed in the same position as the original display area when the display area is moved in response to an operation to move the display area, The display control means moves the display area to the second position and displays it in response to an operation to move the display area from the first position to the second position. The aforementioned display device further, The object management means, in response to the operation, moves the position of the selected object relative to the inputable area to a position corresponding to the same position of the display area before the movement of the display area after moving from the first position to the second position, The display control means is characterized in that it causes the moved object to be displayed in the same position as the display area before the movement of the display area after the movement from the first position to the second position.

2. The display device according to claim 1, wherein the selection receiving means accepts the selection of the object pressed and held by the input means.

3. The display device according to claim 1, wherein the selection receiving means accepts the selection of the object enclosed by the stroke after a predetermined mark has been handwritten.

4. The display device according to claim 1, wherein the selection receiving means accepts the selection of an object when a predetermined mark is handwritten within a threshold distance from the object.

5. The display control means displays an operation menu, The display device according to claim 1, wherein the selection receiving means receives the selection of the object pressed by the input means after a predetermined button of the operation menu has been pressed.

6. The display control means displays an operation menu, The display device according to claim 1, wherein the selection receiving means accepts the selection of the object enclosed by the stroke after a predetermined button of the operation menu is pressed.

7. The display control means displays a first display component around the object pressed and held by the input means. The display device according to claim 2, wherein the selection receiving means accepts the selection of the object when the first display component is pressed.

8. The display control means displays an operation menu, After a predetermined button on the operation menu is pressed, the display control means moves the object selected by the input means to the position where the input means made contact. The display device according to claim 1, wherein the selection receiving means receives the selection of the moved object.

9. In a selection mode where object selection is possible using input means, When the selection receiving means receives a rectangular input from the input means, The display device according to claim 1, wherein the display control means moves the object selected by the input means to the rectangular area, and the selection receiving means accepts the selection of the moved object.

10. The selection receiving means receives the selection of the object pressed and held by the first input means, The display device according to claim 1, wherein the object management means moves the object in response to an operation to move the display area by the second input means when the object is pressed and held by the first input means.

11. After the aforementioned object has been moved, If the moved object is pressed and held by the input means, a second display component is displayed around the object. The selection receiving means deselects the object when the second display component is pressed. The display device according to any one of claims 1 to 7, wherein the position of the object relative to the display area is changed according to the amount by which the display area is moved.

12. A display system in which multiple display devices communicate with each other via a network, A first display device that displays a display area, which is a portion of the input area, which is the entire range in which a user can handwrite objects, A display control means for displaying the display area located at a first position within the inputable area on a screen, the display control means for displaying one or more objects handwritten by a user within the display area on the screen, The system includes a selection receiving means that accepts the selection of one or more of the displayed objects to be displayed in the same position as the original display area when the display area is moved in response to an operation to move the display area, The display control means moves the display area to the second position and displays it in response to an operation to move the display area from the first position to the second position. The first display device further includes, The object management means, in response to the operation, moves the position of the selected object relative to the inputable area to a position corresponding to the same position of the display area before the movement of the display area after moving from the first position to the second position, The display control means causes the moved object to display in the same position as the display area before the movement, after the movement from the first position to the second position. The system includes a transmission means for transmitting data relating to the object received by the selection receiving means, the coordinates of the object in the display area, and data relating to the movement of the object to a second display device. The second display device is, A second display control means that displays the object in the display area based on the coordinates of the object in the display area, which has been moved based on data relating to the movement of the object. A display system having the following features.

13. A display device that shows a display area which is a portion of the input area which is the entire range in which a user can handwrite objects, A display control means for displaying the display area located at a first position within the inputable area on a screen, the display control means for displaying one or more objects handwritten by a user within the display area on the screen, It functions as a selection receiving means that accepts the selection of one or more of the displayed objects to be displayed in the same position as the original display area in the moved display area when the display area is moved in response to an operation to move the display area. The display control means moves the display area to the second position and displays it in response to an operation to move the display area from the first position to the second position. The aforementioned display device is further, In response to the above operation, the object management means functions to move the position of the selected object relative to the input-enabled area to a position corresponding to the same position as the display area before the movement of the display area after moving from the first position to the second position. The display control means causes the moved object to display in the same position as the display area before the movement, after the movement from the first position to the second position. A program for that purpose.

14. A display method in which a display device displays a display area which is a part of the input area which is the entire range in which a user can handwrite an object, The display control means is a display control means that displays the display area located at a first position within the inputable area on a screen, and includes the steps of displaying one or more objects handwritten by a user within the display area on the screen, The selection receiving means receives a selection of one or more of the displayed objects to be displayed in the same position as the original display area in the moved display area when the display area is moved in response to an operation to move the display area. The display control means performs the steps of moving the display area from the first position to the second position and displaying it in response to an operation to move the display area from the first position to the second position, The object management means, in response to the operation, moves the position of the selected object relative to the inputable area to a position corresponding to the same position of the display area before the movement of the display area after moving from the first position to the second position, The display control means makes the moved object display in the same position as the display area before the movement, after the movement from the first position to the second position. A method for displaying with thumbnails.

Images