Device, display method, program, and display system

Abstract

To allow a device to more easily display a content outside the region of a display unit.SOLUTION: The present invention provides a device for displaying a content on a display unit. The device includes indicated-position detection means for detecting coordinates indicated by an indication object, and display control means for, if the coordinates detected by the indicated position detection means are outside the region of the display unit but within the range of a content including the detected coordinates, causing the display unit to display a part of a region where the content including the coordinates exists.SELECTED DRAWING: Figure 17

Description

[Technical Field]

[0001] The present invention relates to a device, a display method, a program, and a display system. [Background technology]

[0002] Some electronic whiteboards allow users to move handwritten data from a display unit to a memory area larger than the display area of ​​the display unit by swiping the data outside the display area. In addition, devices such as electronic whiteboards may require large content, such as architectural drawings. Because of limitations in the size and resolution of the device's display, displaying the entire content results in a very small and difficult-to-read display. For this reason, when meeting participants discuss one or more pieces of content, they first begin the discussion by displaying a portion of the area where the one or more pieces of content are located, and then display other areas as needed.

[0003] There is known a technique for displaying the position of a currently displayed area in the entirety of a content when the device is displaying a portion of the content (see, for example, Patent Document 1). Patent Document 1 discloses a technique for making it possible to grasp the area outside the displayed area by reducing and displaying an overhead screen area including the area outside the displayed area in the display area. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technologies have had the problem of requiring a large operational load for devices to display content outside the display area. For example, when discussing content outside the display area, the user must either zoom out the content and then zoom in on the desired area, or swipe to display the desired area without zooming out. The former method requires the user to zoom out once, while the latter method requires multiple swipes to display content far from the display area. Swiping requires the user to touch a touch panel, which requires the user to go to the display each time, resulting in a decrease in meeting efficiency.

[0005] In view of the above-mentioned problems, the present invention provides a technique that enables a device to more easily display content outside the area of ​​a display unit. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides an apparatus for displaying content on a display unit, which comprises: an indication position detection means for detecting coordinates pointed to by an indicating object; and a display control means for causing the display unit to display a portion of an area in which content including the detected coordinates is present when the coordinates detected by the indication position detection means are outside the area of ​​the display unit and within the range of content including the detected coordinates. [Effects of the Invention]

[0007] It is possible to provide a technique that enables a device to more easily display content outside the display area. [Brief explanation of the drawings]

[0008] [Figure 1] 10A and 10B are diagrams illustrating an example of content displayed by a device. [Figure 2] This is a diagram showing the positional relationship between range A and range B in an architectural drawing. [Figure 3] FIG. 10 is a diagram showing the direction in which a user points. [Figure 4]FIG. 10 is an image diagram of a process for switching the display area of ​​a device. [Figure 5] 10A and 10B are diagrams illustrating an example of a display of a device after a display area switching process is performed. [Figure 6] FIG. 1 is a diagram showing an example of equipment used in an actual conference room. [Figure 7] 1 is a diagram illustrating an overall configuration of an example of a communication system according to an embodiment of the present invention. [Figure 8] FIG. 2 is a diagram illustrating an example of a hardware configuration of a device. [Figure 9] FIG. 1 is an example of a functional block diagram illustrating the functions of a device divided into blocks. [Figure 10] FIG. 2 is a diagram illustrating object data stored in an object data storage means. [Figure 11] FIG. 10 is a diagram illustrating an example of the arrangement of finger photographing cameras. [Figure 12] 10A and 10B are diagrams illustrating an image (right captured image) captured by a finger photographing camera (right) and an image (left captured image) captured by a finger photographing camera (left). [Figure 13] 1 is a front view of an example of a display of a device. [Figure 14] 10 is a diagram illustrating an example of a range of finger-pointed position coordinates for which a device determines that it is necessary to switch the display area. [Figure 15] This is a diagram for explaining the finger pointing position coordinates T(x1, y1) with the upper left corner of the device as the origin (0, 0). [Figure 16] FIG. 10 is a diagram illustrating a display area switching process. [Figure 17] FIG. 10 is a flowchart illustrating an example of a process in which a user makes a gesture operation to display an area outside the display area in the display area. [Figure 18] FIG. 1 is an example of a functional block diagram illustrating the functions of a device divided into blocks. [Figure 19] FIG. 10 is a flowchart illustrating an example of a process for limiting the display area switching process for a certain period of time. [Figure 20]10 is an example of a flowchart illustrating a process for permitting the same user to switch display areas even within a certain period of time. [Figure 21] 10A and 10B are diagrams illustrating an outer region of content in the coordinate direction on the display pointed to by one finger. [Figure 22] FIG. 10 is a diagram illustrating the center coordinates Ta(xa, ya) after switching of the display area. [Figure 23] FIG. 10 is a flowchart illustrating an example of a process in which a user makes a gesture operation to display an area outside the display area in the display area. [Figure 24] This is an example of an image diagram when image data of a building blueprint is imported into the device as content to be used in a discussion. [Figure 25] FIG. 10 is a diagram showing content displayed by the device after a user performs an enlarged display operation. [Figure 26] FIG. 10 is a diagram illustrating an example of finger-pointing position coordinates. [Figure 27] 10A and 10B are diagrams showing a part of content displayed by a device after a display area switching process has been performed, together with the entire content. [Figure 28] FIG. 10 is a diagram showing an example of text data input to a device. [Figure 29] FIG. 10 is a diagram showing an example of a strategy input by a participant. [Figure 30] FIG. 10 is a diagram showing an example of an idea added by a meeting participant to a plan displayed by a device. [Figure 31] 10 is an example of a diagram showing the relationship between the overall mind map and the display area of ​​a device. [Figure 32] FIG. 10 is a diagram showing an example of finger-pointed position coordinates calculated in the overall mind map. [Figure 33] FIG. 10 is a diagram showing a display example after display area switching processing is performed. [Figure 34] FIG. 1 is a diagram illustrating an example of a system configuration of a display system. [Figure 35] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing system. [Figure 36]FIG. 2 is an example of a functional block diagram illustrating functions of the display system in blocks. [Figure 37] 10 is an example of a sequence diagram illustrating a process in which a device and an information processing system communicate with each other and display a recognition result of handwritten data. [Figure 38] FIG. 10 is a diagram showing an example of a pointing position image displayed on a display of a device. [Figure 39] FIG. 10 is a flowchart illustrating an example of a process for displaying an area outside the display area in the display area by a user's gesture operation. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, as an example of an embodiment of the present invention, a device and a display method performed by the device will be described with reference to the drawings.

[0010] <Terminology> The pointing object may be any object that can recognize a gesture operation by the device. For example, the pointing object may be a human hand, a pointer stick, a prosthetic hand, similar objects thereof, a humanoid robot, a non-humanoid robot, etc.

[0011] The information for detecting a pointing operation by a pointing object is information that the pointing object can indicate in order to detect the pointing operation. The information for detecting a pointing operation by a pointing object includes, for example, at least one of two-dimensional luminance information (image data) and a three-dimensional shape. In this embodiment, the pointing operation will be described using the term gesture operation.

[0012] A gesture is a type of body language expressed through bodily movements, and refers to gestures and hand movements. In this embodiment, a user can operate a device using gestures, and operating a device using gestures is referred to as gesture operation.

[0013] The term "device" refers to an electronic device that recognizes gesture operations and accepts the operations. The term "device" is sometimes used in contrast to an instrument or tool having a simple structure. In this embodiment, an electronic whiteboard may be used as an example of the device.

[0014] Content is the data displayed by the device. The data can be in JPEG, GIF, PDF, Word, Excel, 3D software project files, etc., and the content can be in any format as long as the device can display it. In this embodiment, the content is a PDF architectural drawing, but this is just an example.

[0015] The display unit may be any device that displays content, such as a display or a monitor, or may be a device that projects and displays content, such as a projector.

[0016] [First embodiment] First, we will explain a usage scenario in which a user displays content with a large display size on a device. The user enters a conference room to hold a meeting. Upon entering the conference room, the user performs an operation to display the content to be used in the meeting on the display of the device. As an example, the content is a PDF (file format does not matter) of architectural drawings created assuming A0 or A1 size. Furthermore, the size is not limited to A0 or A1, and this embodiment can be suitably applied when the device displays content larger than the display.

[0017] The device runs a cloud file sharing application and downloads content previously uploaded using a cloud service. Alternatively, the device may read content from external memory or send content to the device via wired or wireless connections from a terminal device such as a PC (Personal Computer). When the device is ready to display the content, it displays an overview of the content on its display.

[0018] FIG. 1 shows content 30 displayed in a reduced size by device 2. A user who is the conference host starts a conference in a conference room using content 30 displayed on device 2. The entire content 30 is displayed in the display area of ​​the display of device 2. The display area is the maximum range of pixels, excluding parts of the display that cannot display content 30 due to menus, etc. The display area is, for example, the area in which content 30 can be displayed. As shown in FIG. 1(a), the user and conference participants in the conference room can check the overall view of the architectural drawings by looking at device 2.

[0019] The user begins a discussion about a specific area (call it area A) on an architectural drawing. Currently, device 2 is displaying the entire architectural drawing, so to check the details of area A, it is necessary to enlarge area A. This enlargement does not involve enlarging the displayed data (such as pixel interpolation), but simply displaying the pixel data in memory that corresponds to area A as is. The user can display area A on the architectural drawing that they wish to discuss by double-clicking on the center of the area.

[0020] Figure 1(b) shows range A of architectural drawing data stored in memory. When the user double-clicks, range A, which is the subject of discussion, is displayed on the display, allowing the user to check the details of the architectural drawing. This allows the user to continue the discussion about range A while remaining seated and away from device 2.

[0021] Figure 2 shows the relative positions of area A and area B on an architectural drawing. The users finish discussing area A and then move on to discuss area B. Area B is located to the upper right of area A on the architectural drawing. Device 2 is currently enlarging area A, and area B is not visible on the display. Therefore, in order to display area B on the display of device 2, the user points to the area where area B, which is outside the display of device 2 while area A is displayed. This type of operation is called a gesture operation.

[0022] Figure 3 shows the direction in which the user points. Device 2 has a built-in or external finger-capturing camera that captures the area in front of device 2, including the user pointing at area B. The finger-capturing camera sends image data to device 2. Upon receiving the image data, device 2 processes the image and identifies the pointed-at position coordinates, with the upper left corner of the display area of ​​device 2's display as the origin (reference coordinates). The pointed-at position coordinates are expressed as the number of pixels in the x-axis direction and the number of pixels in the y-axis direction from the origin. The process of identifying the pointed-at position coordinates is performed on a two-dimensional plane that extends the display of device 2. Although the pointed-at position coordinates are in an area outside the display, they are expressed in the same coordinate system as within the display. After device 2 has completed the process of identifying the pointed-at position coordinates using the image data, it switches the display area to the area around the pointed-at position coordinates.

[0023] Figure 4 is an image diagram of the display area switching process for device 2. When the pointed-at position coordinates are (x1, y1), the display area switching process moves (x1, y1) to the current center position of the display of device 2. When device 2 completes the display area switching process, it displays range B, which corresponds to the pointed-at position on the architectural drawing, on the display of device 2.

[0024] Figure 5 shows an example of the display on device 2 after the display area has been switched. Range B is displayed on the display. Note that device 2 does not switch the display area for five seconds (this is just an example and can be set by the user) after switching to range B. This prevents another user from switching the display area by pointing at it immediately after a user switches the display by pointing at it.

[0025] In this way, in the device 2 of this embodiment, in order to display an area that is not currently displayed on the display, it is not necessary to first reduce the size of the content 30 so that the entire content is displayed, or to move the display area by swiping. Also, the user can display a desired display area on the display of the device 2 by performing a gesture operation while away from the device 2.

[0026] <System configuration example> FIG. 6 shows device 2 used in an actual conference room. In FIG. 6, device 2 is placed in a conference room, and participants are seated at a table in front of device 2. If participants could switch display areas or write using gesture operations while seated, applications would be expanded. For example, when a participant tries to explain a hidden drawing area by pointing and saying "top right of the drawing," the other participants may not understand, and the participant may have to move to device 2 and move the display area using a swipe operation. In this embodiment, participants can switch display areas while seated, reducing the need to move to device 2.

[0027] Fig. 7 is an overall configuration diagram of the communication system 1 of this embodiment. Note that, for the sake of simplicity, Fig. 7 shows only two devices 2a and 2b and accompanying electronic pens 4a and 4b, but three or more devices 2 and electronic pens may be used.

[0028] 7, the communication system 1 includes a plurality of devices 2a and 2b, a plurality of electronic pens 4a and 4b, USB memories 5a and 5b, notebook PCs 6a and 6b, video conference terminals 7a and 7b, and a PC 8. The devices 2a and 2b and the PC 8 are communicatively connected via a communication network 9. Furthermore, the devices 2a and 2b are provided with displays 3a and 3b with touch sensors, respectively.

[0029] Furthermore, the device 2a can display on the display 3a an image drawn by an event generated by the electronic pen 4a (touching the display 3a with the tip or bottom of the electronic pen 4a). Note that the image displayed on the display 3a can also be changed based on an event generated by touching the display 3a with not only the electronic pen 4a but also the user's hand Ha or the like (gesture operations such as zooming in, zooming out, and page switching).

[0030] A USB memory 5a can be connected to the device 2a, and the device 2a can read electronic files such as PDFs from the USB memory 5a and record electronic files to the USB memory 5a. The device 2a also has interfaces conforming to standards such as DisplayPort (registered trademark), DVI (Digital Visual Interface), HDMI (High-Definition Multimedia Interface, registered trademark), and VGA (Video Graphics Array). The user connects the device 2a to the notebook PC 6a using a cable 10a1 conforming to the standards.

[0031] The device 2a generates an event when the display 3a is touched, and transmits event information indicating this event to the laptop PC 6a in the same way as events from input devices such as a mouse or keyboard. Similarly, the device 2a is connected to a video conference terminal 7a via a cable 10a2 that is capable of communication according to the above standard. The laptop PC 6a and the video conference terminal 7a may communicate with the device 2a via wireless communication conforming to various wireless communication protocols such as Wi-Fi (registered trademark).

[0032] Meanwhile, at another location where device 2b is installed, similar to the above, device 2b equipped with display 3b, electronic pen 4b, USB memory 5b, notebook PC 6b, video conference terminal 7b, cable 10b1, and cable 10b2 are used. Furthermore, the image displayed on display 3b can be changed based on an event generated by a user's hand Hb touching display 3b.

[0033] As a result, an image drawn on the display 3a of device 2a at one location is also displayed on the display 3b of device 2b at another location, and conversely, an image drawn on the display 3b of device 2b at another location is displayed on the display 3a of device 2a at one location. In this way, communication system 1 is capable of performing remote sharing processing to share the same image at remote locations, making it extremely convenient when used for remote conferences, etc.

[0034] In the following, any one of the multiple devices 2 will be referred to as "device 2." Any one of the multiple displays will be referred to as "display 3." Any one of the multiple electronic pens will be referred to as "electronic pen 4." Any one of the multiple USB memory sticks will be referred to as "USB memory stick 5." Any one of the multiple notebook PCs will be referred to as "notebook PC 6." Any one of the multiple video conference terminals will be referred to as "video conference terminal 7." Any one of the multiple user's hands will be referred to as "hand H." Any one of the multiple cables will be referred to as "cable 10."

[0035] In addition, in this embodiment, an electronic whiteboard is described as an example of device 2, but the present invention is not limited to this, and other examples of device 2 may include an electronic billboard (digital signage), a telestrator (a technology that synthesizes handwritten text with an image displayed on a monitor) used in sports and weather forecasts, or a remote image (video) diagnostic device. Furthermore, device 2 may be a headset device such as VR (Virtual Reality) goggles, AR (Augmented Reality) goggles, or MR (Mixed Reality) goggles.

[0036] Furthermore, a notebook PC 6 will be described as an example of an external device, but the external device is not limited to this, and other examples of the external device may include a terminal capable of supplying image frames, such as a desktop PC, a tablet PC, a smartphone, a digital video camera, a digital camera, a game console, etc. Furthermore, the communication network includes the Internet, a LAN (Local Area Network), a mobile phone communication network, etc. Furthermore, in this embodiment, a USB memory will be described as an example of a recording medium, but the external device is not limited to this, and other examples of the recording medium may include various recording media such as an SD card.

[0037] <Hardware configuration example> Fig. 8 is a hardware configuration diagram of device 2. As shown in Fig. 8, device 2 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, an SSD (Solid State Drive) 404, a wired LAN controller 417, a network I / F 405, a wireless LAN controller 420, an antenna 421, and an external device connection I / F (Interface) 406.

[0038] Of these, the CPU 401 controls the overall operation of the device 2. The ROM 402 stores programs used to start the OS, such as an IPL (Initial Program Loader). The RAM 403 is used as a work area for the CPU 401. The SSD 404 stores various data, such as programs for the device 2. The wired LAN controller 417 controls communication with other devices connected to a communication network via a network I / F 405. The wireless LAN controller 420 executes a communication protocol compliant with the IEEE (Institute of Electrical and Electronics Engineers) 802.11ax standard and controls communication with a finger photographing camera (right) 471 and a finger photographing camera (left) 472 by transmitting and receiving radio waves via an antenna 421. The external device connection I / F 406 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory 430 and external devices (a microphone 440, a speaker 450, and a distance image sensor 460). These external devices may also be built-in.

[0039] The range image sensor 460 has a structure in which, for example, 500 pairs of infrared laser diodes and light receiving elements are arranged vertically and 500 horizontally, and measures distance from the time between when the infrared laser diodes emit light and when the reflected light is received. Therefore, the range image sensor 460 outputs 500 x 500 pieces of range data as one frame of image at a rate of 30 to 60 fps. The range image sensor 460 may be a stereo camera or LiDAR (Light Detection and Ranging). Note that the range image data may include at least one of distance data and image data. Hereinafter, the range image data may be simply referred to as image data.

[0040] Finger photographing camera (right) 471 and finger photographing camera (left) 472 are photographing devices that photograph the finger held out by the user to specify an area outside the display area of ​​display 3. As will be described later, finger photographing camera (right) 471 photographs the finger from the right side, and finger photographing camera (left) 472 photographs the finger from the left side. Note that finger photographing camera (right) 471 and finger photographing camera (left) 472 transmit the captured image data to device 2 via wireless LAN. Note that finger photographing camera (right) 471 and finger photographing camera (left) 472 may also transmit image data to device 2 via wired LAN or the like.

[0041] The device 2 also includes a capture device 411, a GPU 412, a display controller 413, a contact sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, an antenna 419a of the short-range communication circuit 419, a power switch 422, and selection switches 423.

[0042] Of these, the capture device 411 displays the display information of the external PC 470 as a still image or video. The GPU (Graphics Processing Unit) 412 is a semiconductor chip specialized in graphics. The display controller 413 controls and manages the screen display to output the output image from the GPU 412 to the display 3, etc. The contact sensor 414 detects contact with the display 3 by the electronic pen 4, the user's hand H, etc. The sensor controller 415 performs processing to identify the contact coordinates based on the signal from the contact sensor 414. The contact sensor 414 detects input coordinates using an infrared blocking method. This input coordinate detection is performed using two light-emitting and receiving devices installed at both ends of the upper side of the display 3. A light-emitting element (laser) in the light-emitting and receiving device rotates and scans an infrared beam parallel to the display 3 within a 90-degree range. The infrared beam is reflected by a reflecting member installed around the display 3. The light-receiving element receives the light returning along the same optical path as the emitted light. The light receiving and emitting device serving as the contact sensor 414 outputs the position (position on the light receiving element) where the infrared light is blocked by an object to the sensor controller 415, and the sensor controller 415 identifies the coordinates of the contact position of the object from these two pieces of position information. The electronic pen controller 416 determines whether the pen tip or the pen tail has touched the display 3 from data input by the short-range communication circuit 419 communicating with the electronic pen 4 via Bluetooth (registered trademark). The short-range communication circuit 419 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 422 is a switch for switching the power of the device 2 on and off. The selection switches 423 are a group of switches for adjusting, for example, the brightness and color of the display 3.

[0043] Furthermore, the device 2 includes a bus line 410. The bus line 410 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 401 shown in FIG.

[0044] The contact sensor 414 is not limited to an infrared blocking type, and may be a capacitive touch panel that identifies the contact position by detecting changes in capacitance. The contact sensor 414 may be a resistive film type touch panel that identifies the contact position by voltage changes across two opposing resistive films. The contact sensor 414 may be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by an object touching the touch panel. In addition, the contact sensor 414 may use various detection means. Furthermore, the electronic pen controller 416 may determine whether or not a touch has occurred not only on the tip and bottom of the electronic pen 4, but also on the part of the electronic pen 4 that the user holds or on other parts of the electronic pen.

[0045] <About the function> Next, the functions of device 2 will be described with reference to Fig. 9. Fig. 9 is a functional block diagram explaining the functions of device 2 in blocks. Device 2 has contact position detection means 11, handwriting data generation means 12, display control means 13, reception means 14, whiteboard control means 15, instruction position detection means 16, mode control means 17, gesture recognition means 18, network communication means 19, data recording means 20, first data acquisition means 21, second data acquisition means 22, and object data storage means 23.

[0046] The contact position detection means 11 converts the position touched by the pen or finger into coordinates. The writing data generation means 12 acquires the coordinates of the contact point of the tip of the pen or finger from the contact position detection means 11. The writing data generation means 12 acquires the coordinates pointed to by the user with one finger from the indication position detection means 16. The writing data generation means 12 generates a writing line by interpolating these coordinate point sequences. The writing line handwritten and input with the pen or finger is displayed on the touch panel of the display 3.

[0047] The display control means 13 displays the handwritten lines, text converted from the handwritten lines, and an operation menu for user operation on the display 3. The display control means 13 also displays a portion of the content present at the coordinates detected by the pointed position detection means 16 on the display 3. The portion of the content present at the coordinates refers to a portion of the content pointed at by the user. When the display area switching process is performed, the display control means 13 displays an area including the pointed position coordinates on the display 3.

[0048] The accepting unit 14 accepts a menu press based on the coordinates of the touch point of the pen or finger or the coordinates calculated by the pointed position detecting unit 16.

[0049] The whiteboard control means 15 performs overall control of the whiteboard application, such as starting up the whiteboard application, authentication, displaying menus, communicating with remote locations, and saving data.

[0050] The first data acquisition means 21 is a means for acquiring image data obtained by capturing an image of the user's hand and the device display 3. The user's hand is pointing at the device display 3. The first data acquisition means 21 acquires image data from the finger capturing camera (right) 471 and the finger capturing camera (left) 472.

[0051] The pointed position detection means 16 is a means for detecting coordinates pointed to by a gesture operation by the user. An example of a gesture operation is extending one finger. For example, the pointed position detection means 16 analyzes image data captured by the finger capturing camera (right) 471 and the finger capturing camera (left) 472 to detect coordinates pointed to by the user's gesture operation that are outside the area of ​​the display 3 and within the range of the content 30. Specifically, the pointed position detection means 16 analyzes image data acquired by the first data acquisition means 21 to determine the coordinates of the intersection between the extension of the one finger extended by the user and the surface of the display 3. This allows the user to switch the display area of ​​the display 3 by a gesture operation.

[0052] The mode control means 17 controls the transition of modes based on gesture operations recognized by the gesture recognition means 18. The mode transitions from the initial state to the gesture recognition mode, and from the gesture recognition mode to the initial state. Note that the mode itself may be transitioned by a menu operation instead of a gesture operation.

[0053] The second data acquisition means 22 is a means for acquiring information about the state of the user's hand from a sensor that acquires the information. The range image data may include information about the three-dimensional shape of the user's hand. The information about the state of the user's hand is at least one of brightness information and distance information (three-dimensional information). Specifically, the second data acquisition means 22 acquires range image data from the range image sensor 460.

[0054] The gesture recognition means 18 is a means for analyzing information about the state of the hand acquired by the range image sensor 460 and recognizing gesture operations. The information about the state of the hand includes at least one of an image and a three-dimensional shape. The gesture recognition means 18 extracts palm shape data from a range image captured by an image sensor of the palm, and stores this shape data on a hard disk. This palm photography is performed on many people, and the shape data of many palms is stored on a hard disk.

[0055] The gesture recognition means 18 first performs processing to recognize the palm of a hand. The gesture recognition means 18 detects a moving object from a distance image input from a distance image sensor, and determines that the object is a palm by comparing the shape data of the object with pre-stored palm shape data. This determination is made using a model that has learned shape data of many palms. When the gesture recognition means 18 recognizes a palm, the mode control means 17 transitions to gesture recognition mode.

[0056] When the gesture recognition means 18 transitions to the gesture recognition mode, it tracks the movement of the palm object and recognizes that this object has changed to a shape in which an index finger (hereinafter sometimes referred to as one finger) is extended. The gesture recognition means 18 determines that this is a gesture specifying a position on the display 3 or on the same plane as the display 3 where the finger is pointing. Furthermore, when the gesture recognition means 18 recognizes that the shape of the hand pointing with one finger has changed to a shape in which one finger is folded (a clenched hand), it determines that this is a gesture operation to end position specification. While the gesture recognition means 18 is recognizing the shape of the hand pointing with one finger, it requests the indication position detection means 16 to calculate the coordinates of the indication position.

[0057] The network communication means 19 is connected to the communication network 9 and performs data communication with other devices 2, any server device, and other external devices.

[0058] The data recording means 20 stores handwritten data written on the device 2, converted text, PC screens, files, etc. in the object data storage means 23.

[0059] FIG. 10 is a diagram illustrating the object data stored in the object data storage means 23. As shown in FIG. The object ID item is identification information that identifies the display data. The Type item indicates the type of object data, and includes handwriting, text, figure, image, table, etc. Handwriting is stroke data (a sequence of coordinate points). Text is one or more characters or symbols (character code) converted from handwritten data. Figures are registered in a menu and selected by the user, or are geometric shapes such as triangles and squares converted from handwritten data. Images are image data such as JPEG, PNG, and TIFF imported from a PC or the Internet. Tables are one-dimensional or two-dimensional table-like objects. The coordinates field indicates the position of the object data relative to a predetermined origin of the device 2. The position of the object data is, for example, the upper left vertex of the bounding rectangle of the object data. The coordinates are expressed in pixel units of the display 3, for example. The size item is the width and height of the bounding rectangle of the object data.

[0060] <Calculation of finger position coordinates> Next, calculation of the coordinates of the position of the user's finger on the display 3 will be described with reference to FIG. 11 and other figures. FIG. 11 is a diagram showing an example of the arrangement of finger-capturing cameras. Finger-capturing camera (right) 471 and finger-capturing camera (left) 472 are cameras for detecting the coordinates of the user's finger on the display 3. In FIG. 11, one finger-capturing camera (right) 471 and one finger-capturing camera (left) 472 are arranged on the table and around the table. The finger-capturing camera (right) 471 and the finger-capturing camera (left) 472 only need to be able to capture images of the area beyond the user's wrist and the display 3. Note that the finger-capturing camera (right) 471 and the finger-capturing camera (left) 472 may be cameras that can acquire distance information to the subject along with the image. This makes it easier for the gesture recognition unit 18 and the pointed position detection unit 16 to determine the state of the fingers (how many fingers there are), etc.

[0061] At least two finger-capturing cameras (right) 471 and finger-capturing cameras (left) 472 are required, and three or more may be provided. Device 2 may use two pieces of image data that are easy to recognize the hand from three or more pieces of image data, or may calculate the coordinates of the finger pointing by combining two pieces of image data extracted from all the image data and use the average. Furthermore, finger-capturing camera (right) 471 or finger-capturing camera (left) 472 may be a spherical camera.

[0062] Next, a description will be given of a process for displaying a pointer at the position on the display 3 pointed by the user. This process also uses image data received from the finger photographing cameras (right and left).

[0063] When the gesture recognition means 18 recognizes that the shape of the hand has changed to a shape with one finger extended in the pointer mode, pen mode, marker mode, or eraser mode, it transmits a command to start an operation to determine the position on the display 3 at which the finger is pointed to, to the pointed position detection means 16. Using a model generated by machine learning in the same way as the gesture recognition means 18, the pointed position detection means 16 extracts an object such as a hand from the right captured image and determines that the object is a hand with one finger extended.

[0064] When the pointed position detection means 16 receives this command, it starts an operation to find the coordinates of the point on the display 3 that is being pointed at, using image data received from the finger shooting camera (right) 471 and the finger shooting camera (left) 472 via the short-range communication circuit 419. The image (right captured image) captured by the finger shooting camera (right) 471 is shown in Fig. 12(a), and the image (left captured image) captured by the finger shooting camera (left) 472 is shown in Fig. 12(b).

[0065] 11, the device 2 has a display 3, but the display 3 may be external. In this case, a finger photographing camera (right) 471, a finger photographing camera (left) 472, and a distance image sensor 460 are arranged so as to be able to photograph the user, and these are connected to the information processing device. The information processing device and the external display 3 are connected by wire or wirelessly.

[0066] Next, the pointing position detection means 16 identifies the base (point P1 in FIG. 12(a)) and tip (point Q1 in FIG. 12(a)) of one finger from the right captured image. The pointing position detection means 16 also compares shape data of the object extracted from the right captured image with pre-stored shape data of the display 3 of the device 2 to determine that the object is the display 3 of the device 2. A model generated by machine learning may also be used for this determination. In the captured image of FIG. 12(a), points A1, B1, C1, and D1 indicate the upper left corner, lower left corner, upper right corner, and lower right corner of the display 3 of the device 2, respectively. In the captured image of FIG. 12(a), point Ei is the intersection of the extension of line segment P1Q1 and the extension of line segment A1B1. Furthermore, point Fi is the intersection of the extension of line segment P1Q1 and the extension of line segment C1D1.

[0067] The pointing position detection means 16 determines the hand with one finger extended from the left captured image in the same manner as above, and further identifies the base (point P2 in FIG. 12(b)) and tip (point Q2 in FIG. 12(b)) of this extended finger. Next, the pointing position detection means 16 determines the display 3 of the device 2 from the left captured image. In the captured image of FIG. 12(b), points A2, B2, C2, and D2 indicate the upper left corner, lower left corner, upper right corner, and lower right corner of the display 3 of the device 2, respectively. In the captured image of FIG. 12(b), point Gi is the intersection of the extension of line segment P2Q2 and line segment A2B2. Furthermore, point Hi is the intersection of the extension of line segment P2Q2 and line segment C2D2.

[0068] 13 is a front view of the display 3 of the device 2. The coordinate T of the position where the user is pointing on the display 3 of the device 2 corresponds to the intersection of the line connecting points Ei and Fi in the image captured by the finger-capturing camera (right) 471 and the line connecting points Gi and Hi in the image captured by the finger-capturing camera (left) 472. Since the coordinates in the image are based on the pixel position of the captured image, when displaying a pointer or the like at the part being pointed at, they need to be converted into coordinates based on the pixel position on the display 3 of the device 2.

[0069] 13 indicate the upper left corner, lower left corner, upper right corner, and lower right corner of the display 3 of the device 2, respectively. Points Ed and Fd are coordinate points based on the display pixel positions of the display 3 of the device 2, obtained by converting points Ei and Fi in the image (right captured image) captured by the finger capturing camera (right) 471 from coordinates based on the pixel positions of the captured image using the coordinate transformation matrix TR described below. Points Gd and Hd are coordinate points based on the display pixel positions of the display 3 of the device 2, obtained by converting points Gi and point Hi in the image (left captured image) captured by the finger capturing camera (left) 472 from coordinates based on the pixel positions of the captured image using the coordinate transformation matrix TL described below.

[0070] The coordinate transformation matrices TR and TL are calculated using the following formulas. If the coordinates of the upper left, lower left, upper right, and lower right corners of display 3 based on the pixel positions of the image (right captured image) captured by finger capture camera (right) 471 are A1(a1x, a1y), B1(b1x, b1y), C1(c1x, c1y), and D1(d1x, d1y), and the coordinates of the upper left, lower left, upper right, and lower right corners of display 3 based on the pixel positions of display 3 of device 2 are A3(a3x, a3y), B3(b3x, b3y), C3(c3x, c3y), and D3(d3x, d3y), a3x = (R11*a1x + R12*a1y + R13) / (R31*a1x + R32*a1y + 1) a3y = (R21*a1x + R22*a1y + R23) / (R31*a1x + R32*a1y + 1) b3x = (R11*b1x + R12*b1y + R13) / (R31*b1x + R32*b1y + 1) b3y = (R21*b1x + R22*b1y + R23) / (R31*b1x + R32*b1y + 1) c3x = (R11*c1x + R12*c1y + R13) / (R31*c1x + R32*c1y + 1) c3y = (R21*c1x + R22*c1y + R23) / (R31*c1x + R32*c1y + 1) d3x = (R11*d1x + R12*d1y + R13) / (R31*d1x + R32*d1y + 1) d3y = (R21*d1x + R22*d1y + R23) / (R31*d1x + R32*d1y + 1) The coordinate transformation matrix TR shown in equation (1) is obtained by the eight simultaneous equations above.

[0071]

number

[0072]

number

[0073]

number

[0074]

number

[0075]

number

[0076]

number

[0077] The coordinates of pixels on the display 3 are expressed as Cartesian coordinates, with the pixel position in the upper left corner of the display area as the origin, with the right direction being the positive direction of the X axis and the downward direction being the positive direction of the Y axis. In other words, display coordinates = (number of pixels from the origin in the X axis direction, number of pixels from the origin in the Y axis direction).

[0078] The device 2 of this embodiment calculates the coordinates of the pointed position outside the display 3, but also virtually enlarges the plane of the display 3 and expresses this virtual plane in display coordinates. There is a one-to-one correspondence between the pixels of the display 3 and the pixels in the memory of the drawing data to be displayed on the display 3. In other words, the device 2 can display the drawing data in the memory as is, without thinning or interpolating it.

[0079] <Switching display area> Next, the display control means 13 switches the display area of ​​the device 2 so that the determined pointed position coordinate T is at the center of the display 3. In this embodiment, the resolution of the display 3 is full HD 1920 × 1080. The resolution may also be 4K, 8K, etc.

[0080] 14 shows the range of pointed position coordinates for which the device 2 determines that it is necessary to switch the display area. The display area is switched only when the display 3 is displaying content that is larger than the display 3. First, the display control means 13 checks whether the pointed position coordinates T are outside the display area.

[0081] The display area is equal to the area of ​​content 30 displayed on display 3. In a coordinate system in which the upper left corner of device 2 is (0,0), the area displayed on display 3 is specified by two points: the upper left corner (0,0) and the lower right corner (1920,1080) of the display area displayed on display 3. If the pointed position coordinate T is included within this display area, it is determined that there is no need to switch the display area.

[0082] If the pointed position coordinate T is outside the display area of ​​the device 2, the display control means 13 next checks whether the pointed position coordinate T is within the range of the content 30. If the coordinates of the pointed position coordinate T (x1, y1) satisfy the following condition, it is determined that there is no need to switch the display area. 0 ≦ x1 ≦ 1920 and 0 ≦ y1 ≦ 1080 Therefore, when the user points at the hatched area in FIG. 14, the device 2 determines that the display area needs to be switched, and proceeds to the display area switching process.

[0083] 15 shows a case where the user points to pointing position coordinates T (x1, y1) outside the range of the display area. In order to display the device 2 with the pointing position coordinates T as the center, the display control means 13 determines the range of the display area centered on the pointing position coordinates T.

[0084] 16 is a diagram illustrating the display area switching process. The size of the display area is 1920 pixels wide and 1080 pixels high, so the coordinates of the upper left corner 51 of the display area centered on the pointed-to position coordinate T (x1, y1) are (x1-1920 / 2 pixels, y1-1080 / 2 pixels). The coordinates of the lower right corner 52 of the display area centered on the pointed-to position coordinate T are (x1+1920 / 2 pixels, y1+1080 / 2 pixels). From the above, the range of the display area is a rectangular area 53 specified by the two points: the coordinates of the upper left corner 51 (x1-960 pixels, y1-540 pixels) and the coordinates of the lower right corner 52 (x1+960 pixels, y1+540 pixels).

[0085] For the purpose of display area switching processing, display control means 13 copies image data of rectangular area 53 in content 30 to the display memory area of ​​display 3. Then, display control means 13 displays the image data copied to the display memory area on display 3, thereby realizing display area switching processing.

[0086] In this embodiment, the finger photographing camera (right) 471 and the finger photographing camera (left) 472 are always in operation. However, the finger photographing camera (right) 471 and the finger photographing camera (left) 472 may photograph only while the palm object is changing into the shape of one finger. For example, the gesture recognition means 18 recognizes that the palm object has changed into the shape of one finger, and transmits a start command to the pointed position detection means 16 to determine the position on the display 3 where the finger is pointed. When the pointed position detection means 16 receives this start command, the finger photographing camera (right) 471 and the finger photographing camera (left) 472 may start photographing.

[0087] <Action or Process> 17 is a flowchart illustrating a process in which a user uses a gesture operation to display an area outside the display area in the display area. At the start of FIG. 17, device 2 has already recognized the palm of the hand and is in gesture recognition mode.

[0088] It is determined whether or not the content 30 is located outside the display area of ​​the display 3 (S11). If the determination in step S11 is Yes, the process proceeds to step S12, and if No, the process repeats step S11.

[0089] The gesture recognition means 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of pointing one finger has been recognized (S12). If the determination in step S12 is Yes, the process proceeds to step S13; if No, the process repeats step S12.

[0090] In step S13, the gesture recognition means 18 requests the pointed position detection means 16 to calculate the coordinates of the position pointed by one finger (S13).

[0091] In response to this request, the pointing position detection means 16 analyzes the image data acquired by the first data acquisition means 21 from the finger photographing camera (right) 471 and the finger photographing camera (left) 472, and calculates the coordinates of the pointing position where one finger is pointing (S14).

[0092] The display control means 13 determines whether the coordinates of the pointed position are outside the display and within the range of the content 30 (S15). If the determination in step S15 is Yes, the process proceeds to step S16, and if No, the process in FIG. 17 ends.

[0093] In step S16, the display control means 13 acquires image data of 1920 pixels in the width direction and 1080 pixels in the height direction from the content 30, centered on the coordinates of the pointed position, and copies it to the display memory area. The display control means 13 displays the image data stored in the display memory area on the display 3 (S16). With the above steps, the area of ​​the content displayed in the display area of ​​the display 3 has been switched.

[0094] In the display area switching process, the device 2 may reduce the content 30 by a fixed factor when displaying a portion of the content 30 centered on the pointed-at position coordinates. Since the pointed-at position coordinates are not necessarily accurate, a wider range can be displayed on the display 3, making it easier for the user to view the desired content 30. If the content 30 is reduced and difficult to see, the user can enlarge it using a different gesture operation.

[0095] <Major Effects> In the device 2 of this embodiment, there is no need to reduce the size of the content 30 so that the entire content is displayed, or to continuously swipe in any direction to move the display area, in order to display details of an area not displayed on the display 3. Furthermore, the user can display a desired display area on the display 3 of the device 2 while being away from the device 2.

[0096] [Second embodiment] In this embodiment, a device 2 is described that, when a user points with one finger and the device 2 switches the display area, restricts other users from switching the display area for a certain period of time. This prevents multiple users from pointing at the display area and switching the content 30 displayed on the display 3 one after another.

[0097] For example, after the display control means 13 displays on the display 3 a part of the area where one or more contents exist at the coordinates detected by the pointed position detection means 16, for a certain period of time, even if the pointed position detection means 16 detects coordinates that are outside the area of ​​the display 3 indicated by the gesture operation and within the range where one or more contents exist, the display control means 13 does not change the area displayed on the display 3. Alternatively, the pointed position detection means 16 may be restricted from detecting coordinates for a certain period of time.

[0098] FIG. 18 is a functional block diagram for explaining the functions of device 2 by dividing them into blocks. The explanation of FIG. 18 will mainly focus on the differences from FIG. 9. Device 2 of this embodiment has timer measurement means 25. Timer measurement means 25 measures a certain period of time (for example, 5 seconds) when a display area switching process is executed by a gesture operation, and notifies display control means 13 that the certain period of time has elapsed. Other functions may be the same as those of FIG. 9, or may differ without interfering with the explanation of this embodiment. It is preferable that the certain period of time can be set by the user.

[0099] FIG. 19 is a flowchart illustrating a process for restricting the display area switching process for a certain period of time.

[0100] It is determined whether or not the display area switching process has been executed by a gesture operation (S21). If the determination in step S21 is Yes, the process proceeds to step S22, and if No, the process repeats step S21.

[0101] When the display area switching process is executed by a gesture operation, the timer measurement means 25 starts measuring a fixed time (S22).

[0102] For a certain period of time, even if the determination in step S15 of Fig. 17 is Yes, the display control means 13 does not accept a process for switching the display area (S23). That is, the display control means 13 fixes the display area displayed on the display 3 until the timer measurement means 25 notifies it. Note that the indication position detection means 16 can detect the coordinates pointed by one finger even before the certain period of time has elapsed. This makes it possible to display a pointer by a gesture operation, etc.

[0103] The display control means 13 determines whether a certain time has elapsed (S24). If the determination in step S24 is Yes, the process in Fig. 19 ends, and if No, the process returns to step S23.

[0104] <Major Effects> In addition to the effect of the first embodiment, the device 2 of this embodiment can restrict the switching of the display area by other users for a certain period of time.

[0105] [Third embodiment] In this embodiment, we will explain a device 2 that tracks the finger of a person pointing a finger and allows the display area to be switched even within a certain period of time if the gesture is performed by the same person. This allows the same user to switch the display area one after another, improving operability.

[0106] For example, even within a certain period of time after the display control means 13 has caused the display 3 to display part of the content at the coordinates detected by the pointing position detection means 16, the pointing position detection means 16 continues to detect coordinates outside the area of ​​the display 3 and within a range where one or more pieces of content exist, due to a gesture operation by the same user's hand. In this case, the display control means 13 causes the display 3 to display part of the content at the coordinates detected by the pointing position detection means 16.

[0107] In this embodiment, the functional block diagram shown in FIG. 18 can be used. The pointing position detection means 16 in this embodiment extracts a hand object from image data captured by the finger capturing camera (right) 471 and the finger capturing camera (left) 472, and stores feature amounts of the hand object when the display area is switched. The feature amounts may be the shape and color (histogram) of the hand object, SIFT feature amounts, etc. The pointing position detection means 16 tracks the same single finger based on the stored feature amounts such as shape data and color data of the hand pointing with one finger. Note that optical flow, etc. is known as a technology for tracking the same object. In other words, the pointing position detection means 16 tracks the hand object with one finger extended by the same user.

[0108] When the hand object moves out of the angle of view of the finger photographing camera (right) 471 or the finger photographing camera (left) 472, the pointing position detection means 16 again compares the feature amount when the hand object entered the angle of view with the stored feature amount of the hand object, and finally determines whether it is the hand object of the user who performed the display area switching process. At this time, the detection range for the hand object may be limited to a predetermined range centered on the position where the hand object was pointed before it moved out of the angle of view.

[0109] While the hand object of the user who performed the display area switching process is detected, the display control means 13 suspends the display area switching process by other users. After a certain period of time has elapsed, the indication position detection means 16 discards the feature amount of the hand object that performed the display area switching process and resumes the display area switching process by other users.

[0110] Note that even within a certain period of time, calculation of the coordinates of the pointing position of another user's finger is permitted, allowing other users to discuss by pointing at the display area with their pointer.

[0111] FIG. 20 is a flowchart illustrating a process for permitting the same user to switch the display area even within a certain period of time.

[0112] It is determined whether or not a display area switching process has been executed in response to the detection of a pointing finger (S31). If the determination in step S31 is Yes, the process proceeds to step S32, and if No, the process repeats step S31.

[0113] The timer measurement means 25 starts measuring a fixed time (S32).

[0114] In step S33, the pointed position detecting means 16 determines whether the timer has timed out. If the determination in step S33 is Yes, the process proceeds to step S35, and if No, the process proceeds to step S34.

[0115] In step S34, the indication position detection means 16 tracks the hand object that last performed the display area switching process, and does not accept any other display area switching process.

[0116] In step S35, the gesture recognition means 18 resumes the display area switching process by a user other than the user who performed the display area switching process. Note that the pointing detection process is performed at predetermined time intervals (for example, every second).

[0117] <Major Effects> According to this embodiment, in addition to the effects of the second embodiment, it is possible to restrict the display area switching process by other users for a certain period of time, and the same user can switch the display area one after another, improving operability.

[0118] [Fourth embodiment] In this embodiment, we will describe a device 2 that, when the coordinates on the display 3 pointed to with one finger are coordinates that are far beyond the range in which one or more contents exist, switches the display area to the edge area in the direction of that coordinate within the area in which one or more contents exist.

[0119] For example, if the indication position detection means 16 detects coordinates indicated by a user's gesture operation that are outside the area of ​​the display 3 and outside the range in which one or more contents exist, the display control means 13 causes the display 3 to display an area that has its center on the line connecting the origin of the display 3 and the detected coordinates and is at the edge of the area in which one or more contents exist.

[0120] 21 is a diagram illustrating an edge region 61 of the content 30 located in the coordinate direction on the display 3 pointed at by one finger. The process is the same as in the first embodiment up to the point where the device 2 determines the pointing position coordinate T on the display 3 where one finger is pointing. After this, the device 2 determines whether it is necessary to switch the display area. In this embodiment, it is assumed that the pointing position coordinate T is outside the area of ​​the display 3 and that the pointing position coordinate T is outside the range of the content 30. In this case, in the first embodiment, it is determined that there is no need to switch the display area and the process is terminated, but in this embodiment, the edge region 61 located in the direction of the pointing position coordinate T relative to the origin is displayed.

[0121] The position coordinates Tc of the pixel in the upper right corner of content 30 are (xc, yc). The center 62 of the display area after switching is on line 64. Display control means 13 determines edge area 61 where center 62 of the display area after switching is on line 64 connecting the coordinates (960, 540) of the center of the display area of ​​device 2 and the pointed position coordinate T, and one of the four sides of the display area after switching coincides with the outer frame 63 of content 30.

[0122] Let Te(xc, ye) be the intersection of line 64, which connects the coordinates (960, 540) of the center of device 2's display area with the pointed-to position coordinate T, and outer frame 63, which corresponds to the edge of content 30. Line 64 can be expressed as a linear function with a slope and b intercept. Note that in the calculation to find the center coordinate of the display area after switching using this linear function, the right direction on the X axis from the reference coordinate (the coordinate of the upper left corner of device 2's display area) is defined as the positive X coordinate direction, the left direction on the X axis is defined as the negative X coordinate direction, the upward direction on the Y axis is defined as the positive Y coordinate direction, and the downward direction on the Y axis is defined as the negative Y coordinate direction. Substituting the pointing position coordinate T(x2, y2) into this linear function, y2 = a*x2 + b …… (7) Substituting the coordinates (960, 540) of the center of the display area of ​​device 2 into this linear function, we get -540 = a*960 + b ……(8) This becomes: From equations (7) and (8), the slope a and intercept b are a = (-540 - y2) / (960 - x2) ……(9) b = y2 - ((-540 - y2) / (960 - x2))*x2 ……(10) This becomes: Since the intersection point Te is also a point on the above linear function, using equations (9) and (10), ye = a*xc + b ……(11) This becomes:

[0123] FIG. 22 is a diagram illustrating the center coordinates Tp(xp, yp) after switching the display area. The x coordinate of the center coordinates Tp is xc-960 based on the coordinates of the intersection point Te. When this x coordinate is substituted into the above linear function, the y coordinate yp' of the center coordinates Tp is given by yp'= a*(xc - 960) + b ……(12) Since this value is positive in the upward direction of the Y axis, the display coordinates of display 3 are yp = -a*(xc - 960) - b ……(13) a is calculated by equation (9), and b is calculated by equation (10).

[0124] As described above, the center coordinate Tp(xc-960, -a*(xc-960)-b) (a is equation (9), b is equation (10)) of the edge region 61 can be calculated, and thereafter, as in the first embodiment, the process of switching to a display region centered on the center coordinate Tp can be performed.

[0125] Fig. 23 is a flowchart illustrating a process in which a user uses a gesture operation to display an area outside the display area in the display area. The explanation of Fig. 23 will mainly focus on the differences from Fig. 17. The processes of steps S41 to S46 in Fig. 23 may be the same as steps S11 to S16 in Fig. 17. If the determination in step S45 is No, the pointing position detection means 16 determines whether the pointed position coordinates are outside the area of ​​the display 3 and outside the range of the content 30 (S47). If the determination in step S47 is Yes, the process proceeds to step S48; if No, the process in FIG. 23 ends.

[0126] In step S48, the display control means 13 identifies an edge area 61 of the content 30 in the direction connecting the origin and the pointed position coordinate T, copies it to the display memory area, and displays it.

[0127] In this embodiment, even if the user points outside the content 30, the edge area 61 is displayed on the display 3. However, if the pointing position coordinates are a certain distance or more from the outer edge of the content 30, the pointing position coordinates may be ignored. For example, if a user points at another user regardless of the content 30, the display area switching process can be prevented.

[0128] <Major Effects> According to this embodiment, in addition to the effects of the first embodiment, even if the coordinates of the position pointed by the user with one finger are outside the range of the content 30, it is possible to display an edge area 61 of the content 30 in the direction of the coordinates being pointed at. Because the user cannot see outside the display area, it is possible to reduce the burden on the user when displaying the edge portion of the content to be displayed.

[0129] [Fifth embodiment] In the first embodiment, image data larger in size than the display 3 is displayed, but in this embodiment, the size of the image to be displayed does not matter. In this embodiment, the system configuration shown in Fig. 7, the hardware block diagram shown in Fig. 8, and the function block diagram shown in Fig. 9 can be used.

[0130] In this embodiment, as in the first embodiment, image data of a building blueprint displayed by device 2 is imported, and a conference is assumed in which this image data is used for discussion. For the sake of explanation, a case in which image data of a size smaller than the size of display 3 is displayed will be described.

[0131] 24 is an illustration of what happens when device 2 imports image data of a building blueprint as content 30 to be used in a discussion. To discuss in detail a portion of content 30 displayed on device 2, the user enlarges content 30 so that the desired area is displayed.

[0132] FIG. 25 shows content 30 displayed by device 2 after the user performs an enlargement operation. The user enlarges content 30 from the state shown in FIG. 24 to the state shown in FIG. 25 to continue the discussion, and when the discussion is over, wishes to move on to discussing another part of content 30. To display another part of content 30 from the state shown in FIG. 25, a gesture operation is performed with one finger as in the first embodiment. When the user points a finger near the center of the range they wish to display, finger photographing camera (right) 471 and finger photographing camera (left) 472 photograph the user, and pointing position detection means 16 calculates the pointing position coordinate T of the pointing position of the single finger. FIG. 26 is a diagram showing the pointing position coordinate T.

[0133] When the device 2 calculates the pointed position coordinate T, it performs a process of switching the display area to the periphery of the pointed position coordinate T, similar to the first embodiment.

[0134] 27 shows a part of the content 30 displayed by the device 2 after the display area switching process has been performed, along with the entire content 30. The device 2 displays the content 30 with the center of the display 3 aligned with the pointing position coordinate T.

[0135] <Example of displaying handwritten data while swiping in a memory area larger than the display area> In this embodiment, the device 2 has a writable area 48 that extends beyond the range of the display 3. The display area displayed by the device 2 can be moved by swiping. This embodiment assumes a conference in which users discuss content 30, such as stamps, text input, or handwritten data input in various locations in the writable area 48 of the device 2.

[0136] A user (hereinafter referred to as User A) who is the meeting organizer gathers several people in a conference room and starts a meeting about improving business performance. User A decides to use a mind map to lead the meeting and generate new ideas by writing down problems and other issues radially around the topic of improving business performance. User A used the text input function to display the software keyboard in the blank display area and entered the text data "Improved performance."

[0137] Figure 28 shows text data 301 input into Device 2. The user inputs handwritten data 302 that supplements the text data 301. In Figure 28, handwritten data 302 "This year" is displayed. The participants in the meeting consider what approaches they can take to "improve business performance," and each writes down the ideas they come up with into Device 2.

[0138] Figure 29 shows an example of a plan 303 entered by a participant. User A solicits ideas related to the entered plan 303. As shown in Figure 30, meeting participants radially add ideas 304 and the like to the plan 303 displayed by device 2. Note that the ideas 304 may also include new keywords, topics, and the like.

[0139] When a certain number of ideas 304 are entered, they may not all fit in the display area of ​​device 2. In this case, user A swipes device 2 as shown in Figure 30 to move the display area and add notes. As user A and the participants move the display area while creating the mind map, a large diagram that does not fit in the display area of ​​device 2 is created.

[0140] Figure 31 shows the relationship between the overall mind map and the display area of ​​device 2. User A and the meeting participants consider whether there are relationships between the keywords on the mind map, and move on to a discussion to summarize ideas. User A thinks that there is a connection between "Collect customer data" 305 and "Conduct survey" 306 on the mind map. However, as shown in Figure 31, only "Collect customer data" 305 is displayed in the display area, and the existence of "Conduct survey" 306 cannot be indicated. User A, wanting to inform the meeting participants that the keyword "Conduct survey" 306 exists, points with one finger at the area where he thinks the keyword "Conduct survey" 306 is written.

[0141] 32 shows the calculated pointed position coordinate T in the overall mind map. When the pointed position coordinate T is calculated, the device 2 performs processing to switch the display area to the periphery of the pointed position coordinate T, as in the first embodiment.

[0142] Fig. 33 shows an example of a display after the display area switching process has been performed. In Fig. 33, "Take a Survey" 306 is displayed in the display area of ​​device 2 as a result of the display area switching process. Device 2 displays "Take a Survey" 306 in the display area by pointing and aligning the center of display 3 with position coordinate T. In this way, user A can use a gesture operation to display content 30 that is not displayed in the display area of ​​device 2, which can be useful for exchanging opinions.

[0143] [Sixth embodiment] In this embodiment, a display system will be described in which an information processing system on a network calculates the coordinates of the pointed position, and the device 2 performs a process of switching the display area.

[0144] In the description of this embodiment, components or figures with the same symbols as in the first embodiment perform similar functions, so descriptions of components that have already been described may be omitted or only differences may be described.

[0145] 34 is an example of a system configuration diagram of a display system 100. The display system 100 has a device 2 and an information processing system 70 that can communicate with each other via a network N.

[0146] The device 2 is located in a facility such as a company, and is connected to a LAN or Wi-Fi installed within the facility. The information processing system 70 is located, for example, in a data center. The device 2 is connected to the Internet i via a firewall 8, and the information processing system 70 is also connected to the Internet i via a high-speed LAN or the like within the data center.

[0147] The device 2 may connect to the Internet i using wireless communication such as a mobile phone network, in which case the wireless communication may be 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), or the like.

[0148] The information processing system 70 has one or more information processing devices, which act as servers to provide services to the device 2. A server is a computer or software that provides information and processing results in response to requests from clients. As will be described later, the information processing system 70 receives range image data and image data from the finger capturing camera from the device 2 and transmits the pointed position coordinates to the device 2.

[0149] The configuration of the device 2 may be similar to that of the first embodiment, but in this embodiment, it is sufficient that the device 2 has a touch panel, a display 3, and a communication function. The device 2 may include multiple computing devices configured to communicate with each other.

[0150] In this embodiment, a general information processing device such as a PC or tablet can execute a web browser or dedicated application. The web browser or dedicated application communicates with an information processing system 70. When the web browser is running, the user inputs or selects the URL of the information processing system 70 to connect the device to the information processing system 70. The device 2 executes a web application provided by the information processing system 70 in the web browser. A web application refers to software or a mechanism executed on a web browser that operates through cooperation between a program written in a programming language (for example, JavaScript (registered trademark)) that runs on the web browser and a program on the web server side.

[0151] When the dedicated application runs, it connects to a pre-registered URL of the information processing system 70. The dedicated application has a program and a user interface, so the program sends and receives necessary information to and from the information processing system 70 and displays it on the user interface.

[0152] The communication method may be a general-purpose communication protocol such as HTTP, HTTPs, or WebSocket, or a dedicated communication protocol.

[0153] <Hardware configuration example> The hardware configuration of the device 2 may be the same as that shown in Fig. 8. In this embodiment, an example of the hardware configuration of an information processing system 70 will be described.

[0154] Fig. 35 is a hardware configuration diagram of the information processing system 70. As shown in Fig. 35, the information processing system 70 is constructed by a computer, and includes a CPU 601, a ROM 602, a RAM 603, an HD 604, an HDD (Hard Disk Drive) controller 605, an external device connection I / F (Interface) 608, a network I / F 609, a bus line 610, and a media I / F 616.

[0155] Of these, the CPU 601 controls the overall operation of the information processing system 70. The ROM 602 stores programs used to drive the CPU 601, such as the IPL. The RAM 603 is used as a work area for the CPU 601. The HD 604 stores various data, such as programs. The HDD controller 605 controls the reading and writing of various data from and to the HD 604 under the control of the CPU 601. The external device connection I / F 608 is an interface for connecting various external devices. In this case, the external devices are, for example, USB (Universal Serial Bus) memories and printers. The network I / F 609 is an interface for data communication using a communication network. The bus line 610 is an address bus, a data bus, or the like for electrically connecting the components, such as the CPU 601, shown in FIG. 35.

[0156] A media I / F 616 controls reading and writing (storing) of data from and to a recording medium 615 such as a flash memory.

[0157] <About the device functions> Next, functions of the display system 100 will be described with reference to Fig. 36. Fig. 36 is an example of a functional block diagram showing functions of the display system 100 in blocks. Note that the description of Fig. 36 will mainly focus on the differences from Fig. 9.

[0158] In this embodiment, the device 2 has a contact position detection means 11, a writing data generation means 12, a display control means 13, a reception means 14, a whiteboard control means 15, a network communication means 19, a data recording means 20, a first data acquisition means 21, a second data acquisition means 22, and an object data storage means 23.

[0159] <<Functions of the information processing system>> The information processing system 70 has a communication means 24, an indication position detection means 16, a mode control means 17, and a gesture recognition means 18. Each function of the information processing system 70 is a function or means realized when any of the components shown in Fig. 35 operates in response to an instruction from the CPU 601 in accordance with a program loaded from the HD 604 onto the RAM 603.

[0160] The communication means 24 receives distance image data and image data from the finger photographing camera from the device 2, and transmits the pointed position coordinates and the like to the device 2. Other functions are the same as those in the first embodiment, or, even if they are different, they will not hinder the explanation of this embodiment.

[0161] <<Action or Process>> FIG. 37 is a sequence diagram illustrating a process in which the device 2 and the information processing system 70 communicate with each other to display the area pointed at by the user.

[0162] S101: First, the user causes the display 3 to display a portion of an area in which one or more contents exist.

[0163] S102: Next, the user points to the outside of the display 3 with one finger.

[0164] S103: The second data acquisition means 22 acquires distance image data from the distance image sensor 460. The network communication means 19 transmits the distance image data to the information processing system .

[0165] S104: The communication means 24 of the information processing system 70 receives the range image data, and the gesture recognition means 18 analyzes the range image data. The gesture recognition means 18 determines whether or not a gesture operation of pointing one finger has been recognized. Here, it is assumed that a gesture operation of pointing one finger has been recognized.

[0166] S105: The communication means 24 requests the device 2 for image data from the finger photographing camera.

[0167] S106: The network communication means 19 receives this request and transmits the image data of the finger photographing camera (right) 471 and the finger photographing camera (left) 472 acquired by the first data acquisition means 21 to the information processing system 70.

[0168] S107: The communication means 24 of the information processing system 70 receives the image data, and the pointing position detection means 16 calculates the coordinates of the position where one finger is pointing. The pointing position detection means 16 determines whether the coordinates of the pointing position are outside the display and within the range where the content 30 is present. Here, it is assumed that the coordinates of the pointing position are outside the display and within the range where the content 30 is present.

[0169] S108: The communication means 24 of the information processing system 70 transmits the coordinates of the pointed position to the device 2.

[0170] S109: The network communication means 19 of the device 2 receives the pointed-at position coordinates, and the display control means 13 acquires image data of 1920 pixels in the width direction and 1080 pixels in the height direction from the area where the content 30 exists, centered on the pointed-at position coordinates, and copies it to the display memory area. The display control means 13 displays the image data stored in the display memory area on the display 3. With this, the content displayed in the display area of ​​the display 3 has been switched.

[0171] <Major Effects> According to this embodiment, in addition to the effects of the first embodiment, if the device 2 is connected to the network, it is possible to display content outside the display 3 by a gesture operation. Note that although the first embodiment has been described in this embodiment, the second to fourth embodiments can also be suitably applied.

[0172] [Seventh embodiment] In this embodiment, we will explain a device 2 that displays a reduced image of the entire content 30 and a message indicating that the content does not exist in a display area displayed based on the coordinates pointed by the user when the display area is not in a range where the content exists.

[0173] In the description of this embodiment, components or figures with the same symbols as in the first embodiment perform similar functions, so descriptions of components that have already been described may be omitted or only differences may be described.

[0174] FIG. 38 shows the display of device 2 on which a finger-pointing position image 311 is displayed. Finger-pointing position image 311 is an image that indicates the range of the entire content 30 that is displayed in the display area when the user points at it. The coordinates on display 3 pointed at by the user with one finger may be coordinates that are far beyond the range in which content 30 exists. In this case, device 2 switches to a display area centered on the coordinates pointed at with one finger, and then displays a reduced image of the entire content 30, along with a message 312 indicating that the content is outside the range. Coordinates that are far beyond the range in which content 30 exists mean that no pixels of content 30 exist in the display area.

[0175] This determination is the same as in the fourth embodiment up to the point that the pointing position coordinate T is outside the area of ​​the display 3 and outside the range of the content 30. In the fourth embodiment, an edge area 61 located in the direction of the pointing position coordinate T with respect to the origin is displayed, but in this embodiment, as in the first embodiment, the pointing position coordinate T is moved to the center position of the display of the device 2.

[0176] This is conditional on the condition that no pixels of content 30 are present in the display area. Therefore, the display area is blank. When a blank display area is displayed, device 2 displays a finger-pointing position indicator image 311 in the upper left (or upper right, etc.) of the display. Pointing position indicator image 311 includes a thumbnail image showing the entire content and a thumbnail image (rectangular frame 313) of the display area currently displayed on the display. Device 2 also displays a message 312 indicating that the display area is outside the area of ​​content 30. Rectangular frame 313 allows the user to understand which parts of the content 30 are displayed in the display area, even if none of content 30 is displayed.

[0177] The pointed position image 311 and the message 312 indicating that the content 30 is outside the area are hidden when the display position is changed by handwriting input or another operation such as pointing to the display 3. The images are also hidden after a predetermined time (for example, 5 seconds) has elapsed even if the user does not perform any operation.

[0178] Fig. 39 is a flowchart illustrating a process in which a user uses a gesture operation to display an area outside the display area in the display area. The explanation of Fig. 39 will mainly focus on the differences from Fig. 23. Steps S49 and S50 have been added to Fig. 39.

[0179] If the determination in step S47 is Yes, the display control means 13 determines whether or not the display area centered on the pointed-at position coordinates includes pixels of the content 30 (S49). That is, the display control means 13 determines whether or not pixels of the content exist within a range of 1920 pixels horizontally and 1080 pixels vertically centered on the pointed-at position coordinates.

[0180] If the determination in step S49 is Yes, the process proceeds to step S50, and if No, the process proceeds to step S48. The process in step S48 is the same as that in FIG.

[0181] In step S50, the display control means 13 displays, in the blank display area, a pointing position illustration image 311 and a message indicating that the display is outside the content area 312. A rectangular frame 313 indicating the display area currently displayed on the display is displayed in the pointing position illustration image 311.

[0182] <Major Effects> According to this embodiment, in addition to the effects of the first embodiment, even if the coordinates of the position pointed by the user with one finger are far outside the range of the content 30, a pointed position image 311 and a rectangular frame 313 indicating the display area currently displayed on the display are displayed. Even if the content 30 is not displayed at all, the user can understand which part of the area around the content is displayed in the display area.

[0183] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0184] For example, in the above embodiment, the device 2 transitions to the gesture recognition mode by recognizing the palm of the hand in the initial state, but the method of transitioning to the gesture recognition mode is not limited to this, and the device 2 may transition to the gesture recognition mode by recognizing a predetermined shape of an arbitrary object or a predetermined movement of an arbitrary object.

[0185] For example, in this embodiment, device 2 calculates the pointing position coordinates by analyzing image data captured by finger capturing camera (right) 471 and finger capturing camera (left) 472, but device 2 may also calculate the pointing position coordinates by analyzing distance image data captured by distance image sensor 460. One way to achieve this is to use a model that learns the correspondence between distance image data and pointing position coordinates by deep learning or the like.

[0186] The content may also be a video. In this case, the image of the range including the coordinates of the pointed position also changes over time, but the device 2 only needs to display the range including the coordinates of the pointed position on the display 3.

[0187] Furthermore, device 2 may be called an electronic whiteboard, an electronic information board, or the like. This embodiment is not limited to electronic whiteboards and can be suitably applied to any information processing device having a touch panel. Examples of information processing devices equipped with a touch panel include PCs, tablet terminals, and smartphones that have touch panels. These are normally general-purpose information processing devices, but when an application that functions as a device is executed, the user can operate them as device 2.

[0188] Alternatively, the device 2 may be a device that projects an image using a projector. In this case, the device 2 detects the coordinates pointed to by the finger using the method described in this embodiment, and the projector projects an area including the coordinates.

[0189] Furthermore, in this embodiment, the device 2 recognizes the gesture operation and calculates the coordinates where the user points with one finger. However, at least one of the recognition of the gesture operation and the calculation of the coordinates may be performed by a server device connected via a network. In this case, the device 2 transmits image data captured by the finger capturing camera (right) 471 and the finger capturing camera (left) 472 and distance image data captured by the distance image sensor 460 to the server device in real time. The server device transmits the recognized gesture operation and the calculated coordinates to the device 2. This reduces the processing load on the device 2.

[0190] Furthermore, in this embodiment, the device 2 calculates the coordinates pointed to by the index finger, but it may be pointed to by another finger.

[0191] In addition, the configuration examples in Figure 9 and the like are divided according to main functions to make it easier to understand the processing by device 2. The method of dividing the processing units or the names of the processing units does not limit the present invention. The processing by device 2 can also be divided into more processing units depending on the processing content. Also, it can be divided so that one processing unit includes more processes.

[0192] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0193] Embodiments of the present invention provide significant improvements in computer power and functionality. These improvements allow users to utilize computers that provide more efficient and robust interaction with tables, which are ways of storing and presenting information in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of more efficient, powerful, and robust user interfaces. Such user interfaces provide better interaction between humans and machines. [Explanation of symbols]

[0194] 2 equipment [Prior art documents] [Patent documents]

[0195] [Patent Document 1] Japanese Patent Publication No. 2022-143264

Claims

1. In a device that displays content on a display, a pointing position detecting means for detecting coordinates pointed to by a pointing object; and a display control means for causing the display unit to display a portion of the area in which content including the coordinates is present when the coordinates detected by the indication position detection means are outside the area of ​​the display unit and within the range of content including the detected coordinates.

2. For a certain period of time after the display control means causes the display unit to display a part of an area where content including the coordinates detected by the pointing position detection means is present, even if the pointed position detection means detects coordinates pointed to by the pointing object that are outside the area of ​​the display unit and within the range of an area in which content including the coordinates exists, the display control means does not change the part of the area in which content including the coordinates exists that is displayed on the display unit.

10. The device of claim 1.

3. a first data acquisition means for acquiring image data obtained by capturing an image of the pointing object and a display unit of the device; the pointing position detection means tracks the pointing object based on the feature amount of the pointing object acquired by the first data acquisition means; Even within a certain time after the display control means has caused the display unit to display a part of an area where content including the coordinates detected by the pointing position detection means exists, When the pointing position detection means detects coordinates outside the area of ​​the display unit and within the range of an area where content including the coordinates exists, by the same pointing object as the pointing object that has been tracked, the display control means causes the display unit to display a part of an area where the content exists, the area including the coordinates detected by the pointing position detection means.

3. The device of claim 2.

4. When the pointing position detection means detects coordinates that are pointed to by the pointing object and are outside the area of ​​the display unit and outside the range of an area in which content including the coordinates exists, the display control means causes the display unit to display an area having a center on a line connecting the center of the display unit and the coordinates, and located at an edge of an area where content including the coordinates exists; The device according to any one of claims 1 to 3.

5. If the pixel of the content does not exist in a display area centered on the coordinates pointed to by the pointing object, as detected by the pointing position detection means, The display control means displays, on the display unit, a reduced image of the entire content and a frame indicating the range of the entire content that is displayed on the display unit, and further displays a message that the display area is outside the range of the content. The device according to any one of claims 1 to 3.

6. A device that displays content on an external display, a pointing position detecting means for detecting coordinates pointed to by a pointing object; and a display control means for causing the display unit to display a portion of the area in which content including the coordinates is present when the coordinates detected by the indication position detection means are outside the area of ​​the display unit and within the range of content including the detected coordinates.

7. A display method performed by a device that displays content on a display unit, comprising: a process in which a pointed position detecting means detects coordinates pointed to by a pointing object; a process in which, when the coordinates detected by the indication position detection means are outside the area of ​​the display unit and within the range of content including the detected coordinates, a display control means displays, on the display unit, a part of the area in which content including the coordinates exists; Display method.

8. A device that displays content on the display, a pointing position detecting means for detecting coordinates pointed to by a pointing object; a display control means for, when the coordinates detected by the pointing position detection means are outside the area of ​​the display unit and within the range of the content including the detected coordinates, causing a part of the area where the content including the coordinates exists to be displayed on the display unit; A program to function as a

9. A display system in which a device that displays content on a display unit and an information processing system communicate with each other via a network, The device comprises: a display control means for displaying the content on a display unit; a first data acquisition means for acquiring information for detecting a pointing operation using a pointing object; a network communication means for transmitting the information to the information processing system; The information processing system includes: a pointing position detection means for analyzing the information received from the device and detecting coordinates indicated by a pointing operation using the pointing object; a communication means for transmitting the coordinates to the device when the coordinates detected by the pointing position detection means are outside the area of ​​the display unit and within the range of content including the detected coordinates, The device is a display system that displays, on the display unit, a part of an area where content including the coordinates received from the information processing system exists.

Images