Device, gesture recognition method, program, and display system

Abstract

To reduce the number of gesture operations.SOLUTION: A device is configured to recognize gesture operation corresponding to the motion of a pointing object and receive an operation according to the gesture operation, the device including: a data acquisition unit which acquires information from a sensor that acquires information on the shape of the pointing object; and a gesture recognition unit which recognizes, based on the information acquired by the data acquisition unit, the gesture operation which can be recognized in each of layers in which gesture operations are classified into at least three layers. When recognizing a first gesture operation in the highest layer, the gesture recognition unit can recognize a gesture operation classified in a second layer. When recognizing a second gesture operation in the second layer, the gesture recognition unit can recognize a gesture operation classified in a third layer.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a device, a gesture recognition method, a program, and a display system. [Background technology]

[0002] Many devices are capable of accepting operations via a touch panel, and some devices also incorporate gesture operations as part of their user interface. Gesture operations are widely used as a means for users to efficiently operate devices.

[0003] A technology for smoothly processing user operations, including gesture operations, has been devised (see, for example, Patent Document 1). Patent Document 1 discloses a device in which a driver corresponding to a webcam is activated when a user makes a specific gesture, and the device receives gesture input or touch input. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technology has the problem of requiring a large number of gesture operations. For example, if there is a one-to-one correspondence between the type of gesture operation and the operations that can be accepted, the number of gesture operations increases according to the number of operations that can be accepted by the gesture operation. Users must memorize all of them, which takes time to master the gesture operations.

[0005] In view of the above-mentioned problems, the present invention provides a technique for reducing the number of gesture operations. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides a device that recognizes gesture operations corresponding to the movement of a pointing object and accepts operations according to the gesture operations, the device comprising: a data acquisition unit that acquires information about the shape of the pointing object from a sensor that acquires the information; and a gesture recognition unit that recognizes the gesture operations that can be recognized in each of at least three hierarchies into which the gesture operations are classified, based on the information acquired by the data acquisition unit, wherein the gesture recognition unit is characterized in that when it recognizes a first gesture operation in the highest hierarchical level, it becomes capable of recognizing gesture operations classified in the second hierarchical level, and when it recognizes a second gesture operation in the second hierarchical level, it becomes capable of recognizing gesture operations classified in the third hierarchical level. [Effects of the Invention]

[0007] The present invention can reduce the number of gesture operations. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 10 is a diagram illustrating an example of hierarchical gesture operations. [Figure 2] FIG. 10 is a diagram showing an example of a screen displayed by the device. [Figure 3] FIG. 1 is a diagram showing an example of a usage scene of devices used in an actual conference room. [Figure 4] 1 is an example of an overall configuration diagram of a communication system. [Figure 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of a device. [Figure 6] FIG. 1 is an example of a functional block diagram illustrating the functions of a device divided into blocks. [Figure 7] 3A and 3B are diagrams illustrating object data stored in an object data storage unit. [Figure 8] FIG. 2 is a diagram for explaining a model for detecting the state of a hand (for example, the palm) from range image data. [Figure 9] FIG. 2 is a diagram illustrating an example of the configuration of a gesture recognition model. [Figure 10]FIG. 10 is a diagram showing an example of the arrangement of the finger photographing cameras (right and left). [Figure 11] 1A and 1B are diagrams for schematically explaining a right photographed image and a left photographed image; [Figure 12] FIG. 2 is a front view of the device display. [Figure 13] 10A and 10B are diagrams illustrating gesture operations for transitioning the device to a pointer mode. [Figure 14] 10A and 10B are diagrams illustrating a gesture operation in which a user points with one finger in the pointer mode. [Figure 15] 10A and 10B are diagrams illustrating a gesture operation performed by a user in the pointer mode to terminate the pointer mode. [Figure 16] FIG. 10 is a flowchart illustrating an example of a process for a user to transition the device to a pointer mode through a gesture operation. [Figure 17] FIG. 10 is a flowchart illustrating an example of a process for a user to end a pointer mode by a gesture operation. [Figure 18] 10A and 10B are diagrams illustrating gesture operations for transitioning the device to a pen mode. [Figure 19] 10A and 10B are diagrams illustrating gesture operations for causing the device to display a pen icon in pen mode. [Figure 20] 10A and 10B are diagrams illustrating gesture operations for writing with the device in pen mode. [Figure 21] 10A and 10B are diagrams illustrating gesture operations for re-writing when the device is in pen mode. [Figure 22] FIG. 10 is a flowchart illustrating an example of a process from when a user switches to a pen mode by a gesture operation until when the mode is terminated. [Figure 23] FIG. 10 is a diagram illustrating a writable area and menu buttons. [Figure 24] 10 is a diagram illustrating an example of a selection menu displayed on a display. [Figure 25] 10A and 10B are diagrams illustrating an example of display of menu buttons that are not used in the pen mode. [Figure 26]FIG. 10 is a flowchart illustrating an example of a method for operating a menu button in a pen mode. [Figure 27] 10A and 10B are diagrams illustrating gesture operations for transitioning the device to a marker mode. [Figure 28] 10A and 10B are diagrams illustrating gesture operations for causing the device to display a marker icon in a marker mode. [Figure 29] 10A and 10B are diagrams illustrating gesture operations for writing with the device in marker mode. [Figure 30] 10A and 10B are diagrams illustrating gesture operations for re-writing when the device is in marker mode. [Figure 31] 10A and 10B are diagrams illustrating gesture operations for transitioning the device to an eraser mode. [Figure 32] 10A and 10B are diagrams illustrating gesture operations for displaying an eraser icon when the device is in eraser mode. [Figure 33] 10A and 10B are diagrams illustrating gesture operations for erasing a written line when the device is in eraser mode. [Figure 34] 10A and 10B are diagrams illustrating a gesture operation for erasing another written line when the device is in eraser mode. [Figure 35] 10A and 10B are diagrams illustrating a gesture operation for erasing another written line when the device is in eraser mode. [Figure 36] FIG. 10 is a flowchart illustrating an example of a process in which a user transitions the device to an eraser mode through a gesture operation and ends the transition. [Figure 37] 10A and 10B are diagrams illustrating gesture operations for page switching. [Figure 38] FIG. 10 is a flowchart illustrating an example of a process in which a user switches pages by performing a gesture operation. [Figure 39] FIG. 10 is a diagram showing an example of hierarchical gesture operations possible on a device. [Figure 40] FIG. 1 is an example of a functional block diagram illustrating the functions of a device divided into blocks. [Figure 41]FIG. 10 is a flowchart illustrating an example of a process in which a user performs a gesture operation to transition the device to a voice recognition mode. [Figure 42] FIG. 10 is a flowchart illustrating an example of a process for a user to terminate a voice recognition mode by a gesture operation. [Figure 43] 10A and 10B are diagrams illustrating gesture operations for displaying a pointer when the device is in language selection mode. [Figure 44] FIG. 10 is a flowchart illustrating an example of a process from when a user transitions to a language selection mode by a gesture operation to when the mode is terminated. [Figure 45] 10A and 10B are diagrams illustrating gesture operations for displaying a pointer when the device is in an industry selection mode. [Figure 46] FIG. 10 is a flowchart illustrating an example of a process from when a user transitions to an industry selection mode by a gesture operation until the mode is terminated. [Figure 47] FIG. 1 is a diagram illustrating an example of a system configuration of a display system. [Figure 48] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing system. [Figure 49] FIG. 2 is an example of a functional block diagram illustrating functions of the display system in blocks. [Figure 50] 10 is an example of a sequence diagram illustrating a process in which a device and an information processing system communicate with each other to perform gesture recognition and display coordinates pointed by one finger. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, as an example of an embodiment of the present invention, a device and a gesture recognition method performed by the device will be described with reference to the drawings.

[0010] [First embodiment] The device of this embodiment can accept many operations with a small number of gesture operations by hierarchizing the gesture operations. In addition, a new gesture operation for drawing a line has been added to the gesture operations, so the user can draw a line using the gesture operation.

[0011] FIG. 1 is a diagram showing an example of hierarchical gesture operations. (1) The device enters its initial state when it is turned on or wakes up from sleep. When the device recognizes the gesture of showing the palm of the hand in the initial state, it transitions to gesture recognition mode (an example of the highest layer, or first layer). The initial state is a state in which only the palm of the hand is recognized. This prevents the device from transitioning to gesture recognition mode without the user realizing it.

[0012] (2) Gesture recognition mode is a mode that can recognize gesture operations to transition to pointer mode. Gesture recognition mode is the state in which the device accepts the first gesture after initialization, so gesture recognition mode is sometimes called the gesture idle state. When the device recognizes a palm-up gesture in gesture recognition mode, it transitions to pointer mode (an example of the second layer).

[0013] (3) Pointer mode is a mode that can detect coordinates pointed to with the index finger (hereinafter sometimes referred to as one finger). Therefore, in pointer mode, the device recognizes gesture operations by pointing with one finger. The user can move the pointer or mouse cursor. Pointer mode transitions to three modes depending on the gesture operation. Pointer mode returns to gesture recognition mode by waving the palm downward.

[0014] (4) When the device is in pointer mode and recognizes a gesture operation of waving two fingers sideways, it transitions to pen mode (an example of the third layer). Pen mode is a mode in which the user draws lines with one finger. Pen mode ends when the user closes one finger (clenches their hand), and the device returns to gesture recognition mode. The gesture operation to end the mode is the same for marker mode (an example of the third layer) and eraser mode (an example of the third layer), which are described below.

[0015] (5) When the device recognizes a three-finger horizontal gesture (an example of a third gesture) in pointer mode, it transitions to marker mode, in which the user writes with a marker using one finger.

[0016] (6) When the device is in pointer mode and recognizes a three-finger vertical gesture, it transitions to eraser mode, in which the user erases handwritten lines with one finger.

[0017] In the pointer mode, pen mode, marker mode, and eraser mode, gesture operations such as pointing with the index finger, extending the thumb, closing the thumb, and shaking four fingers left and right are all possible.

[0018] In this way, the device recognizes pre-defined gesture operations in each of the three layers (2) → (3) → (4) to (6). By hierarchizing gesture operations, many operations are possible with a small number of gesture operation types, and users do not need to memorize many gesture operations. For example, in pen mode, marker mode, and eraser mode, it is possible to write with a pen, write with a highlighter, and erase pen writing with one finger, respectively. If there were no hierarchy, users would have to learn three times as many gesture operations.

[0019] Furthermore, in the past, there was a one-to-one correspondence between the types of gesture operations and commands for devices, making it difficult to use gesture operations to draw lines such as curves for which no commands were available. Users had to go directly to the device to handwrite lines on the device's display. In this embodiment, a pen mode is provided, allowing users to use gesture operations to draw red lines or other lines on a part of the device's display while seated, eliminating the need to go directly to the device to perform such operations.

[0020] For example, if gesture operations have four or more layers, a corresponding gesture operation is required for each layer, making the operation complicated and difficult for users to use. With two layers, a different gesture operation must be determined for each operation, and the more operations there are, the more types of gesture operations there are, making it difficult for users to use. With three layers, two gesture operations can be combined and associated with each operation, reducing the number of types of gesture operations and making it easier for users to use.

[0021] In pen mode and marker mode, you can transition to pen-down mode or pen-up mode. In eraser mode, you can transition to a mode where the virtual eraser touches the display or a mode where it is released. Since the user can transition to a lower mode within the same mode, even if gesture operations are hierarchical, operations are not complicated.

[0022] <Terminology> The pointing object may be any object that can recognize a gesture operation by the device. For example, the pointing object may be a human hand, a pointer stick, a prosthetic hand, similar objects thereof, a humanoid robot, a non-humanoid robot, etc.

[0023] Shaking the palm or one to four fingers is a human movement that may be expressed as moving the palm or fingers, moving the fingers, or vibrating quickly.

[0024] A hierarchy refers to a state in which multiple layers are connected vertically. In this embodiment, the device modes form a hierarchical structure, and recognizable gesture operations are determined according to the hierarchy or mode. Therefore, the gesture operations also form a hierarchical structure. However, the same gesture operation may be recognized in different hierarchies or modes. In this embodiment, a case of three hierarchies is described, but the hierarchy may be two or four or more hierarchies.

[0025] Each layer has one or more modes. Gesture operations can be used not only to transition between layers, but also to transition to a lower mode within a single mode. For example, pen mode and marker mode have pen-down mode and pen-up mode. Eraser mode has a mode in which a virtual eraser touches the display and a mode in which it is away from the display.

[0026] A gesture is a type of body language expressed through bodily movements, and refers to gestures and hand movements. In this embodiment, a user can operate a device using gestures, and operating a device using gestures is referred to as gesture operation.

[0027] The first gesture operation is a gesture operation for transitioning from the top layer to the second layer, and the second gesture operation is a gesture operation for transitioning from the second layer to the third layer.

[0028] The term "device" refers to an electronic device that recognizes gesture operations and accepts the operations. The term "device" is sometimes used in contrast to an instrument or tool having a simple structure. In this embodiment, an electronic whiteboard will be described as an example of the device.

[0029] <Usage scenarios> Figure 2 is an example of a screen displayed by device 2. Device 2 is one of the devices that uses a touch panel to display characters and figures written with a pen or finger on a display in real time. The user can freely set the color and thickness of the lines. Device 2 has a marker function that draws lines in a semi-transparent color. Device 2 can use the marker function to highlight characters and figures. The marker function is automatically erased after a certain period of time has passed. Device 2 also has the function of recognizing the characters written in the lines and converting them into character strings or figures. Furthermore, when device 2 is connected to a PC (Personal Computer) or similar device via a cable, it can display the screen displayed by the PC on the display.

[0030] Device 2 also has an eraser function that allows it to erase handwritten lines, character strings, shapes, etc. Device 2 accepts the selection of character strings, etc., and can move or shrink / enlarge selected character strings, etc. as a whole. Device 2 treats one screen displayed on the display as one page, and can save one screen as a one-page PDF file, etc., either automatically or in response to a user operation. Device 2 can also treat an area larger than the size of the display as one screen, and when there is no more space to write, the user can slide the screen to create new space without switching pages.

[0031] Device 2 also has the ability to connect to a network, allowing it to communicate with Device 2 at other locations. Device 2 at each location shares the contents of its screen, making it possible to hold not only face-to-face meetings in a conference room, but also remote meetings between different locations. Furthermore, a general-purpose information processing device can receive and display the screen from Device 2, allowing users to participate in meetings from home or other locations where Device 2 is not present.

[0032] FIG. 3 shows device 2 used in an actual conference room. In FIG. 3, device 2 is placed in a conference room, and participants are seated at a table in front of device 2. If participants could use gesture operations to write while seated, the range of applications would be expanded. For example, if a participant points and says "there," other participants might not understand, and the participant would have to move to the front of device 2. In this embodiment, participants can write while seated, reducing the need to move to the front of device 2.

[0033] <System configuration example> Fig. 4 is an overall configuration diagram of the communication system 1 of this embodiment. Note that, for the sake of simplicity, Fig. 4 shows only two devices 2a and 2b and accompanying electronic pens 4a and 4b, but three or more devices 2 and electronic pens may be used.

[0034] 4, the communication system 1 includes a plurality of devices 2a and 2b, a plurality of electronic pens 4a and 4b, USB memories 5a and 5b, notebook PCs 6a and 6b, video conference terminals 7a and 7b, and a PC 8. The devices 2a and 2b and the PC 8 are communicatively connected via a communication network 9. The devices 2a and 2b are further provided with displays 3a and 3b, respectively.

[0035] Furthermore, the device 2a can display on the display 3a an image drawn by an event generated by the electronic pen 4a (touching the display 3a with the tip or bottom of the electronic pen 4a). Note that the image displayed on the display 3a can also be changed based on an event generated not only by the electronic pen 4a but also by the user's hand Ha or the like (gesture operations such as zooming in, zooming out, and page switching).

[0036] A USB memory 5a can be connected to the device 2a, and the device 2a can read electronic files such as PDFs from the USB memory 5a and record electronic files to the USB memory 5a. The device 2a also has interfaces conforming to standards such as DisplayPort (registered trademark), DVI (Digital Visual Interface), HDMI (High-Definition Multimedia Interface, registered trademark), and VGA (Video Graphics Array). The user connects the device 2a to the notebook PC 6a using a cable 10a1 conforming to the standards.

[0037] The device 2a generates an event when the display 3a is touched, and transmits event information indicating this event to the laptop PC 6a in the same way as events from input devices such as a mouse or keyboard. Similarly, the device 2a is connected to a video conference terminal 7a via a cable 10a2 that is capable of communication according to the above standard. The laptop PC 6a and the video conference terminal 7a may communicate with the device 2a via wireless communication conforming to various wireless communication protocols such as Bluetooth (registered trademark).

[0038] Meanwhile, at another location where device 2b is installed, similar to the above, device 2b equipped with display 3b, electronic pen 4b, USB memory 5b, notebook PC 6b, video conference terminal 7b, cable 10b1, and cable 10b2 are used. Furthermore, the image displayed on display 3b can be changed based on an event caused by the user's hand Hb or the like.

[0039] As a result, an image drawn on the display 3a of device 2a at one location is also displayed on the display 3b of device 2b at another location, and conversely, an image drawn on the display 3b of device 2b at another location is displayed on the display 3a of device 2a at one location. In this way, communication system 1 can perform remote sharing processing to share the same image at remote locations, making it very convenient when used for conferences with remote locations, etc.

[0040] In the following, any one of the multiple devices 2 will be referred to as "device 2." Any one of the multiple displays will be referred to as "display 3." Any one of the multiple electronic pens will be referred to as "electronic pen 4." Any one of the multiple USB memory sticks will be referred to as "USB memory stick 5." Any one of the multiple notebook PCs will be referred to as "notebook PC 6." Any one of the multiple video conference terminals will be referred to as "video conference terminal 7." Any one of the multiple user's hands will be referred to as "hand H." Any one of the multiple cables will be referred to as "cable 10."

[0041] In addition, in the present embodiment, an electronic whiteboard is described as an example of device 2, but the present invention is not limited to this, and other examples of device 2 may include an electronic billboard (digital signage), a telestrator (a technology that synthesizes handwritten text with an image displayed on a monitor) used in sports and weather forecasts, or a remote image (video) diagnostic device. Furthermore, device 2 may be a headset device such as VR (Virtual Reality) goggles, AR (Augmented Reality) goggles, or MR (Mixed Reality) goggles.

[0042] Furthermore, a notebook PC 6 will be described as an example of an external device, but the external device is not limited to this, and other examples of the external device may include a terminal capable of supplying image frames, such as a desktop PC, a tablet PC, a smartphone, a digital video camera, a digital camera, a game console, etc. Furthermore, the communication network includes the Internet, a LAN (Local Area Network), a mobile phone communication network, etc. Furthermore, in this embodiment, a USB memory will be described as an example of a recording medium, but the external device is not limited to this, and other examples of the recording medium may include various recording media such as an SD card.

[0043] <Hardware configuration example> Fig. 5 is a hardware configuration diagram of device 2. As shown in Fig. 5, device 2 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, an SSD (Solid State Drive) 404, a wired LAN controller 417, a network I / F 405, a wireless LAN controller 420, an antenna 421, and an external device connection I / F (Interface) 406.

[0044] Of these, the CPU 401 controls the overall operation of the device 2. The ROM 402 stores programs used to start the OS, such as an IPL (Initial Program Loader). The RAM 403 is used as a work area for the CPU 401. The SSD 404 stores various data, such as programs for the device 2. The wired LAN controller 417 controls communication with other devices connected to a communication network via a network I / F 405. The wireless LAN controller 420 executes a communication protocol compliant with the IEEE (Institute of Electrical and Electronics Engineers) 802.11ax standard and controls communication with a finger photographing camera (right) 471 and a finger photographing camera (left) 472 by transmitting and receiving radio waves via an antenna 421. The external device connection I / F 406 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory 430 and external devices (a microphone 440, a speaker 450, and a distance image sensor 460). These external devices may also be built-in.

[0045] The range image sensor 460 has a structure in which, for example, infrared laser diodes and light receiving elements are paired and arranged in a vertical array of 500 elements and a horizontal array of 500 elements, and measures distance from the time between when the infrared laser diode emits light and when the reflected light is received. The range image sensor 460 has an imaging element with a predetermined resolution for detecting brightness gradations. Therefore, the range image sensor 460 outputs 500 x 500 pieces of range image data at a rate of 30 to 60 fps. The range image sensor 460 may also be a stereo camera or LiDAR (Light Detection and Ranging). Note that the range image data may contain at least one of distance data and image data. Hereinafter, the range image data may be simply referred to as image data.

[0046] The finger photographing camera (right) 471 and the finger photographing camera (left) 472 are photographing devices that photograph the finger held out by the user in order to display a pointer or a drawn line on the display 3. As will be described later, the finger photographing camera (right) 471 photographs the finger from the right side, and the finger photographing camera (left) 472 photographs the finger from the left side. The finger photographing camera (right) 471 and the finger photographing camera (left) 472 transmit the captured image data to the device 2 via wireless LAN. The finger photographing camera (right) 471 and the finger photographing camera (left) 472 may also transmit image data to the device 2 via a wired LAN or the like.

[0047] The device 2 also includes a capture device 411, a GPU 412, a display controller 413, a contact sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, an antenna 419a of the short-range communication circuit 419, a power switch 422, and selection switches 423.

[0048] Of these, the capture device 411 displays the display information of the external PC 470 as a still image or video. The GPU (Graphics Processing Unit) 412 is a semiconductor chip specialized in graphics. The display controller 413 controls and manages the screen display to output the output image from the GPU 412 to the display 3, etc. The contact sensor 414 detects contact with the display 3 by the electronic pen 490, the user's hand H, etc. The sensor controller 415 performs processing to identify the contact coordinates based on the signal from the contact sensor 414. The contact sensor 414 detects input coordinates using an infrared blocking method. This input coordinate detection is performed using two light-emitting and receiving devices installed at both ends of the upper side of the display 3. A light-emitting element (laser) in the light-emitting and receiving device rotates and scans an infrared beam parallel to the display 3 within a 90-degree range. The infrared beam is reflected by a reflecting member installed around the display 3. The light-receiving element receives the light returning along the same optical path as the emitted light. The light receiving and emitting device serving as the contact sensor 414 outputs the position (position on the light receiving element) where the infrared light is blocked by an object to the sensor controller 415, and the sensor controller 415 identifies the coordinate position, which is the contact position of the object, from these two pieces of position information. The electronic pen controller 416 determines whether the pen tip or the pen tail has touched the display 3 from data input by the short-range communication circuit 419 communicating with the electronic pen 490 via Bluetooth (registered trademark). The short-range communication circuit 419 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 422 is a switch for switching the power of the device 2 on and off. The selection switches 423 are a group of switches for adjusting, for example, the brightness and color of the display 3.

[0049] Furthermore, the device 2 includes a bus line 410. The bus line 410 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 401 shown in FIG.

[0050] The contact sensor 414 is not limited to an infrared blocking type, and may be a capacitive touch panel that identifies the contact position by detecting changes in capacitance. The contact sensor 414 may be a resistive film type touch panel that identifies the contact position by voltage changes across two opposing resistive films. The contact sensor 414 may be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by an object touching the display unit. In addition, the contact sensor 414 may use various detection means. Furthermore, the electronic pen controller 416 may determine whether or not the part of the electronic pen 490 that the user grips or other parts of the electronic pen are touched, in addition to the pen tip and pen butt.

[0051] <About the function> Next, the functions of the device 2 will be described with reference to Fig. 6. Fig. 6 is a functional block diagram explaining the functions of the device 2 in blocks. The device 2 has a contact position detection unit 11, a handwriting data generation unit 12, a display control unit 13, a reception unit 14, a whiteboard control unit 15, an instruction position detection unit 16, a hierarchical control unit 17, a gesture recognition unit 18, a network communication unit 19, a data recording unit 20, a first data acquisition unit 21, a second data acquisition unit 22, and an object data storage unit 23.

[0052] The contact position detection unit 11 converts the position touched by the pen or finger into coordinates. The writing data generation unit 12 acquires the coordinates of the contact point of the tip of the pen or finger from the contact position detection unit 11. The writing data generation unit 12 acquires the coordinates pointed to by the user with one finger from the indication position detection unit 16. The writing data generation unit 12 generates a writing line by connecting these coordinate point sequences through interpolation. The writing line handwritten and input with the pen or finger is displayed on the touch panel of the display 3.

[0053] The display control unit 13 displays the handwritten lines, text converted from the handwritten lines, and an operation menu for user operation on the display 3. The reception unit 14 receives a menu press based on the coordinates of the touch point of the pen or finger or the coordinates calculated by the indication position detection unit 16.

[0054] The whiteboard control unit 15 performs overall control such as starting up the whiteboard application, authentication, displaying menus, communicating with remote locations, and saving data.

[0055] The first data acquisition unit 21 acquires image data from the finger imaging camera (right) 471 and the finger imaging camera (left) 472. The pointing position detection unit 16 is a means for analyzing the image data captured by the finger imaging camera (right) 471 and the finger imaging camera (left) 472, and detecting the coordinates at which the user's single finger points on the display 3. Specifically, the pointing position detection unit 16 analyzes the image data acquired by the first data acquisition unit 21, and determines the coordinates of the intersection between the extension of the user's single outstretched finger and the surface of the display 3. This allows the user to handwrite lines and the like using gesture operations.

[0056] The hierarchical control unit 17 controls the transition of modes based on the gesture operation recognized by the gesture recognition unit 18. Note that the mode itself may be transitioned by a menu operation instead of a gesture operation.

[0057] The second data acquisition unit 22 is a means for acquiring information about the shape of the user's hand from a sensor that acquires the information. The distance image data includes information about the shape of the user's hand. The information about the shape of the user's hand is at least one of brightness information and distance information (three-dimensional information). Specifically, the second data acquisition unit 22 acquires distance image data from the distance image sensor 460.

[0058] The gesture recognition unit 18 is a means for recognizing gesture operations that can be recognized in each of the at least three hierarchical levels into which gesture operations are classified, based on the distance image data acquired by the distance image sensor 460. Specifically, the gesture recognition unit 18 recognizes the user's gesture operations from the distance image data acquired by the second data acquisition unit 22. The gesture recognition unit 18 recognizes recognizable gesture operations that are preset in each mode. When the gesture recognition unit 18 recognizes a first gesture operation in the highest hierarchical level, it becomes capable of recognizing gesture operations classified in the second hierarchical level, and when the gesture recognition unit 18 recognizes a second gesture operation in the second hierarchical level, it becomes capable of recognizing gesture operations classified in the third hierarchical level.

[0059] The network communication unit 19 is connected to the communication network 9 and performs data communication with other devices 2, any server device, and other external devices.

[0060] The data recording unit 20 stores handwritten data written on the device 2, converted text, PC screens, files, etc. in the object data storage unit 23. The data recording unit 20 also stores training data (distance image data of the palm) used when detecting the palm through machine learning in the object data storage unit 23.

[0061] FIG. 7 is a diagram illustrating the object data stored in the object data storage unit 23. As shown in FIG. The object ID item is identification information that identifies the display data. The Type item is the type of object data, and includes handwriting, text, figure, image, table, etc. Handwriting is stroke data (a sequence of coordinate points). Text is one or more characters or symbols (character code) converted from handwritten data. Figures are geometric shapes such as triangles and squares converted from handwritten data. Images are image data such as JPEG, PNG, and TIFF imported from a PC or the Internet. Tables are one-dimensional or two-dimensional table-like objects. One screen of device 2 is called a page. The item on the page is the page number. The coordinates field indicates the position of the object data relative to a predetermined origin of the device 2. The position of the object data is, for example, the upper left vertex of the bounding rectangle of the object data. The coordinates are expressed in pixel units of the display 3, for example. The size item is the width and height of the bounding rectangle of the object data.

[0062] <Mode transitions and recognizable gesture operations> The mode transitions in the device 2 of this embodiment and the gesture operations that can be recognized in each mode will be described in detail below. Gesture Recognition 0. Gesture recognition mode When Device 2 in its initial state recognizes the palm of your hand, it enters gesture recognition mode.

[0063] 1. Pointer Mode In the gesture recognition mode, when the device 2 recognizes a gesture operation in which the user waves their palm upward, it switches from the gesture recognition mode to the pointer mode. The pointer mode is a mode in which a single finger is recognized and a pointer (coordinates) on the display 3 is calculated. In the pointer mode, it is also possible to move the mouse cursor or press a menu button. In the pointer mode, the device 2 recognizes a preset gesture operation and transitions to one of the modes 2-1 to 2-3. The pointer mode ends when the user waves their palm downward.

[0064] 2-1. Pen Mode Pen mode is entered when device 2 recognizes a gesture operation in which the user shakes two fingers sideways. The user can write by pointing one finger at display 3, move the pen icon, and press the menu button. Pen mode is exited when device 2 recognizes a gesture operation in which the user closes one finger.

[0065] 2-2. Marker Mode Marker mode is entered when device 2 recognizes a gesture operation in which the user shakes three fingers sideways. The user can write with a highlighter by pointing one finger at display 3, or move the marker icon and press the menu button. The highlighter automatically disappears after a predetermined time (e.g., five seconds) has elapsed since writing. Marker mode is exited when device 2 recognizes a gesture operation in which the user closes one finger.

[0066] 2-3. Eraser mode Eraser mode is entered when device 2 recognizes a user gesture of waving three fingers vertically. The user can erase handwritten lines by pointing one finger at display 3, or move the eraser icon and press the menu button. Eraser mode is ended when device 2 recognizes a user gesture of closing one finger.

[0067] Note that the hierarchical control unit 17 may cause a mutual transition between the pen mode, marker mode, and eraser mode in the same hierarchical level by a gesture operation by the user. The user does not need to return to the pointer mode. However, even if it is necessary to return to the pointer mode, the user can simply close one finger and perform a gesture operation to transition to the pen mode, marker mode, or eraser mode again, which is less of an operational burden than menu operations, etc. In other applications having different modes, the gesture operation to transition between hierarchical levels may be the same as the gesture operation to transition from the pointer mode to the pen mode, marker mode, or eraser mode.

[0068] 3. Page switching Page switching is possible in any of the pointer mode, pen mode, marker mode, and eraser mode. If page switching were possible only in a specific mode, the user would need to perform a gesture operation to transition between modes in order to switch pages. Device 2 can return to the previous page by recognizing a gesture operation in which the user waves four fingers to the left, and can advance to the next page by recognizing a gesture operation in which the user waves four fingers to the right.

[0069] In this way, the device 2 of this embodiment detects the shape and movement of the user's palm, N (1 to 4) fingers, clenched hand, and other parts beyond the wrist through image analysis.

[0070] <Palm and gesture recognition> The method by which the gesture recognition unit 18 recognizes a palm will now be described. A person in charge at the manufacturer of the device 2 takes a photograph of the palm using the distance image sensor 460, extracts distance image data (three-dimensional data and brightness data of the palm), and stores the data in an arbitrary storage unit. The person in charge takes photographs of the palms of many people and stores the distance image data of many palms in the storage unit.

[0071] When recognizing the user's palm, the gesture recognition unit 18 detects a moving object from the distance image data input from the distance image sensor 460 and compares the distance image data of the object with pre-stored distance image data of the palm. The moving object is detected as a part where there is a change in the distance image. If the similarity of the compared distance image data is equal to or greater than a threshold, the gesture recognition unit 18 determines that the object is a palm (pattern matching). When the gesture recognition unit 18 recognizes a palm, it enters "gesture recognition mode." In gesture recognition mode, the gesture recognition unit 18 can recognize gesture operations to transition to pointer mode, pen mode, marker mode, and eraser mode.

[0072] It is preferable to use a machine-learned model for image analysis to detect the palm of a hand from such range image data.

[0073] FIG. 8 schematically shows a model for detecting the state of a hand (e.g., a palm) from range image data. FIG. 8 shows, as an example, the configuration of a palm recognition model using a CNN (Convolutional Neural Network) 60. As an example, the CNN 60 has convolutional layers 62 and 64, pooling layers 63 and 65, and a fully connected layer 70. An input image 61 is range image data captured by a range image sensor 460. This range image data includes pixel values ​​as well as distance data for each pixel. The input image 61 is processed in the order of the convolutional layer 62, pooling layer 63, convolutional layer 64, pooling layer 65, and fully connected layer 70.

[0074] The convolutional layers 62 and 64 are filters that extract features. The pooling layers 63 and 65 aggregate the feature map into a representative value for each local area (window). For example, maximum pooling selects the maximum value in the window, and mean pooling selects the average value in the window.

[0075] The output of the pooling layer 65 is input to the fully connected layer 70. The fully connected layer 70 is called a neural network. In a neural network, L layers are fully connected from the nodes in the input layer 66 to the nodes in the output layer 68. A neural network with a deep hierarchy is called a DNN. The layer between the input layer 66 and the output layer 68 is called an intermediate layer 67. The number of intermediate layers 67 and the number of nodes are merely examples.

[0076] In this embodiment, a classification model is generated to distinguish between, for example, a palm or a clenched hand (one finger closed), etc. (Another model is a regression model). For this reason, nodes for palm, clenched hand, other states, etc. are provided in the output layer 68 of the model that detects the hand state. In Figure 8, the output layer 68 has three nodes, but the number of nodes varies depending on the number of hand states to be identified.

[0077] In a classification model, each node in the output layer 68 generally outputs the probability of being classified into that node. Therefore, in Figure 8, the output layer 68 outputs the probability of a state associated with each node, such as node 71 being the "probability of a palm," node 72 being the "probability of a clenched hand," and node 73 being the "probability of other." The classification model may also be generated to recognize one finger, two fingers, three fingers, and four fingers, respectively.

[0078] In the model learning phase, range image data in which the state of the palm is known is prepared. The training data is a vector in which only the node corresponding to the state of the palm captured in the range image data is "1" and the other nodes are zero. For example, when range image data known to be the palm of a hand is input, only node 71 is "1" and the other nodes are "0". The learning device (any information processing device) calculates the difference between the output (probability) of each node in the output layer 68 and the training data using a loss function, and transmits this to the input layer side using the backpropagation method. The weights between the nodes are learned using the backpropagation method, and gradually nodes 71 to 73 in the output layer begin to output the correct probabilities.

[0079] In the inference phase using the model, for example, when range image data of a palm is input, it is expected that node 71 in output layer 68 corresponding to the palm will output a probability close to "1," and other nodes 72 and 73 will output probabilities close to "0." The gesture recognition unit 18 determines (infers) that the hand state corresponds to the node with the highest probability.

[0080] Furthermore, a model that detects human movements from time-series data is known. Figure 9 is a diagram illustrating the configuration of a gesture recognition model. As an example, an example of the configuration of a gesture recognition model using LSTM (Long Short Term Memory) is explained here. LSTM is known to be effective in recognizing time-series data. For example, whether to shake one's palm up or down is not determined solely by the state of the hand at a given moment, but is determined by the state of the hand over time. Therefore, it is advisable to use a model that is suitable for time-series data.

[0081] FIG. 9(a) shows the data flow of the LSTM, and FIG. 9(b) is a diagram explaining the data flow in FIG. 9(a) by breaking it down. The LSTM neural network has three layers: an input layer 301, one hidden layer 302, and an output layer 303. However, the hidden layer 302 may include one or more fully connected layers. The LSTM outputs the calculation results in the hidden layer 302 to the output layer 303, and also inputs the same calculation results back into itself (the hidden layer 302) for use in the next calculation. FIG. 9(b) shows that the calculation results performed in the hidden layer 302 are substituted back into the hidden layer 302 by arranging the input layer 301, hidden layer 302, and output layer 303 in chronological order. The hidden layer 302 inherits past data, allowing the LSTM to retain past memories (in this embodiment, feature quantities of past depth image data).

[0082] The input data to the input layer 301 may be range image data, feature values ​​of the range image data, or palm joint coordinates. The process of detecting the palm joint coordinates from the range image data can be performed using an existing model. For example, the input data X t is a vector whose elements are the two-dimensional coordinates of the first to third joints of each of the five fingers. The subscript t indicates the number of times the input data is input. (1) The result of the operation performed by the hidden layer 302 on the input data X0 becomes the output data h0, and also becomes the input of the hidden layer 302 of X1. (2) The result of the operation performed by the hidden layer 302 on the input data X1 becomes the output data h1, and also becomes the input to the hidden layer of X2. (3) The result of the operation performed by the hidden layer 302 on the input data X2 becomes the output data h2, and also becomes the input of the hidden layer 302 of X3. (4) Input data X t The calculation result of the hidden layer 302 is the output data h t As well as becoming X t+1 It also serves as input for the intermediate layer.

[0083] The output data h output by the output layer 303 t is a vector containing elements equal to the number of gesture operations to be identified. For example, suppose the gesture operations are "upward movement of the palm," "one finger," "sideways movement of two fingers," "sideways movement of three fingers," "vertical movement of three fingers," "leftward movement of four fingers," and "rightward movement of four fingers." The output data is a vector with seven elements, each of which represents the probability corresponding to these gesture operations. If the actual gesture operation is "upward movement of the palm," the corresponding element of the output data is expected to be close to 1, such as (1,0,0,0,0,0,0).

[0084] Known LSTM learning methods include the BPTT (back propagation through time) method and the RTRL (real time recurrent learning) method. The input data is the joint coordinates of each joint, and the training data is a one-hot vector in which only the elements corresponding to the annotated gesture operations (description of body movements) are 1. The weights between the input layer 301, hidden layer 302, and output layer 303, as well as the weights that regress from the hidden layer 302 to the hidden layer 302, are adjusted using the backpropagation method, which propagates the difference between the output value of each node in the output layer 303 and each element of the one-hot vector.

[0085] Note that the gesture recognition method in Figure 9 is just one example, and other recognition algorithms such as "spatio-temporal convolutional neural network (ST-GCN)", a model using an attention mechanism, a support vector machine, a logistic regression, a decision tree, and a random forest may also be used.

[0086] <Gesture operations recognized in gesture recognition mode> Next, various gesture operations will be described. The gesture recognition unit 18 recognizes the following gesture operations in each mode. Note that the gesture operations also include simple shapes with no hand movement.

[0087] Recognition of palm-up gestures (gesture recognition mode) When the gesture recognition unit 18 recognizes that the recognized palm (hereinafter, this object will be referred to as the palm object regardless of the orientation of the palm or the state of the hand) moves upward by a predetermined distance, it determines that this is a gesture operation to start the "pointer mode."

[0088] Index finger (single finger) recognition (pointer mode, pen, marker, eraser mode) The gesture recognition unit 18 tracks the movement of the palm object, and when it recognizes that this object has changed into a shape with one finger outstretched, it determines that this is a gesture operation to display a pointer on the display 3 at the coordinates pointed by one finger.

[0089] - Recognition of index and middle fingers (two fingers) (pointer mode) The gesture recognition unit 18 tracks the movement of the palm object, recognizes that this object has changed into a shape with two fingers outstretched, and when it recognizes that this object is being swung sideways, it determines that this is a gesture operation to start "pen mode."

[0090] - Recognition of index, middle and ring fingers (3 fingers) (pointer mode) The gesture recognition unit 18 tracks the movement of the palm object, recognizes that this object has changed into a shape with three fingers outstretched, and when it recognizes that this object is being swung sideways, it determines that this is a gesture operation to start "marker mode."

[0091] - Recognition of index, middle and ring fingers (3 fingers) (pointer mode) The gesture recognition unit 18 tracks the movement of the palm object, recognizes that this object has changed into a shape with three fingers outstretched, and when it recognizes that this object is being swung vertically, it determines that this is a gesture operation to start "eraser mode."

[0092] - Recognition of index finger, middle finger, ring finger and pinky finger (4 fingers) (pointer mode, pen, marker, eraser mode) The gesture recognition unit 18 tracks the movement of the palm object, recognizes that this object has changed into a shape with four fingers outstretched, and when it recognizes that this object is being swung to the left, it determines that this is a gesture operation to switch the page of the whiteboard displayed on the display 3 to the previous page.

[0093] Furthermore, when the gesture recognition unit 18 recognizes that this object is being swung to the right, it determines that this is a gesture for switching the page of the whiteboard displayed on the display 3 to the next page.

[0094] Palm down gesture recognition (pointer mode) When the gesture recognition unit 18 recognizes that the palm is waved downward while the pointer mode is being executed, it determines that this is a gesture operation to end the pointer mode.

[0095] Hand grip recognition (pointer mode, pen mode, marker mode, eraser mode) When the gesture recognition unit 18 recognizes that the shape of one finger has changed to a closed shape while the pointer mode, pen mode, marker mode, or eraser mode is being executed, it determines that the gesture is for ending the mode.

[0096] Index finger and thumb recognition (pointer mode, pen, marker, eraser mode) When the gesture recognition unit 18 detects that the thumb is extended in a state where the mouse cursor is displayed at the coordinates pointed by one finger after switching to the pointer mode, it recognizes that this is a gesture operation in which the virtual pen touches the touch panel. When the gesture recognition unit 18 detects that the thumb changes to a closed shape in this state, it determines that this is a gesture operation in which the virtual pen is removed from the touch panel.

[0097] When the gesture recognition unit 18 detects that the thumb is extended in a state where the pen icon is displayed at the coordinates pointed by one finger after switching to pen mode, it recognizes this as a pen-down gesture operation. When it detects that the thumb has changed to a closed shape in this state, it determines this as a pen-up gesture operation. Pen-up mode refers to a mode in which the pen is not in contact with the touch panel. In pen-up mode, writing does not occur even if the pen icon moves. Pen-down mode refers to a mode in which the pen is in contact with the touch panel. In this embodiment, the pen-down mode refers to a writing state, even though a physical pen is not used.

[0098] When the gesture recognition unit 18 recognizes that the thumb is extended in the marker mode and a marker icon is displayed at the coordinates pointed by one finger, it recognizes that the gesture is a pen-down gesture operation. When the gesture recognition unit 18 recognizes that the thumb has changed to a closed shape in the pen-down mode, it determines that the gesture is a pen-up gesture operation.

[0099] When the gesture recognition unit 18 recognizes that the thumb is extended in a state in which the mode has been switched to eraser mode and an eraser icon is displayed at the coordinates pointed by one finger, it recognizes this as a gesture operation of virtually bringing the eraser into contact with the display 3. When the gesture recognition unit 18 recognizes that the thumb has changed to a closed shape in this mode in which the virtual eraser is in contact with the display 3, it determines this as a gesture operation of virtually removing the eraser from the display 3.

[0100] <Displays the pointer at the point you point at> Next, calculation of the coordinates of the display 3 pointed by the user's finger will be described with reference to FIG. 10 and other figures. FIG. 10 is a diagram showing an example of the arrangement of finger-capturing cameras. Finger-capturing camera (right) 471 and finger-capturing camera (left) 472 are cameras for detecting the coordinates of the pointing finger on the display 3. In FIG. 10, one finger-capturing camera (right) 471 and one finger-capturing camera (left) 472 are arranged on the desk and around the table. Since the finger-capturing camera (right) 471 and the finger-capturing camera (left) 472 only need to capture images of the user's wrist and the display 3, the cameras may be built into the upper or lower part of the device 2. The finger-capturing camera (right) 471 and the finger-capturing camera (left) 472 may be built-in or external. The finger-capturing camera (right) 471 and the finger-capturing camera (left) 472 may be cameras that can acquire distance information of the subject along with the image. This makes it easier for the gesture recognition unit 18 and the pointed position detection unit 16 to determine the state of the fingers (how many fingers are involved), etc.

[0101] At least two finger-capturing cameras (right) 471 and finger-capturing cameras (left) 472 are required, and three or more may be provided. Device 2 may use two pieces of image data that are easy to recognize the hand from three or more pieces of image data, or may calculate the coordinates of the finger pointing by combining two pieces of image data extracted from all the image data and use the average of these. Furthermore, finger-capturing cameras (right) 471 or finger-capturing cameras (left) 472 may be spherical cameras.

[0102] Next, a description will be given of a process for displaying a pointer at the position on the display 3 pointed by the user. This process also uses image data received from the finger photographing cameras (right and left).

[0103] In the pointer mode, pen mode, marker mode, and eraser mode, when the gesture recognition unit 18 recognizes that the shape of the hand has changed to a shape with one finger extended, it transmits a command to start an operation to determine the position on the display 3 at which the finger is pointed to, to the pointed position detection unit 16. Using a model generated by machine learning in the same way as the gesture recognition unit 18, the pointed position detection unit 16 extracts an object such as a hand from the right captured image and determines that the object is a hand with one finger extended.

[0104] When the pointed position detection unit 16 receives this command, it starts an operation to find the coordinates of the point on the display 3 that is being pointed at, using image data received via wireless LAN from the finger shooting camera (right) 471 and the finger shooting camera (left) 472. An image (right captured image) captured by the finger shooting camera (right) 471 is shown in Fig. 11(a), and an image (left captured image) captured by the finger shooting camera (left) 472 is shown in Fig. 11(b).

[0105] Next, the pointing position detection unit 16 identifies the base (point P1 in FIG. 11(a)) and tip (point Q1 in FIG. 11(a)) of one finger from the right captured image. The pointing position detection unit 16 also compares shape data of the object extracted from the right captured image with pre-stored shape data of the display 3 of the device 2 to determine that the object is the display 3 of the device 2. A model generated by machine learning may also be used for this determination. In the captured image of FIG. 11(a), points A1, B1, C1, and D1 indicate the upper left corner, lower left corner, upper right corner, and lower right corner of the display 3 of the device 2, respectively. In the captured image of FIG. 11(a), point Ei is the intersection of the extension of line segment P1Q1 and the extension of line segment A1B1. Furthermore, point Fi is the intersection of the extension of line segment P1Q1 and the extension of line segment C1D1.

[0106] The pointing position detection unit 16 determines the hand with one finger extended from the left captured image in the same manner as above, and further identifies the base (point P2 in FIG. 11(b)) and tip (point Q2 in FIG. 11(b)) of this extended finger. Next, the pointing position detection unit 16 determines the display 3 of the device 2 from the left captured image. In the captured image of FIG. 11(b), points A2, B2, C2, and D2 indicate the upper left corner, lower left corner, upper right corner, and lower right corner of the display 3 of the device 2, respectively. In the captured image of FIG. 11(b), point Gi is the intersection of the extension of line segment P2Q2 and line segment A2B2. Furthermore, point Hi is the intersection of the extension of line segment P2Q2 and line segment C2D2.

[0107] 12 is a front view of the display 3 of the device 2. Point T on the display 3 of the device 2 at which the user is pointing corresponds to the intersection of the line connecting points Ei and Fi in the image captured by the finger-capturing camera (right) 471 and the line connecting points Gi and Hi in the image captured by the finger-capturing camera (left) 472. Since the coordinates in the image are based on the pixel position of the captured image, when displaying a pointer or the like at the part being pointed at, they need to be converted into coordinates based on the pixel position on the display 3 of the device 2.

[0108] 12 indicate the upper left corner, lower left corner, upper right corner, and lower right corner of the display 3 of the device 2, respectively. Points Ed and Fd are coordinate points based on the display pixel positions of the display 3 of the device 2, obtained by converting points Ei and Fi in the image (right captured image) captured by the finger capturing camera (right) 471 from coordinates based on the pixel positions of the captured image using a coordinate transformation matrix TR described below. Points Gd and Hd are coordinate points based on the display pixel positions of the display 3 of the device 2, obtained by converting points Gi and Hi in the image (left captured image) captured by the finger capturing camera (left) 472 from coordinates based on the pixel positions of the captured image using a coordinate transformation matrix TL described below.

[0109] The coordinate transformation matrices TR and TL are calculated using the following formulas. If the coordinates of the upper left, lower left, upper right, and lower right corners of display 3 based on the pixel positions of the image (right captured image) captured by finger capture camera (right) 471 are A1(a1x, a1y), B1(b1x, b1y), C1(c1x, c1y), and D1(d1x, d1y), and the coordinates of the upper left, lower left, upper right, and lower right corners of display 3 based on the pixel positions of display 3 of device 2 are A3(a3x, a3y), B3(b3x, b3y), C3(c3x, c3y), and D3(d3x, d3y), a3x = (R11*a1x + R12*a1y + R13) / (R31*a1x + R32*a1y + 1) a3y = (R21*a1x + R22*a1y + R23) / (R31*a1x + R32*a1y + 1) b3x = (R11*b1x + R12*b1y + R13) / (R31*b1x + R32*b1y + 1) b3y = (R21*b1x + R22*b1y + R23) / (R31*b1x + R32*b1y + 1) c3x = (R11*c1x + R12*c1y + R13) / (R31*c1x + R32*c1y + 1) c3y = (R21*c1x + R22*c1y + R23) / (R31*c1x + R32*c1y + 1) d3x = (R11*d1x + R12*d1y + R13) / (R31*d1x + R32*d1y + 1) d3y = (R21*d1x + R22*d1y + R23) / (R31*d1x + R32*d1y + 1) The coordinate transformation matrix TR shown in equation (1) is obtained by the eight simultaneous equations above.

[0110]

number

[0111]

number

[0112]

number

[0113]

number

[0114]

number

[0115]

number

[0116] <Gesture operations and operations accepted by the device> The relationship between some gesture operations and operations accepted by the device 2 will be described below with reference to FIGS.

[0117] <<Pointer mode>> 13 shows a gesture operation for transitioning device 2 to pointer mode. As shown in FIG. 13(a), a user causes device 2 to capture an image of the palm of their hand. When distance image sensor 460 recognizes this gesture operation, hierarchical control unit 17 transitions to gesture recognition mode. That is, when hierarchical control unit 17 recognizes the user's palm in the initial state, it determines that the mode is the highest hierarchical level (e.g., gesture recognition mode), and gesture recognition unit 18 becomes able to recognize a first gesture operation in the highest hierarchical level.

[0118] 13(b), when the user shakes their palm upward, the range image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the range image data and recognizes that this is a gesture operation to transition to the pointer mode. As a result, the hierarchical control unit 17 transitions to the pointer mode.

[0119] 14 shows a gesture operation in which the user points with one finger in the pointer mode. When the user points one finger toward the device 2, the range image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the range image data and detects that one finger is pointing at the display 3.

[0120] When it is detected in the pointer mode that one finger is pointing at the display 3, the pointing position detection unit 16 calculates the coordinates of the display 3 at which the finger is pointing, using image data input from the finger capturing camera (right) 471 and the finger capturing camera (left) 472. The display control unit 13 displays the pointer 51 at the calculated coordinates. Since the pointing position detection unit 16 calculates the coordinates periodically (e.g., 30 times per second), when the user moves one finger in this state, the pointing coordinates change, and the display control unit 13 moves the pointer 51 to follow those coordinates. In this way, when the gesture recognition unit 18 recognizes a gesture operation of waving a palm upward in the highest layer (e.g., gesture recognition mode), the layer control unit 17 transitions from the highest layer to the pointer mode, which is the second layer, and displays the pointer at the coordinates of the display detected by the pointing position detection unit 16.

[0121] FIG. 15 shows a gesture operation by the user to end the pointer mode in the pointer mode. When the user shakes their palm downward, the range image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the range image data and recognizes it as a gesture operation to end the pointer mode. This causes the layer control unit 17 to end the pointer mode. When the pointer mode ends, if the display control unit 13 was displaying a pointer 51, it erases the pointer 51. The layer control unit 17 transitions to the gesture recognition mode.

[0122] <<Flow from starting to ending pointer mode>> 16 is a flowchart illustrating a process in which a user makes a gesture operation to transition device 2 to the pointer mode. At the start of FIG. 16, device 2 has recognized the palm of the hand and is in the gesture recognition mode.

[0123] The user waves their palm upward (S11). The gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of waving the palm upward has been recognized (S12).

[0124] If the gesture recognition unit 18 recognizes this gesture operation (Yes in S12), the hierarchical control unit 17 transitions to the pointer mode (S13).

[0125] 16, the gesture operation for starting the pointer mode is a palm-up gesture, but the gesture operation for starting the pointer mode may be any gesture that is easy for the user to perform with their hand. For example, this gesture operation may be a palm-down gesture, opening and closing five fingers, or turning the palm over.

[0126] 17 is a flowchart illustrating a process for the user to end the pointer mode by a gesture operation, where the device 2 is in the pointer mode.

[0127] In the pointer mode, the user waves their palm down (S21). The gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a palm-down gesture operation has been recognized (S22).

[0128] If the gesture recognition unit 18 recognizes this gesture operation (Yes in S22), the hierarchical control unit 17 ends the pointer mode (S23). When the gesture recognition unit 18 recognizes a gesture operation that ends recognition of the gesture operations classified in the second hierarchical level (pointer mode) in the second hierarchical level, the gesture recognition unit 18 becomes able to recognize gesture operations classified in the highest hierarchical level (gesture recognition mode).

[0129] 17, the gesture operation for terminating the pointer mode is a downward movement of the palm, but the gesture operation for terminating the pointer mode may be any gesture operation that is easy for the user to perform with their hand. For example, this gesture operation may be a clenched finger, an extended finger and pinky finger, or an extended pinky finger.

[0130] <Starting and ending pen mode> Next, with reference to FIG. 18 and other figures, a description will be given of gesture operations from the start to the end of the pen mode. FIG. 18 shows gesture operations for transitioning the device 2 to the pen mode. When the user points two fingers toward the device 2, the range image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the range image data and detects that the two fingers have been swung sideways. This causes the hierarchical control unit 17 to transition to the pen mode.

[0131] 19 shows a gesture operation for displaying the pen icon 53 on the device 2 in pen mode. When the user points one finger at the device 2, the range image sensor 460 captures a still image or video including the gesture operation. The gesture recognition unit 18 analyzes the range image data and detects that one finger is pointing at the display 3.

[0132] When it is detected that one finger is pointing at the display 3 in the pen mode, the pointing position detection unit 16 calculates the coordinates of the pointing point using image data input from the finger photographing camera (right) 471 and the finger photographing camera (left) 472. The display control unit 13 displays the pen icon 53 at these coordinates. When the user moves the finger in this state, the display control unit 13 moves the pen icon 53 to follow the pointing coordinates.

[0133] FIG. 20 shows a gesture operation for the device 2 to write in pen mode. When the user extends one finger and extends his thumb, the distance image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the distance image data and recognizes a gesture operation in which the thumb is extended with one finger extended. The display control unit 13 brings the pen down at the coordinates where the pen icon 53 is displayed. The display control unit 13 may change the color or shape of the pen icon 53 to indicate that a virtual pen is in contact with the display 3. In FIG. 20, the color of the pen icon 53 has changed. Upon pen down, the device 2 starts writing. When the user moves one finger with the thumb extended, the writing data generation unit 12 generates a writing line 54 based on the trajectory of the coordinates where the user is pointing, and the display control unit 13 displays this writing line 54.

[0134] In this way, in pen mode, when the gesture recognition unit 18 recognizes a gesture operation of pointing with one finger and extending the thumb, it determines that the mode is pen down mode, and the display control unit 13 displays the pen icon 53 in a manner different from that before the pen down mode.

[0135] 21 shows a gesture operation for re-writing when the device 2 is in pen mode. When the user closes their thumb, the range image sensor 460 captures a still image or video containing this gesture operation. The gesture recognition unit 18 analyzes the range image data and recognizes it as a pen-up gesture operation.

[0136] When the user moves one finger, the display control unit 13 moves the pen icon 53 to follow the coordinates pointed at. When the user extends their thumb while pointing with one finger, the gesture recognition unit 18 recognizes the gesture operation of the thumb being extended, and the device 2 enters pen-down mode. When the user moves one finger with their thumb extended, the writing data generation unit 12 generates a writing line 55 based on the trajectory of the coordinates pointed at, and the display control unit 13 displays this writing line 55. Therefore, the user can change the position of the pen icon 53 and write again.

[0137] When the user closes one finger, the distance image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the distance image data and recognizes that this is a gesture to end the pen mode. The hierarchical control unit 17 ends the pen mode and transitions to the pointer mode.

[0138] In the pen mode, the user can press the menu button. In the pen-up mode, the user moves the pen icon 53 to the menu button and extends his / her thumb. This causes the reception unit 14 to receive the press of the menu button. This allows the user to select the color, thickness, type, etc. of the handwritten line. The display control unit 13 does not display the handwritten line in the area overlapping the menu button.

[0139] <<Flow from starting to ending pen mode>> 22 is a flowchart illustrating the process from when a user transitions to pen mode using a gesture operation to when the device 2 ends the operation. At the start of the process in FIG. 22, the device 2 is in pointer mode because it has recognized a gesture operation of waving a palm up.

[0140] The gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of shaking two fingers sideways has been recognized (S31). If the determination in step S31 is Yes, the process proceeds to step S32, and if No, the process proceeds to step S43.

[0141] In step S32, the hierarchical control unit 17 transitions to the pen mode (S32).

[0142] Next, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of pointing one finger toward the display 3 has been recognized (S33). If the determination in step S33 is Yes, the process proceeds to step S34; if the determination is No, the process repeats step S33.

[0143] In step S34, the pointing position detection unit 16 calculates the coordinates where the finger is pointing, and the display control unit 13 displays the pen icon 53 at those coordinates (S34).

[0144] Next, the pointing position detection unit 16 determines whether or not movement of one finger has been detected based on the coordinates of the pointing finger (S35). If the determination in step S35 is Yes, the process proceeds to step S36, and if No, the process proceeds to step S37.

[0145] In step S36, the display control unit 13 moves the pen icon 53 to the coordinates where the one finger is pointing, calculated by the pointing position detection unit 16 (S36).

[0146] In step S37, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of extending the thumb while pointing one finger has been recognized (S37). If the determination in step S37 is Yes, the process proceeds to step S38; if No, the process returns to step S35.

[0147] In step S38, the gesture recognition unit 18 recognizes that the pen mode is the pen-down mode within the pen mode. The indication position detection unit 16 determines whether or not movement of one finger has been detected based on the coordinates of the pointing finger (S38). If the determination in step S38 is Yes, the process proceeds to step S39; if the determination is No, the process proceeds to step S40.

[0148] In step S39, the handwriting data generating unit 12 generates a handwritten line based on the locus of coordinates where one finger is pointing, calculated by the pointing position detecting unit 16, and the display control unit 13 displays the handwritten line (S39).

[0149] In step S40, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of closing the thumbs has been recognized (S40). If the determination in step S40 is Yes, the process proceeds to step S41, and if No, the process returns to step S38.

[0150] In step S41, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of closing one finger has been recognized (S41). If the determination in step S41 is Yes, the process proceeds to step S42, and if No, the process returns to step S35.

[0151] In step S42, since a gesture operation of closing one finger is recognized, the hierarchical control unit 17 ends the pen mode (S42). The pen mode transitions to the pointer mode. When the gesture recognition unit 18 recognizes a gesture operation that ends recognition of a gesture operation classified in the third hierarchical level (e.g., pen mode) in the third hierarchical level, it becomes possible to recognize a gesture operation classified in the second hierarchical level (e.g., pointer mode). Note that the gesture operation that ends the pen mode is the same as the gesture operation that ends the marker mode and the gesture operation that ends the eraser mode. This reduces the number of gesture operations that the user needs to remember.

[0152] In step S43, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not another gesture operation has been recognized (S43). The other gesture operation is a gesture operation of waving three fingers horizontally, waving three fingers vertically, or waving four fingers left and right. If the determination in step S43 is Yes, the process proceeds to step S44; if the determination is No, the process returns to step S31.

[0153] In step S44, the gesture recognition unit 18 executes a process corresponding to another gesture operation (S44).

[0154] 22, the gesture operation for starting the pen mode is a horizontal wave of two fingers, but the gesture operation for starting the pen mode may be any gesture operation that is easy for the user to perform with their hand. For example, this gesture operation may be a vertical wave of two fingers, a horizontal wave of three fingers, a vertical wave of three fingers, etc.

[0155] 22, closing one finger is the gesture operation for ending the pen mode, but the gesture operation for ending the pen mode may be any gesture operation that is easy for the user to perform with their hand. For example, this gesture operation may be extending only one finger and the little finger, or extending only the little finger, etc.

[0156] <<Pressing the menu button in pen mode>> The following describes a gesture operation in which the user moves the pen down in pen mode, writes, and then moves the pen up to change the color of the drawn line.

[0157] 23 is a diagram illustrating the writable area and menu buttons. A writable area 48 where handwriting can be done is preset in the display area of ​​the display 3 of the device 2. If the coordinates where the user is pointing with one finger are within the writable area, the device 2 displays a line along the locus of the coordinates, but does not display a line outside the writable area.

[0158] A menu button for accepting various settings is displayed on the right edge of the display 3. When a user writes a line by hand in pen mode, closes their thumb to switch to pen-up mode, and moves the coordinates pointed by one finger outside the writable area, the display control unit 13 displays a pen icon 53 at those coordinates. When the user moves the coordinates pointed by one finger (the display position of the pointer) to the pen setting button and extends their thumb, the gesture recognition unit 18 recognizes the gesture operation of extending the thumb. Since the coordinates are outside the writable area, the display control unit 13 does not display the line. The display control unit 13 may change the icon from the pen icon 53 to the pointer 51. The whiteboard control unit 15 also determines that the coordinates where the pen was brought down are the menu button and determines that the menu button has been selected. In this case, the menu button 49, which selects the color and thickness of the pen's line, has been pressed. This is the same process as when a touch panel detects a touch. The whiteboard control unit 15 then displays a selection menu for the color and thickness of the pen's line.

[0159] FIG. 24 shows the selection menu 52 displayed on the display 3. When the user closes their thumb to switch to pen-up mode and extends their thumb while placing the coordinates (displayed by the pointer 51) pointed by one finger on the color they want to set in the selection menu 52, the gesture recognition unit 18 recognizes the gesture operation of extending the thumb. The whiteboard control unit 15 identifies the color displayed at the coordinates pointed by the one finger when the thumb is extended and determines that this color has been selected. The display control unit 13 then erases the selection menu. Next, when the user moves one finger to the pen-down position and extends their thumb, the gesture recognition unit 18 recognizes this gesture operation, and the display control unit 13 switches the pen icon 53 to pen-down mode. When the user moves one finger in pen-down mode, a line of the selected color is displayed along the path of the coordinates pointed by the one finger. In this way, the user can change settings such as the color and thickness of the line while maintaining the pen mode.

[0160] When the pen mode is entered by a gesture operation, the display control unit 13 may increase the size of the menu buttons and hide the menu buttons that are not used in the pen mode. An example of the display in this case is shown in FIG.

[0161] Fig. 25 shows an example of how menu buttons are displayed in pen mode. In Fig. 25, menu button 49, which allows you to select the color and thickness of the pen line, is larger, and menu buttons that are not used in pen mode are hidden, making it easier for the user to point at menu button 49 with one finger and preventing the user from accidentally selecting an unintended menu button.

[0162] Fig. 26 is a flow chart for explaining how to operate the menu buttons in the pen mode, which starts when the pen mode is selected.

[0163] When the pen mode is entered, the display control unit 13 enlarges the menu buttons to be used in the pen mode and hides the menu buttons that are not used (S101).

[0164] Next, the whiteboard control unit 15 determines whether or not the pointing position of one finger has gone outside the writable area based on the coordinates of the pointing position detected by the pointing position detection unit 16 (S102).

[0165] If the determination in step S102 is Yes, the display control unit 13 changes the pen icon to a pointer (S103). If the coordinates pointed to by one finger detected by the pointing position detection unit 16 are in the writable area, the pen icon remains.

[0166] <Start and end of marker mode> Next, with reference to FIG. 27 etc., a description will be given of gesture operations from the start to the end of the marker mode. FIG. 27 shows gesture operations for the device 2 to transition to the marker mode. When the user points three fingers at the device 2 and shakes them sideways, the range image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the range image data and detects that the three fingers have been shaken sideways. As a result, the hierarchical control unit 17 transitions to the marker mode.

[0167] 28 shows a gesture operation for causing the device 2 to display the marker icon 56 in the marker mode. When the user points one finger toward the device 2, the range image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the range image data and detects that one finger is pointing at the display 3.

[0168] In the marker mode, when it is detected that one finger is pointing at the display 3, the pointing position detection unit 16 calculates the coordinates of the pointing point using image data input from the finger photographing camera (right) 471 and the finger photographing camera (left) 472. The display control unit 13 displays the marker icon 56 at these coordinates. When the user moves one finger in the marker mode, the display control unit 13 moves the marker icon 56 to follow the pointing coordinates.

[0169] FIG. 29 shows a gesture operation for the device 2 to write in marker mode. When the user extends his / her thumb with one finger extended, the distance image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the distance image data and recognizes the gesture operation of extending the thumb (the pen-down mode is entered within the marker mode). The display control unit 13 brings the pen down at the coordinates where the marker icon 56 is displayed. The display control unit 13 may change the color or shape of the marker icon 56 to indicate that a virtual marker is in contact with the display 3. In FIG. 29, the color of the marker icon 56 has changed. The device 2 begins writing a marker line. When the user moves one finger in pen-down mode, the writing data generation unit 12 generates a marker line 57 based on the trajectory of the coordinates where the finger is pointed, and the display control unit 13 displays this marker line 57.

[0170] In this way, in the marker mode, when the gesture recognition unit 18 recognizes a gesture operation of pointing with one finger and extending the thumb, it determines that the mode is pen-down mode, and the display control unit 13 displays the marker icon 56 in a manner different from that before the pen-down mode.

[0171] 30 shows a gesture operation for re-writing when the device 2 is in the marker mode. When the user closes their thumbs, the range image sensor 460 captures a still image or video containing this gesture operation. The gesture recognition unit 18 analyzes the range image data and recognizes it as a pen-up gesture operation.

[0172] When the user moves one finger, the display control unit 13 moves the marker icon 56 to follow the coordinates pointed at. When the user extends their thumb while pointing with one finger, the gesture recognition unit 18 recognizes the gesture operation of the thumb being extended, and the device 2 enters pen-down mode. When the user moves one finger in pen-down mode, the writing data generation unit 12 generates a marker line 58 based on the trajectory of the coordinates pointed at, and the display control unit 13 displays this marker line 58. Therefore, the user can change the position of the marker icon 56 and write again.

[0173] When the user closes one finger, the range image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the range image data and recognizes that this is a gesture to end the marker mode. The hierarchical control unit 17 ends the marker mode and transitions to the pointer mode.

[0174] In the marker mode, the user can press the menu button. In the pen-up mode, the user moves the marker icon 56 to the menu button and extends his / her thumb. This causes the reception unit 14 to receive the press of the menu button. This allows the user to select the color, thickness, line type, etc. of the marker line. The display control unit 13 does not display the marker line in the area overlapping the menu button.

[0175] Also, the flowchart for explaining recognition of gesture operations in the marker mode is explained in the same manner as in FIG. 22, since the only difference from the flowchart in FIG. 22 is the number of fingers required to transition to the marker mode.

[0176] In this embodiment, the gesture operation for starting the marker mode is a horizontal wave of three fingers, but the gesture operation for starting the marker mode may be any gesture operation that is easy for the user to perform with their hands. For example, the gesture operation may be a horizontal wave of two fingers, a vertical wave of two fingers, a vertical wave of three fingers, etc.

[0177] Although closing one finger is defined as the gesture operation for ending the marker mode, the gesture operation for ending the marker mode may be any gesture operation that is easy for the user to perform with their hand. For example, this gesture operation may be extending only one finger and the little finger, or extending only the little finger, etc.

[0178] Also in the marker mode, similarly to FIGS. 23 to 26, it is preferable that display control unit 13 enlarges only the menu buttons to be used in the marker mode and hides the menu buttons that are not used.

[0179] <Start and end of eraser mode> 31 shows a gesture operation for transitioning device 2 to eraser mode. When the user points three fingers at device 2 and waves them vertically, distance image sensor 460 captures a still image or video including this gesture operation. Gesture recognition unit 18 analyzes the distance image data and detects that three fingers have been waved vertically. This causes hierarchical control unit 17 to transition to eraser mode.

[0180] 32 shows a gesture operation for causing the device 2 to display the eraser icon 81 in the eraser mode. When the user points one finger toward the device 2 in the eraser mode, the range image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the range image data and detects that one finger is pointing at the display 3.

[0181] When it is detected that one finger is pointing at the display 3, the pointing position detection unit 16 calculates the coordinates of the pointing point using image data input from the finger photographing camera (right) 471 and the finger photographing camera (left) 472. The display control unit 13 displays an eraser icon 81 at the coordinates of the pointing point. When the user moves one finger in this state, the display control unit 13 moves the eraser icon 81 to follow the pointing coordinates.

[0182] FIG. 33 shows a gesture operation for erasing a writing line 55 when the device 2 is in eraser mode. When a user extends their thumb with one finger extended, the distance image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the distance image data and recognizes the gesture operation of extending the thumb. The gesture recognition unit 18 recognizes that a virtual eraser is touching the display 3 (a virtual eraser touching mode is entered in the eraser mode). The display control unit 13 may change the color or shape of the eraser icon 81 to indicate that the virtual eraser is touching the display 3. In FIG. 33, the color of the eraser icon 81 has changed. When a user moves one finger in the virtual eraser touching mode, the display control unit 13 erases the writing line 54 drawn at the coordinates pointed to.

[0183] In this way, in the eraser mode, when the gesture recognition unit 18 recognizes a gesture operation of pointing with one finger and extending the thumb, it determines that the virtual eraser has come into contact with the display 3, and the display control unit 13 displays the eraser icon 81 in a manner different from that before the virtual eraser came into contact with the display 3.

[0184] 34 and 35 show a gesture operation for erasing another writing line 55 when the device 2 is in the eraser mode. When the user closes their thumbs, the distance image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the distance image data and recognizes that the gesture operation is for moving the virtual eraser away from the display 3.

[0185] When the user moves one finger, the display control unit 13 moves the eraser icon 81 to follow the coordinates pointed at. Next, when the user extends their thumb while keeping one finger pointed, the gesture recognition unit 18 recognizes that the virtual eraser is in contact with the display 3 (the mode changes to one in which the virtual eraser is in contact with the display 3). The display control unit 13 may change the color or shape of the eraser icon 81 to indicate that the virtual eraser is in contact with the display 3. When the user moves one finger in the mode in which the virtual eraser is in contact with the display 3, the display control unit 13 erases the writing line 55 drawn at the coordinates pointed at. Therefore, the user can change the position of the eraser icon 81 and erase it again.

[0186] When the user closes one finger, the range image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the range image data and recognizes that this is a gesture to end the eraser mode. The hierarchical control unit 17 ends the eraser mode.

[0187] <<The process from starting to ending eraser mode>> 36 is a flowchart illustrating the process of a user performing a gesture operation to transition the device 2 to the eraser mode and then terminating the process. At the start of FIG. 36, the device 2 is in the pointer mode.

[0188] The gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of waving three fingers vertically has been recognized (S51). If the determination in step S51 is Yes, the process proceeds to step S52, and if No, the process proceeds to step S63.

[0189] In step S52, the hierarchical control unit 17 transitions to the eraser mode (S52).

[0190] Next, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of pointing one finger toward the display 3 has been recognized (S53). If the determination in step S53 is Yes, the process proceeds to step S54; if the determination is No, the process repeats step S53.

[0191] In step S54, the pointing position detection unit 16 calculates the coordinates where one finger is pointing, and the display control unit 13 displays the eraser icon 81 at the coordinates (S54).

[0192] Next, the pointing position detection unit 16 determines whether or not movement of one finger has been detected based on the coordinates of the pointing finger (S55). If the determination in step S55 is Yes, the process proceeds to step S56, and if No, the process proceeds to step S57.

[0193] In step S56, the display control unit 13 moves the eraser icon 81 to the coordinates where the one finger is pointing, calculated by the pointing position detection unit 16 (S56).

[0194] In step S57, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of extending the thumb while pointing one finger has been recognized (S57). If the determination in step S57 is Yes, the process proceeds to step S58, and if No, the process returns to step S55.

[0195] In step S58, the gesture recognition unit 18 recognizes that the mode is one in which a virtual eraser touches the touch panel. The pointing position detection unit 16 determines whether or not movement of one finger has been detected based on the coordinates of the pointing finger (S58). If the determination in step S58 is Yes, the process proceeds to step S59; if the determination is No, the process proceeds to step S60.

[0196] In step S59, the display control unit 13 erases the written line that exists on the locus of coordinates where the one finger is pointing, which is calculated by the indication position detection unit 16 (S59).

[0197] In step S60, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of closing the thumbs has been recognized (S60). If the determination in step S60 is Yes, the process proceeds to step S61, and if No, the process returns to step S58.

[0198] In step S61, it is determined that the mode is one in which the virtual eraser is separated from the touch panel. The gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of closing one finger has been recognized (S61). If the determination in step S61 is Yes, the process proceeds to step S62; if No, the process returns to step S55.

[0199] In step S62, since a gesture operation of closing one finger is recognized, the hierarchical control unit 17 ends the eraser mode (S62), and the eraser mode transitions to the pointer mode.

[0200] In step S63, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not another gesture operation has been recognized (S63). The other gesture operation is a gesture operation of waving two fingers sideways, waving three fingers sideways, or waving four fingers side to side. If the determination in step S63 is Yes, the process proceeds to step S64; if the determination is No, the process returns to step S51.

[0201] In step S64, the gesture recognition unit 18 executes a process corresponding to another gesture operation (S64).

[0202] 36, the gesture operation for starting the eraser mode is a vertical wave of three fingers, but the gesture operation for starting the eraser mode may be any gesture operation that is easy for the user to perform with their hand. For example, this gesture operation may be a horizontal wave of two fingers, a vertical wave of two fingers, a horizontal wave of three fingers, etc.

[0203] In Fig. 36, closing one finger is the gesture operation for ending the pen mode, but the gesture operation for ending the eraser mode can be any gesture operation that is easy for the user to perform with their hand. For example, this gesture operation can be extending only one finger and the little finger, or extending only the little finger, etc.

[0204] Also in the eraser mode, similarly to FIGS. 23 to 26, the display control unit 13 may enlarge only the menu buttons to be used in the eraser mode and hide the menu buttons that are not used.

[0205] As described above, in the pen mode, marker mode, and eraser mode, the gesture recognition unit 18 recognizes, from the same gesture operation, a gesture operation for bringing the pen down or a gesture operation for bringing the virtual eraser into contact with the display 3, and recognizes, from the same gesture operation, a gesture operation for bringing the pen up or a gesture operation for removing the virtual eraser from the display 3. Furthermore, in the pen mode, marker mode, and eraser mode, the gesture recognition unit 18 recognizes, from the same gesture operation, a gesture operation for ending the pen mode, marker mode, or eraser mode.

[0206] <Page switching> 37 shows a gesture operation for page switching. When the user points four fingers at the device 2 and waves them to the right or left, the range image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the range image data and detects that the four fingers have been waved to the right or left. As a result, the acceptance unit 14 accepts an operation to switch to the previous page (if waved to the left) or to switch to the next page (if waved to the right).

[0207] 37, the gesture operation for page turning is a horizontal wave of four fingers, but the gesture operation for page turning may be any gesture operation that is easy for the user to perform with their hands. For example, the gesture operation may be a horizontal wave of two fingers, a vertical wave of two fingers, a horizontal wave of three fingers, a vertical wave of three fingers, a vertical wave of four fingers, etc.

[0208] <<Page switching flow>> 38 is a flowchart illustrating a process in which the user switches pages by performing a gesture operation. The device 2 may be in any of the pointer mode, pen mode, marker mode, and eraser mode.

[0209] The gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of waving four fingers to the left has been recognized (S71). If the determination in step S71 is Yes, the process proceeds to step S72, and if No, the process proceeds to step S73.

[0210] In step S72, the display control unit 13 switches the currently displayed page to the previous page (S72).

[0211] In step S73, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of waving four fingers to the right has been recognized (S73). If the determination in step S73 is Yes, the process proceeds to step S74, and if No, the process in FIG. 38 ends.

[0212] In step S74, the display control unit 13 switches the currently displayed page to the next page (S74).

[0213] <Major Effects> The device 2 of this embodiment recognizes gesture operations preset in each of three layers. By hierarchizing the gesture operations, many operations can be performed with a small number of gesture operation types, eliminating the need for the user to memorize many gesture operations. For example, in the pen mode, marker mode, and eraser mode, it is possible to write with a pen, write with a marker, and erase pen writing with one finger, respectively. If the hierarchical structure were not implemented, the user would have to learn three times as many gesture operations.

[0214] Furthermore, in the past, because there was a one-to-one correspondence between the types of gesture operations and commands for devices, it was difficult to use gesture operations to draw lines such as curves for which no commands were available. Users had to go to the device to handwrite lines. In this embodiment, a pen mode is provided, allowing users to use gesture operations to indicate a certain part of the device's display with a red line or the like while seated, eliminating the need to go to the device to perform such operations.

[0215] [Second embodiment] In this embodiment, a device 2 having multiple modes for each layer and multiple gesture operations corresponding to each mode will be described. That is, the device 2 has multiple modes per layer rather than one mode per layer, and can accept multiple gesture operations per mode.

[0216] 39 shows an example of hierarchical gesture operations possible in the device 2 of this embodiment. The device 2 has multiple modes in one layer. For example, the second layer has a pointer mode and a voice recognition mode, and the third layer has a pen mode, a marker mode, an eraser mode, a language selection mode, and an industry selection mode. The gesture operations for transitioning from (2) gesture recognition mode in the first layer to (3) pointer mode, or from (3) pointer mode to (4) pen mode, (5) marker mode, or (6) eraser mode may be the same as those in the first embodiment.

[0217] In addition, in the voice recognition mode, language selection mode, and industry selection mode, gesture operations such as pointing with the index finger, extending the thumb, closing the thumb, and shaking four fingers left and right are possible.

[0218] (7) When device 2 recognizes, for example, a gesture operation of drawing a circle with the palm (an example of a first gesture operation) in gesture recognition mode, it transitions to voice recognition mode. Voice recognition mode is a mode in which the user's voice is recognized and converted into text data. When device 2 recognizes, for example, a gesture operation of waving two fingers sideways in voice recognition mode, it transitions to language selection mode. When device 2 recognizes, for example, a gesture operation of waving three fingers sideways in voice recognition mode, it transitions to industry selection mode. (8) The language selection mode is a mode that accepts the selection of a language to be recognized by voice in the voice recognition mode. When the device 2 transitions to the language selection mode, it displays a list of languages. When the user moves the pointer to a desired language with one finger and extends their thumb, the device 2 accepts the selection of the language at the pointer position. (9) The industry selection mode is a mode that accepts the selection of a voice recognition dictionary corresponding to the industry to be used for voice recognition. Since different dictionaries are used for voice recognition depending on the industry, specialized voice recognition dictionaries are required for medical and construction industries. When device 2 transitions to industry selection mode, it displays a list of industries. When the user moves the pointer with one finger to a desired industry and extends their thumb, device 2 accepts the selection of the industry at the pointer position.

[0219] In this way, in the device 2 of this embodiment, one layer has multiple modes, and when transitioning to a lower layer, the layer also has multiple modes. Therefore, the user can become familiar with gesture operations for each layer and each mode, and can perform many operations with a small number of gesture operations.

[0220] <About the function> Figure 40 is a functional block diagram illustrating the functions of device 2 in this embodiment, broken down into blocks. The explanation of Figure 40 will mainly focus on the differences from Figure 6. Device 2 newly includes a voice recognition unit 25 and a pronunciation dictionary storage unit 26.

[0221] The speech recognition unit 25 extracts speech features (acoustic analysis) from PCM-encoded speech data input from the microphone 440, extracts phonemes, identifies words using a pronunciation dictionary, creates sentences from these word groups, and outputs text data. Note that a well-known language model for creating sentences from word groups is the DNN-HMM (Deep Neural Network-Hidden Markov Mode) using an RNN (Recurrent Neural Network).

[0222] The pronunciation dictionary storage unit 26 stores pronunciation dictionaries specialized for specific industries such as medicine, construction, and chemistry in addition to general-purpose pronunciation dictionary data. The pronunciation dictionary storage unit 26 is formed, for example, on the SSD 404, but may be located on a network. The voice recognition unit 25 reads out pronunciation dictionary data selected by the user from the pronunciation dictionary storage unit 26 and uses it.

[0223] <Flow from starting to ending voice recognition mode> 41 is a flowchart illustrating a process in which a user uses a gesture operation to transition device 2 to the voice recognition mode. At the start of Fig. 41, device 2 has recognized the palm of the hand and is in the gesture recognition mode.

[0224] The user draws a circle with the palm of the hand (S101). The gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not the gesture operation of drawing a circle with the palm of the hand has been recognized (S102).

[0225] If the gesture recognition unit 18 recognizes this gesture operation (Yes in S102), the hierarchical control unit 17 transitions to the voice recognition mode (S103).

[0226] In Fig. 41, drawing a circle with the palm is the gesture operation for starting the voice recognition mode, but the gesture operation for starting the voice recognition mode may be any gesture operation that is easy for the user to perform with their hand. For example, this gesture operation may be shaking the palm downward, opening and closing five fingers, turning the palm over, etc.

[0227] 42 is a flowchart illustrating a process for the user to end the voice recognition mode by a gesture operation, where the device 2 is in the voice recognition mode.

[0228] In the voice recognition mode, the user shakes their palm down (S111). The gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a palm-down gesture operation has been recognized (S112).

[0229] If the gesture recognition unit 18 recognizes this gesture operation (Yes in S112), the hierarchical control unit 17 ends the voice recognition mode (S113). When the gesture recognition unit 18 recognizes a gesture operation that ends recognition of the gesture operations classified in the second hierarchical level in the second hierarchical level (voice recognition mode), the gesture recognition unit 18 becomes able to recognize gesture operations classified in the highest hierarchical level (gesture recognition mode).

[0230] 42, shaking the palm downward is the gesture operation for terminating the voice recognition mode, but the gesture operation for terminating the voice recognition mode may be any gesture operation that is easy for the user to perform with their hand. For example, this gesture operation may be clenching one finger, extending only one finger and the little finger, or extending only the little finger.

[0231] <Starting and ending language selection mode> Next, with reference to FIG. 43 etc., a description will be given of gesture operations from the start to the end of the language selection mode. FIG. 43 shows gesture operations for causing the device 2 to display the pointer 51 in the language selection mode. When the device 2 transitions to the language selection mode, the display control unit 13 displays a list of languages ​​311. When the user points one finger toward the device 2, the range image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the range image data and detects that one finger is pointing at the display 3.

[0232] When it is detected that one finger is pointing at the display 3 in the language selection mode, the pointing position detection unit 16 calculates the coordinates of the pointing point using image data input from the finger photographing camera (right) 471 and the finger photographing camera (left) 472. The display control unit 13 displays the pointer 51 at these coordinates. When the user moves one finger in this state, the display control unit 13 moves the pointer 51 to follow the pointing coordinates. When the user extends his or her thumb, the device 2 accepts the language selection.

[0233] <<Flow from starting to ending the language selection mode>> Fig. 44 is a flowchart illustrating the process from when the user enters language selection mode using a gesture operation to when the mode is terminated. At the start of Fig. 44, device 2 is in voice recognition mode because it has recognized a palm-drawing gesture operation.

[0234] The gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of shaking two fingers sideways has been recognized (S121). If the determination in step S121 is Yes, the process proceeds to step S122, and if No, the process proceeds to step S132.

[0235] In step S122, the hierarchical control unit 17 transitions to the language selection mode (S122). The display control unit 13 causes the display 3 to display a list of languages ​​311.

[0236] Next, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of pointing one finger toward the display 3 has been recognized (S123). If the determination in step S123 is Yes, the process proceeds to step S124; if the determination is No, the process repeats step S123.

[0237] In step S124, the pointing position detection unit 16 calculates the coordinates where one finger is pointing, and the display control unit 13 displays the pointer 51 at the calculated coordinates (S124).

[0238] Next, the pointing position detection unit 16 determines whether or not movement of one finger has been detected based on the coordinates of the pointing finger (S125). If the determination in step S125 is Yes, the process proceeds to step S126, and if No, the process proceeds to step S127.

[0239] In step S126, the display control unit 13 moves the pointer 51 to the coordinates at which the one finger is pointing, calculated by the pointing position detection unit 16 (S126).

[0240] In step S127, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of extending the thumb while pointing one finger has been recognized (S127). If the determination in step S127 is Yes, the process proceeds to step S128, and if No, the process returns to step S125.

[0241] In step S128, the accepting unit 14 accepts the selection of the language displayed at the coordinates pointed to by the one finger detected by the pointing position detecting unit 16 (S128).

[0242] In step S129, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of closing the thumbs has been recognized (S129). If the determination in step S129 is Yes, the process proceeds to step S130, and if No, the process returns to step S129.

[0243] In step S130, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of closing one finger has been recognized (S130). If the determination in step S130 is Yes, the process proceeds to step S131, and if No, the process returns to step S125.

[0244] In step S131, since a gesture operation of closing one finger is recognized, the hierarchical control unit 17 ends the language selection mode (S131). The language selection mode transitions to the voice recognition mode. When the gesture recognition unit 18 recognizes a gesture operation that ends recognition of a gesture operation classified in the third hierarchical level (e.g., language selection mode) in the third hierarchical level, it becomes possible to recognize gesture operations classified in the second hierarchical level (e.g., voice recognition mode). Note that the gesture operation that ends the language selection mode is the same as the gesture operation that ends the industry selection mode. This reduces the number of gesture operations that the user needs to remember.

[0245] In step S132, the gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not another gesture operation has been recognized (S132). The other gesture operation is, for example, a gesture operation of shaking three fingers horizontally, shaking three fingers vertically, or shaking four fingers left and right. If the determination in step S132 is Yes, the process proceeds to step S133; if the determination is No, the process returns to step S121.

[0246] In step S133, the gesture recognition unit 18 executes a process corresponding to another gesture operation (S133).

[0247] 44, the gesture operation for starting the language selection mode is a horizontal wave of two fingers, but the gesture operation for starting the language selection mode may be any gesture operation that is easy for the user to use with their hands. For example, the gesture operation may be a vertical wave of two fingers, a horizontal wave of three fingers, a vertical wave of three fingers, etc.

[0248] 44, closing one finger is the gesture operation for ending the language selection mode, but the gesture operation for ending the language selection mode may be any gesture operation that is easy for the user to perform with their hand. For example, this gesture operation may be extending only one finger and the little finger, or extending only the little finger, etc.

[0249] <From the start to the end of the industry selection mode> Next, with reference to Fig. 45, a description will be given of gesture operations from the start to the end of the industry selection mode. Fig. 45 shows gesture operations for causing the device 2 to display the pointer 51 in the industry selection mode. When the device transitions to the industry selection mode, the display control unit 13 displays a list of industries 312. When the user points one finger toward the device 2, the range image sensor 460 captures a still image or video including this gesture operation. The gesture recognition unit 18 analyzes the range image data and detects that one finger is pointing at the display 3.

[0250] In the business type selection mode, when it is detected that one finger is pointing at the display 3, the pointing position detection unit 16 calculates the coordinates of the pointing point using image data input from the finger photographing camera (right) 471 and the finger photographing camera (left) 472. The display control unit 13 displays the pointer 51 at these coordinates. When the user moves one finger in this state, the display control unit 13 moves the pointer 51 to follow the pointing coordinates. When the user extends his or her thumb, the device 2 accepts the selection of the business type.

[0251] <<Flow from start to finish of industry selection mode>> Figure 46 is a flowchart illustrating the process from when a user transitions to industry selection mode using a gesture operation to when the mode is terminated. At the start of Figure 46, device 2 is in voice recognition mode because it has recognized a gesture operation of drawing a circle with the palm of the hand. The explanation of Figure 46 will mainly focus on the differences from Figure 44.

[0252] The gesture recognition unit 18 analyzes the range image data from the range image sensor 460 and determines whether or not a gesture operation of shaking three fingers sideways has been recognized (S141). If the determination in step S141 is Yes, the process proceeds to step S142, and if No, the process proceeds to step S152.

[0253] In step S142, the hierarchical control unit 17 transitions to an industry selection mode (S142). The display control unit 13 displays an industry list 312 on the display 3. The processing in steps S143 to S147 may be the same as in FIG.

[0254] In step S148, the reception unit 14 receives the selection of the business type displayed at the coordinates pointed to by one finger detected by the pointing position detection unit 16 (S148). The subsequent processing may be the same as in FIG.

[0255] <Major Effects> In addition to the effects of the first embodiment, the device 2 of this embodiment has a hierarchy with multiple modes, and when a user transitions to a lower hierarchy, the hierarchy also has multiple modes. Therefore, the user can become proficient in gesture operations for each hierarchy and each mode, and can perform many operations with a small number of gesture operations.

[0256] [Third embodiment] In this embodiment, a display system in a system form will be described in which an information processing system on a network performs gesture recognition and returns the recognition result, ie, a gesture operation and the coordinates where one finger is pointing, to the device 2.

[0257] In the description of this embodiment, components or contents of figures with the same symbols as those in the first or second embodiment perform similar functions, so descriptions of components that have already been described may be omitted or only differences may be described.

[0258] 47 is an example of a system configuration diagram of the display system 100. The display system 100 has a device 2 and an information processing system 700 that can communicate with each other via a network N.

[0259] The device 2 is located in a facility such as a company, and is connected to a LAN or Wi-Fi installed within the facility. The information processing system 700 is located, for example, in a data center. The device 2 is connected to the Internet i via a firewall 8, and the information processing system 700 is also connected to the Internet i via a high-speed LAN or the like within the data center.

[0260] The device 2 may connect to the Internet i using wireless communication such as a mobile phone network, in which case the wireless communication may be 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), or the like.

[0261] The information processing system 700 has one or more information processing devices, and the one or more information processing devices act as a server to provide services to the device 2. A server is a computer or software that performs the function of providing information and processing results in response to a request from a client. As will be described later, the information processing system 700 receives from the device 2 image data captured by the finger capturing camera (right) 471 and the finger capturing camera (left) 472, and distance image data captured by the distance image sensor 460. The information processing system 700 transmits the recognized gesture operation and the calculated coordinates of the pointing finger to the device 2. This reduces the processing load on the device 2.

[0262] The configuration of the device 2 may be the same as that of the first embodiment, but the device 2 of this embodiment only needs to have a touch panel, a display, a camera (a distance image sensor, a finger photographing camera), and a communication function. The device 2 may include multiple computing devices configured to communicate with each other.

[0263] In this embodiment, device 2 is a general information processing device such as a PC or tablet that runs a web browser or a dedicated application. The web browser or dedicated application communicates with information processing system 700. When the web browser is running, the user inputs or selects the URL of the information processing system 700 to connect the display device to the information processing system 700. Device 2 runs a web app provided by the information processing system 700 in the web browser. A web app refers to software or a mechanism thereof that runs on a web browser through cooperation between a program written in a programming language (for example, JavaScript (registered trademark)) that runs on the web browser and a program on the web server.

[0264] When the dedicated application runs, it connects to a pre-registered URL of the information processing system 700. The dedicated application has a program and a user interface, so the program sends and receives necessary information to and from the information processing system 700 and displays it on the user interface.

[0265] The communication method may be a general-purpose communication protocol such as HTTP, HTTPs, or WebSocket, or a dedicated communication protocol.

[0266] <Hardware configuration example> The hardware configuration of the device 2 may be the same as that shown in Fig. 5. In this embodiment, an example of the hardware configuration of an information processing system 700 will be described.

[0267] Fig. 48 is a hardware configuration diagram of an information processing system 700. As shown in Fig. 48, the information processing system 700 is constructed by a computer, and includes a CPU 601, a ROM 602, a RAM 603, an HD 604, an HDD (Hard Disk Drive) controller 605, an external device connection I / F (Interface) 608, a network I / F 609, a bus line 610, and a media I / F 616.

[0268] Of these, the CPU 601 controls the overall operation of the information processing system 700. The ROM 602 stores programs used to drive the CPU 601, such as the IPL. The RAM 603 is used as a work area for the CPU 601. The HD 604 stores various data, such as programs. The HDD controller 605 controls the reading and writing of various data from and to the HD 604 under the control of the CPU 601. The external device connection I / F 608 is an interface for connecting various external devices. In this case, the external devices are, for example, USB (Universal Serial Bus) memories and printers. The network I / F 609 is an interface for data communication using a communication network. The bus line 610 is an address bus, a data bus, or the like for electrically connecting the components, such as the CPU 601, shown in FIG. 48.

[0269] A media I / F 616 controls reading and writing (storing) of data from and to a recording medium 615 such as a flash memory.

[0270] <About the function> Next, functions of the display system 100 will be described with reference to Fig. 49. Fig. 49 is an example of a functional block diagram showing functions of the display system 100 in blocks. Note that the description of Fig. 49 will mainly focus on the differences from Fig. 6.

[0271] In this embodiment, the device 2 has a contact position detection unit 11, a handwriting data generation unit 12, a display control unit 13, a reception unit 14, a whiteboard control unit 15, a network communication unit 19, a data recording unit 20, a first data acquisition unit 21, a second data acquisition unit 22, and an object data storage unit 23. These functions may be the same as those in the first or second embodiment, or may be different from those in the first or second embodiment, but this will not hinder the explanation of this embodiment.

[0272] <<Functions of the information processing system>> The information processing system 700 has a communication unit 24, an indication position detection unit 16, a hierarchical control unit 17, and a gesture recognition unit 18. Each function of the information processing system 700 is a function or means realized when any of the components shown in Fig. 48 operates in response to an instruction from the CPU 601 in accordance with a program loaded from the HD 604 onto the RAM 603.

[0273] The communication unit 24 receives distance image data and image data captured by the finger photographing camera (right) 471 and the finger photographing camera (left) 472 from the device 2, and transmits the recognized gesture operation and the coordinates pointed by one finger to the device 2. Other functions are the same as those of the first or second embodiment, or, even if they are different, they will not hinder the explanation of this embodiment.

[0274] <<Action or Processing>> FIG. 50 is a sequence diagram illustrating a process in which the device 2 and the information processing system 700 communicate with each other to perform gesture recognition and display the coordinates pointed by one finger.

[0275] S201: First, it is assumed that the device 2 is in an initial state. The distance image sensor 460, the finger photographing camera (right) 471, and the finger photographing camera (left) 472 photograph the user.

[0276] S202: The first data acquisition unit 21 acquires image data from the finger photographing camera (right) 471 and the finger photographing camera (left) 472. The second data acquisition unit 22 acquires distance image data from the distance image sensor 460. The network communication unit 19 transmits the distance image data and the image data photographed by the finger photographing camera (right) 471 and the finger photographing camera (left) 472 to the information processing system 700. The network communication unit 19 may transmit to the information processing system 700 a message indicating that it is in an initial state.

[0277] S203: The communication unit 24 of the information processing system 700 receives these, and the gesture recognition unit 18 analyzes the range image data. The gesture recognition unit 18 determines whether or not it has recognized a palm that can be recognized in the initial state. Assume that the gesture recognition unit 18 has recognized a palm. The hierarchical control unit 17 determines that it has transitioned to gesture recognition mode because it has recognized a palm in the initial state.

[0278] S204: The communication unit 24 requests the device 2 to transition to the gesture recognition mode.

[0279] S205: The network communication unit 19 receives this request, and the device 2 transitions to gesture recognition mode.

[0280] The processes of S206 to S214 are repeatedly executed until the initial state is returned to.

[0281] S206: The distance image sensor 460, the finger photographing camera (right) 471, and the finger photographing camera (left) 472 photograph the user.

[0282] S207: The network communication unit 19 transmits the current level and mode, the distance image data, and the image data captured by the finger photographing camera (right) 471 and the finger photographing camera (left) 472 to the information processing system 700.

[0283] S208: The communication unit 24 of the information processing system 700 receives and analyzes the range image data. The gesture recognition unit 18 recognizes a gesture operation according to the current layer and mode.

[0284] S209: In the case of a gesture operation for transitioning between layers or modes, the layer control unit 17 determines the layer and mode corresponding to the recognized gesture operation.

[0285] S210: The communication unit 24 of the information processing system 700 transmits the determined layer and mode to the device 2.

[0286] S211: The network communication unit 19 of the device 2 receives the hierarchy and mode, and the device 2 transitions to this hierarchy and mode. For example, the display control unit 13 displays a menu corresponding to the pointer mode, pen mode, marker mode, eraser mode, voice recognition mode, language selection mode, or business type selection mode.

[0287] S212: If a gesture operation that is valid in the current mode and not a gesture operation for transitioning between layers or modes is recognized, and the gesture recognition unit 18 recognizes one finger, the indication position detection unit 16 calculates the coordinates pointed to by the one finger. Also, if the gesture recognition unit 18 recognizes a gesture operation of extending or folding the thumb, it determines that the mode is pen-down mode or pen-up mode. In the eraser mode, pen-down means transition to a mode in which a virtual eraser touches the display, and pen-up means transition to a mode in which the virtual eraser is removed from the display. If the gesture recognition unit 18 recognizes a gesture operation of waving four fingers left and right, it detects page switching.

[0288] S213: The communication unit 24 of the information processing system 700 transmits to the device 2 the coordinates pointed by one finger, the coordinates pointed by one finger and pen-down mode, the coordinates pointed by one finger and pen-up mode, or the page change.

[0289] S214: The network communication unit 19 of the device 2 receives these and executes processing related to the pointer, writing line, highlighter, eraser, page switching, or voice recognition.

[0290] <Major Effects> According to this embodiment, in addition to the effects of the first or second embodiment, the device 2 can be controlled by gesture operations if the device 2 is connected to a network.

[0291] [Other application examples] The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0292] For example, in the above embodiment, the device 2 transitions to the gesture recognition mode by recognizing the palm of a hand in the initial state, but the method of transitioning to the gesture recognition mode is not limited to this, and the device may transition to the gesture recognition mode by recognizing a predetermined shape of an arbitrary object or a predetermined movement of an arbitrary object. Furthermore, gesture operations for transitioning to the pointer mode, pen mode, marker mode, or eraser mode and gesture operations for page switching may also be gestures other than those described above.

[0293] For example, device 2 may be called an electronic whiteboard, an electronic information board, or the like. This embodiment is not limited to electronic whiteboards and can be suitably applied to any information processing device having a touch panel. Examples of information processing devices equipped with a touch panel include PCs, tablet terminals, and smartphones that have touch panels. These are generally general-purpose information processing devices, but when an application that functions as a device is executed, the user can operate them as device 2.

[0294] Alternatively, the device 2 may be a device that projects an image using a projector. In this case, the device 2 detects the coordinates pointed to by the finger using the method described in this embodiment, and the projector projects a handwritten line or the like based on the trajectory of the coordinates.

[0295] Furthermore, in this embodiment, the device 2 recognizes the gesture operation and calculates the coordinates where the user points with one finger. However, at least one of the gesture operation recognition and coordinate calculation may be performed by a server device connected via a network. In this case, the device 2 transmits image data captured by the finger capturing camera (right) 471 and the finger capturing camera (left) 472 and distance image data captured by the distance image sensor 460 to the server device in real time. The server device transmits the recognized gesture operation and the calculated coordinates to the device 2. This reduces the processing load on the device 2.

[0296] In addition, in this embodiment, the device 2 calculates the coordinates pointed by the index finger, but other fingers may be used to point to the coordinates on the display 3. In addition, in this embodiment, extending the thumb is a pen-down gesture operation and folding the thumb is a pen-up gesture operation, but the pen-down and pen-up gesture operations may be other gesture operations.

[0297] Furthermore, the device 2 may display the current mode or state on the display 3. For example, the device 2 may display "initial state," "gesture recognition mode," "pen mode," "marker mode," "eraser mode," or other such mode using text or icons in the upper right corner of the display 3. In this case, the device 2 may display valid gesture operations in the current mode or state using animations or other means. Alternatively, when the user presses the icon, an animation or video guiding the user through the gesture operations may be played.

[0298] In addition, the configuration examples in Figure 6 and the like are divided according to main functions to make it easier to understand the processing by device 2. The method of dividing the processing units or the names of the processing units does not limit the present invention. The processing by device 2 can be divided into even more processing units depending on the processing content. Also, it can be divided so that one processing unit includes even more processes.

[0299] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0300] Embodiments of the present invention provide significant improvements in computer power and functionality. These improvements allow users to utilize computers that provide more efficient and robust interaction with tables, which are ways of storing and presenting information in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of more efficient, powerful, and robust user interfaces. Such user interfaces provide better interaction between humans and machines.

[0301] <Aspect> [Aspect 1] A device that recognizes a gesture operation corresponding to a movement of a pointing object and accepts an operation corresponding to the gesture operation, a data acquisition unit that acquires information about the shape of the pointing object from a sensor that acquires the information; a gesture recognition unit that recognizes the gesture operations that can be recognized in each of the at least three hierarchical levels into which the gesture operations are classified, based on the information acquired by the data acquisition unit; When the gesture recognition unit recognizes a first gesture operation in the highest hierarchy, the gesture recognition unit becomes capable of recognizing a gesture operation classified in a second hierarchy; The device is characterized in that, when a second gesture operation is recognized in the second layer, it becomes possible to recognize gesture operations classified in a third layer. [Aspect 2] the gesture recognition unit becomes capable of recognizing gesture operations classified in the highest hierarchy when it recognizes a gesture operation that ends recognition of the gesture operations classified in the second hierarchy in the second hierarchy, and The device described in aspect 1 is characterized in that the gesture recognition unit becomes capable of recognizing gesture operations classified in the second hierarchy when it recognizes a gesture operation that ends recognition of gesture operations classified in the third hierarchy at the third hierarchy. [Aspect 3] a hierarchical level control unit that transitions the current hierarchical level to a higher or lower hierarchical level; 3. The device of claim 1 or 2. [Aspect 4] a hierarchical level control unit that determines that the hierarchical level is the highest level when the pointing object is recognized in an initial state; 3. The device according to aspect 1 or 2, wherein the gesture recognition unit is capable of recognizing gesture operations classified in the highest hierarchy. [Aspect 5] the second layer has a plurality of modes for using predetermined functions of the device, When the gesture recognition unit recognizes the first gesture operation in the highest layer, the hierarchical control unit causes a transition to the second hierarchical level and also causes a transition to the mode of the second hierarchical level associated with the recognized first gesture operation. 5. The device of embodiment 4, [Aspect 6] an imaging device that captures an image of the user's hand, which is the pointing object, and a display of the device; a display that displays handwritten lines input by a pen or finger on a touch panel; a pointing position detection unit that analyzes image data captured by the image capture device and detects the coordinates at which one finger of the user points on the display, When the gesture recognition unit recognizes a gesture operation for transitioning to a pointer mode as a mode switched by the first gesture operation in the highest hierarchy, the hierarchy control unit transitions from the highest hierarchy to a pointer mode included in the second hierarchy, a display control unit that displays a pointer at the coordinates of the display detected by the pointed position detection unit; 6. The device of claim 5, comprising: [Aspect 7] the third layer has a plurality of modes for using predetermined functions of the device, When the gesture recognition unit recognizes the second gesture operation in the second layer, the hierarchical control unit causes a transition to the third hierarchical level and also causes a transition to the mode of the third hierarchical level associated with the recognized second gesture operation. 5. The device of embodiment 4, [Aspect 8] the plurality of modes in the second layer are associated with one or more second gesture operations that can be recognized by the gesture recognition unit, When the gesture recognition unit recognizes the second gesture operation associated with the mode in the mode included in the second layer, the hierarchical control unit causes a transition to the third hierarchical level and also causes a transition to the mode associated with the recognized second gesture operation. 8. The device of embodiment 7, [Aspect 9] When the hierarchical control unit transitions the third hierarchical layer to a pen mode among the plurality of modes of the third hierarchical layer, The device according to aspect 7 or 8, wherein the gesture recognition unit recognizes a gesture operation for displaying a writing line on a display. [Aspect 10] an imaging device that captures an image of the user's hand, which is the pointing object, and a display of the device; an indication position detection unit that analyzes image data captured by the imaging device and detects the coordinates at which one finger of the user points on the display; a display control unit that displays a writing line that is a locus of the coordinates detected by the indication position detection unit, In the pen mode, when the gesture recognition unit recognizes a gesture operation of pointing with one finger and then lowering the pen, it determines that the mode is a pen-down mode, When the gesture recognition unit recognizes a gesture operation of lifting the pen, it determines that the mode is a pen-up mode, The device according to aspect 9, wherein the display control unit terminates the drawing of the line. [Aspect 11] an imaging device that captures an image of the user's hand, which is the pointing object, and a display of the device; a display control unit that, in the pen mode, displays a pen icon at the coordinates detected by the indication position detection unit; 11. The device of claim 10, comprising: [Aspect 12] In the pen mode, when the gesture recognition unit recognizes a gesture operation of pointing with one finger and then lowering the pen, it determines that the mode is a pen-down mode, The device according to aspect 11, wherein the display control unit displays the pen icon in a mode different from the mode before the pen-down mode. [Aspect 13] When the hierarchical control unit transitions the third hierarchical layer to a marker mode among the plurality of modes that the third hierarchical layer has, The device according to aspect 7 or 8, wherein the gesture recognition unit recognizes a gesture operation for displaying a marker line on a display. [Aspect 14] an imaging device that captures an image of the user's hand, which is the pointing object, and a display of the device; and a pointing position detection unit that analyzes image data captured by the image capturing device and detects the coordinates at which one finger of the user points on the display. In the marker mode, when the gesture recognition unit recognizes a gesture operation of pointing with one finger and then downing the pen, it determines that the mode is a pen-down mode, the pointing position detection unit detects a coordinate at which the one finger points on the display; a display control unit that displays the marker line, which is a locus of the coordinates detected by the indication position detection unit; When the gesture recognition unit recognizes a gesture operation of lifting the pen, it determines that the mode is a pen-up mode, The device according to aspect 13, wherein the display control unit terminates writing of the marker line. [Aspect 15] a display control unit that, in the marker mode, displays a marker icon at the coordinates detected by the pointed position detection unit; 15. The device of claim 14, comprising: [Aspect 16] In the marker mode, when the gesture recognition unit recognizes a gesture operation of pointing with one finger and then lowering the pen, it determines that the mode is a pen-down mode, The device according to aspect 15, wherein the display control unit displays the marker icon in a mode different from the mode before the pen-down mode. [Aspect 17] In the pen mode, marker mode, and eraser mode, which are one of the third hierarchical levels, the gesture recognition unit recognizes a gesture operation of bringing a pen down or a gesture operation of bringing a virtual eraser into contact with a display of the device from the same gesture operation; The device according to any one of aspects 7 to 16, wherein a gesture operation for lifting a pen or a gesture operation for removing a virtual eraser from the display is recognized from the same gesture operation. [Aspect 18] In the pen mode, marker mode, and eraser mode, which are one of the third hierarchical levels, 17. The device according to any one of aspects 7 to 16, wherein the gesture recognition unit recognizes gesture operations for terminating the pen mode, the marker mode, and the eraser mode from the same gesture operation. [Aspect 19] In the pen mode, marker mode, and eraser mode, which are one of the third hierarchical levels, a display control unit that displays only menu buttons used in the pen mode, the marker mode, and the eraser mode, and hides menu buttons that are not used; The device according to any one of aspects 7 to 16. [Explanation of symbols]

[0302] 2 equipment [Prior art documents] [Patent documents]

[0303] [Patent Document 1] Special Publication No. 2020-532007

Claims

1. A device that recognizes a gesture operation corresponding to a movement of a pointing object and accepts an operation corresponding to the gesture operation, a data acquisition unit that acquires information about the shape of the pointing object from a sensor that acquires the information; a gesture recognition unit that recognizes the gesture operations that can be recognized in each of the at least three hierarchical levels into which the gesture operations are classified, based on the information acquired by the data acquisition unit; when the gesture recognition unit recognizes a first gesture operation in the highest hierarchy, it becomes capable of recognizing a gesture operation classified in a second hierarchy; When a second gesture operation is recognized in the second hierarchy, the device becomes capable of recognizing gesture operations classified in a third hierarchy.

2. the gesture recognition unit becomes capable of recognizing gesture operations classified in the highest hierarchical level when it recognizes, in the second hierarchical level, a gesture operation that ends recognition of the gesture operations classified in the second hierarchical level; The device according to claim 1 , wherein the gesture recognition unit becomes capable of recognizing gesture operations classified in the second hierarchical level when it recognizes a gesture operation that ends recognition of gesture operations classified in the third hierarchical level in the third hierarchical level.

3. a hierarchical level control unit that transitions the current hierarchical level to a higher or lower hierarchical level; 3. The device according to claim 1 or 2.

4. a hierarchical level control unit that determines that the hierarchical level is the highest level when the pointing object is recognized in an initial state; The device according to claim 1 or 2, wherein the gesture recognition unit is capable of recognizing gesture operations classified in the highest hierarchy.

5. the second layer has a plurality of modes for using predetermined functions of the device, When the gesture recognition unit recognizes the first gesture operation in the highest layer, the hierarchical control unit causes a transition to the second hierarchical level and also causes a transition to the mode of the second hierarchical level associated with the recognized first gesture operation.

5. The device according to claim 4.

6. an imaging device that captures an image of the user's hand, which is the pointing object, and a display of the device; a display that displays handwritten lines input by a pen or finger on a touch panel; a pointing position detection unit that analyzes image data captured by the image capture device and detects coordinates at which one finger of the user points on the display, When the gesture recognition unit recognizes a gesture operation for transitioning to a pointer mode as a mode switched by the first gesture operation in the highest hierarchy, the hierarchy control unit transitions from the highest hierarchy to a pointer mode included in the second hierarchy, a display control unit that displays a pointer at the coordinates of the display detected by the pointed position detection unit; 6. The device of claim 5, further comprising:

7. the third layer has a plurality of modes for using predetermined functions of the device, When the gesture recognition unit recognizes the second gesture operation in the second layer, the hierarchical control unit causes a transition to the third hierarchical level and also causes a transition to the mode of the third hierarchical level associated with the recognized second gesture operation.

5. The device according to claim 4.

8. the plurality of modes in the second layer are associated with one or more second gesture operations that can be recognized by the gesture recognition unit, When the gesture recognition unit recognizes the second gesture operation associated with the mode in the mode included in the second hierarchical level, the hierarchical control unit causes a transition to the third hierarchical level and also causes a transition to the mode associated with the recognized second gesture operation.

8. The device of claim 7.

9. When the hierarchical control unit transitions the third hierarchical layer to a pen mode among the plurality of modes of the third hierarchical layer, The device according to claim 7 , wherein the gesture recognition unit recognizes a gesture operation for displaying a written line on a display.

10. an imaging device that captures an image of the user's hand, which is the pointing object, and a display of the device; an indication position detection unit that analyzes image data captured by the imaging device and detects coordinates at which one finger of the user points on the display; a display control unit that displays a writing line that is a locus of the coordinates detected by the indication position detection unit, In the pen mode, when the gesture recognition unit recognizes a gesture operation of pointing with one finger and then lowering the pen, it determines that the mode is a pen-down mode, When the gesture recognition unit recognizes a gesture operation of lifting the pen, it determines that the mode is a pen-up mode, The device according to claim 9 , wherein the display control unit terminates the drawing of the writing line.

11. an imaging device that captures an image of the user's hand, which is the pointing object, and a display of the device; a display control unit that, in the pen mode, displays a pen icon at the coordinates detected by the indication position detection unit; 11. The device of claim 10, comprising:

12. In the pen mode, when the gesture recognition unit recognizes a gesture operation of pointing with one finger and then lowering the pen, it determines that the mode is a pen-down mode, The device of claim 11 , wherein the display control unit displays the pen icon in a manner different from a mode prior to the pen-down mode.

13. When the hierarchical control unit transitions the third hierarchical layer to a marker mode among the plurality of modes of the third hierarchical layer, The device according to claim 7 , wherein the gesture recognition unit recognizes a gesture operation for displaying a marker line on a display.

14. an imaging device that captures an image of the user's hand, which is the pointing object, and a display of the device; and a pointing position detection unit that analyzes image data captured by the image capture device and detects the coordinates at which one finger of the user points on the display. In the marker mode, when the gesture recognition unit recognizes a gesture operation of pointing with one finger and then lowering the pen, it determines that the mode is a pen-down mode, the pointing position detection unit detects a coordinate at which the one finger points on the display; a display control unit that displays the marker line, which is a locus of the coordinates detected by the indication position detection unit; When the gesture recognition unit recognizes a gesture operation of lifting the pen, it determines that the mode is a pen-up mode, The device according to claim 13 , wherein the display control unit terminates the writing of the marker line.

15. a display control unit that, in the marker mode, displays a marker icon at the coordinates detected by the pointed position detection unit; 15. The device of claim 14, comprising:

16. In the marker mode, when the gesture recognition unit recognizes a gesture operation of pointing with one finger and then lowering the pen, it determines that the mode is a pen-down mode, The device of claim 15 , wherein the display control unit displays the marker icon in a manner different from a mode prior to the pen-down mode.

17. In the pen mode, marker mode, and eraser mode, which are one of the third layer, the gesture recognition unit recognizes a gesture operation of bringing a pen down or a gesture operation of bringing a virtual eraser into contact with a display of the device from the same gesture operation; The device according to claim 7 , wherein a gesture operation for lifting a pen or a gesture operation for removing a virtual eraser from the display is recognized from the same gesture operation.

18. In the pen mode, marker mode, and eraser mode, which are one of the third layer, The device according to claim 7 , wherein the gesture recognition unit recognizes gesture operations for terminating the pen mode, the marker mode, and the eraser mode from the same gesture operation.

19. In the pen mode, marker mode, and eraser mode, which are one of the third layer, a display control unit that displays only menu buttons used in the pen mode, the marker mode, and the eraser mode, and hides menu buttons that are not used; 8. The device of claim 7.

20. A gesture recognition method performed by a device that recognizes a gesture operation corresponding to a movement of a pointing object and accepts an operation corresponding to the gesture operation, comprising: a process in which a data acquisition unit acquires information about the shape of the pointing object from a sensor; a process in which a gesture recognition unit recognizes the gesture operations that can be recognized in each of the at least three hierarchical levels into which the gesture operations are classified, based on the information acquired by the sensor; when the gesture recognition unit recognizes a first gesture operation in the highest hierarchy, it becomes capable of recognizing a gesture operation classified in a second hierarchy; A gesture recognition method characterized in that, when a second gesture operation is recognized in the second layer, a gesture operation classified in a third layer can be recognized.

21. a device that recognizes a gesture operation corresponding to a movement of a pointing object and accepts an operation corresponding to the gesture operation, a data acquisition unit that acquires information about the shape of the pointing object from a sensor that acquires the information; a gesture recognition unit that recognizes the gesture operations that can be recognized in each of the at least three hierarchical levels into which the gesture operations are classified, based on the information acquired by the sensor; when the gesture recognition unit recognizes a first gesture operation in the highest hierarchy, it becomes capable of recognizing a gesture operation classified in a second hierarchy; The program is characterized in that, when a second gesture operation is recognized in the second hierarchy, a gesture operation classified in a third hierarchy can be recognized.

22. A display system in which a device that recognizes a gesture operation corresponding to a movement of a pointing object and accepts an operation corresponding to the gesture operation communicates with an information processing system via a network, The device comprises: a data acquisition unit that acquires information about the shape of the pointing object from a sensor that acquires the information; a network communication unit that transmits the information to the information processing system; The information processing system includes: a gesture recognition unit that analyzes the information received from the device and recognizes the gesture operations that are recognizable in each of at least three hierarchical levels into which the gesture operations are classified, based on the information acquired by the sensor; when the gesture recognition unit recognizes a first gesture operation in the highest hierarchy, it becomes capable of recognizing a gesture operation classified in a second hierarchy; When the second gesture operation is recognized in the second hierarchy, the gesture operation classified in the third hierarchy can be recognized; a communication unit that transmits the layer recognized by the gesture recognition unit to the device; The device comprises: Execute a process in the layer transmitted from the information processing system. Display system.

Images