Display device, operation restriction method, and program

Abstract

To restrict gesture-based operation during writing on a display device.SOLUTION: A display device outputs drawing data on the basis of touch data and includes a touch sensor input acquisition unit which acquires the touch data from a tough sensor, a gesture sensor input acquisition unit which acquires gesture data from a gesture sensor, and a gesture operation restriction unit which restricts operation based on the gesture data while the touch sensor input acquisition unit continuously acquires the touch data.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a display device, an operation restriction method, and a program. [Background technology]

[0002] Display devices such as electronic whiteboards use touch sensors to identify the position (coordinates) where a user touches the display with a pen or finger. Display devices may also use one or more sensors in addition to the touch sensor to detect user operations in order to accommodate various user usage scenarios. For example, a gesture sensor is known that detects a user's gesture as an operation without the user touching the display. In this way, there are known use cases in which one sensor is used as a touch sensor to detect writing operations and the other as a gesture sensor to detect gesture operations.

[0003] There is also known a technology for detecting gestures based on the movement of an electronic pen. For example, Patent Document 1 discloses a technology for performing a pre-assigned function by combining a gesture based on the movement of an electronic pen with an image displayed on an electronic whiteboard when the touch panel (display) is not in contact. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, there is a problem that the gesture sensor detects a user's operation while the touch sensor detects a user's operation. For example, a user other than the user writing on the display of the display device can operate the display device by gestures. If a user who operates by gestures operates the display device by gestures without the consent of the user who is writing, the user who is writing may feel uncomfortable because an operation that he or she did not perform is reflected on the display device. [Means for solving the problem]

[0005] The present invention is a display device that outputs drawing data based on touch data, and is characterized by having a touch sensor input acquisition unit that acquires the touch data from a touch sensor, a gesture sensor input acquisition unit that acquires gesture data from a gesture sensor, and a gesture operation restriction unit that restricts operations based on the gesture data during the period when the touch sensor input acquisition unit is continuously acquiring the touch data. [Effects of the Invention]

[0006] According to the present invention, gesture operations can be restricted while writing is being performed on the display device. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic diagram illustrating an operation or processing of a display device according to an embodiment. [Figure 2] FIG. 1 is a block diagram showing an example of a hardware configuration of a display device according to an embodiment. [Figure 3] FIG. 2 is a diagram showing an example of a functional configuration of the display device according to the embodiment. [Figure 4] FIG. 10 is a flowchart showing an example of processing performed when the display device detects an input from a gesture sensor. [Figure 5] FIG. 5 is an explanatory diagram of step S2 in FIG. 4. [Figure 6] FIG. 10 is an explanatory diagram showing an example of a gesture invalid period. [Figure 7] FIG. 10 is an explanatory diagram showing a display example when gesture operations are disabled. [Figure 8] FIG. 10 is a flowchart showing an example of processing performed when the display device detects an input from the gesture sensor in a case where some gesture-based operation contents are enabled. [Figure 9] FIG. 1 is an external view showing an example of a display device equipped with a human detection sensor. [Figure 10] FIG. 10 is a block diagram showing an example of the functional configuration of a display device including a human detection sensor. [Figure 11]FIG. 10 is a flowchart showing an example of processing when a display device equipped with a human detection sensor detects an input from a gesture sensor. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, a display device and an operation restriction method performed by the display device will be described as an example of an embodiment of the present invention.

[0009] <Outline of operation or processing> In this embodiment, during a period when a sensor with a short detection distance is detecting a user operation and for a certain period thereafter, a sensor with a long detection distance is restricted from detecting a user operation. For example, the display device disables gesture operations based on a gesture sensor with a detection distance longer than that of the touch sensor during a period when a touch sensor that detects handwriting continues to detect handwriting and for a certain period thereafter.

[0010] 1 is a schematic diagram illustrating the operation or processing of a display device according to an embodiment, and illustrates the operation or processing in which, while a user is writing on the display device 20, the display device 20 restricts a user who is not writing from operating with gestures. In the diagram, user A is writing on the display of the display device 20 with a pen 23. Meanwhile, user B is making a gesture toward the display device 20 (gesture sensor 16) to perform an operation. Various operations can be performed with gestures, such as changing pages or displaying a menu.

[0011] The display device 20 disables the gesture data or operation content based on the gesture data detected by the gesture sensor 16 during the period when the touch sensor that detects the writing operation is detecting writing, and for a certain period thereafter, thereby preventing user A from being interrupted by a gesture operation from a distance by user B while he is writing.

[0012] This makes it possible to prevent user B, who is different from user A who is writing, from operating display device 20 with gestures when display device 20 has two or more sensors with different detection distances.

[0013] As for the two sensors with different detection distances, the touch sensor that detects writing operations has a detection distance of zero because it requires contact by the user to use it for operation, but the gesture sensor has a detection distance greater than zero because it must be possible to operate it from a distance. Here, the detection distance is the distance between the sensor and the object that the sensor detects (the pen 23 or finger for the touch sensor, and the user for the gesture sensor).

[0014] <Terminology> Touch data is the coordinates of an input means (pen 23 or finger) on the touch panel detected by the touch sensor. The touch data may include information indicating the type of input means used to determine whether the pen 23 or a finger has touched the touch panel.

[0015] The gesture data is data detected by a gesture sensor, and is image data when the gesture sensor is a camera. The gesture data may also be operation details that are the analysis results of the gesture data.

[0016] The drawing data is stroke data that is displayed on a display based on the coordinates included in the touch data. The drawing data has a sequence of coordinate points in time series, and when displayed on a display, the data is displayed as lines interpolated between the coordinates.

[0017] Restricting an operation based on gesture data means that the display device 20 does not use at least a part or all of the gesture data, or a part or all of the operation content based on the gesture data. Not using means that the operation content is not reflected in the drawing data or screen displayed on the display. Not using also means that the operation content is not reflected in processes that do not appear on the display. In this embodiment, restricting an operation may be referred to as disabling the gesture data or the operation content based on the gesture data.

[0018] The period during which the touch sensor input acquisition unit continuously acquires touch data includes not only the period during which the pen 23 or finger is touching the display, but also the period during which the pen 23 or finger is removed from the display and then touches it again intermittently and repeatedly within a time period less than the threshold value.

[0019] <Configuration example> The hardware configuration of the display device 20 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of the display device 20 according to the embodiment. The display device 20 has a CPU (Central Processing Unit) 10, a display 11, a video input unit 12, a wireless communication unit 13, a memory 14, a storage device 15, a gesture sensor 16, and a touch sensor 17.

[0020] The CPU 10 controls the entire display device 20 , reads programs from the storage device 15 , and executes application programs installed on the display device 20 .

[0021] Display 11 displays image data for display received from CPU 10. Display 11 may be, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display. Display 11 may be externally attached to display device 20. Note that image data displayed by display 11 includes handwritten drawing data, character strings recognized as language information, external images (for example, from a PC (Personal Computer)), web pages, operation menus, etc.

[0022] The video input unit 12 has a function of inputting external video (for example, a PC) to the CPU 10 via a video interface. The video interface can be an existing interface such as HDMI (High-Definition Multimedia Interface) (registered trademark), Display Port, or VGA (Video Graphics Array). When the video input unit 12 receives input of video with a resolution of, for example, 1920 x 1080p and a frame rate of 60 fps via the video interface (cable), it has a function of continuously importing the 1920 x 1080 video into the CPU 10 at 60 frames per second.

[0023] The wireless communication unit 13 wirelessly connects to an external device or an access point and communicates with the external device directly or via the access point. Examples of standards used for communication include Bluetooth (registered trademark), Bluetooth Low Energy (registered trademark), and Wi-Fi (registered trademark). Note that the display device 20 may have a function for communicating via a wired cable instead of wirelessly.

[0024] The memory 14 is used to load the programs executed by the CPU 10 from the storage device 15 and to temporarily store operation data, etc. An example of the memory 14 is a volatile memory such as a DDR (Double Data Rate) memory.

[0025] The storage device 15 is used to store programs used by the CPU 10 and to store drawing data handwritten by the user on a page-by-page basis. Examples of the storage device 15 include non-volatile storage such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive).

[0026] The gesture sensor 16 detects the movement of a person at a distance from the display device 20. The detection distance is assumed to be within the same conference room (approximately 5 meters). An example of the gesture sensor 16 is a camera. The camera can determine the distance to the user by using a stereo camera or a camera equipped with LiDAR (Light Detection and Ranging). Therefore, the user may be able to set the detection distance (at least one of the upper and lower limits).

[0027] Furthermore, pattern recognition of image data makes it possible to detect people and recognize their movements (gestures). Specifically, the three-dimensional coordinates and postures of a person's head, palm, wrist, elbow, shoulder, waist, etc. can be obtained. The Kinect (registered trademark) sensor is known as a gesture sensor suitable for detecting such gestures. When using the Kinect (registered trademark) sensor, before starting a gesture operation on the display device 20, an image of the person is captured using a camera or the like provided in the Kinect (registered trademark) sensor to obtain a skeleton model (position information of the person's joints and pseudo-skeletal information connecting each joint with lines). The obtained skeleton model is stored in the storage device 15, for example, and used for analyzing gesture data in a gesture data analysis unit described later. Note that the gesture sensor may be a sensor with a similar function to the Kinect (registered trademark) sensor, a camera that detects distance using TOF (Time Of Flight), or the like.

[0028] By using a Kinect (registered trademark) sensor as the gesture sensor 16 as described above, it becomes possible to link more operations of the display device 20 with gestures, and it is possible to provide a display device with a wide variety of gesture operations.

[0029] The CPU 10 analyzes the person's movements using gesture data from the gesture sensor 16 and determines the details of the gesture operation. Gesture operations include selecting an icon or handwritten characters on the display device 20, moving pages, deleting drawing data, canceling an operation, enlarging or reducing drawing data, displaying a marker, and moving a cursor. Note that AI (artificial intelligence) may be used to analyze the gesture data. The gesture sensor 16 may be external to the display device 20.

[0030] The touch sensor 17 has an input interface function that transmits to the CPU 10 the coordinates where the user touches the touch panel with a finger or pen 23. The touch sensor 17 also has a function that can distinguish between pen input and finger input. An example of the touch sensor 17 is an optical touch sensor. In this touch sensor 17, two light-emitting / receiving devices installed at both ends of the upper side of the display emit multiple infrared rays parallel to the display. The two light-emitting / receiving devices receive light that is reflected by reflective members installed around the display and returns along the same optical path as the emitted light. The touch sensor 17 is not limited to an optical type, and may also be a capacitive touch panel that identifies the touch position by detecting changes in capacitance. The touch sensor 17 may also be a resistive touch panel that identifies the touch position by detecting voltage changes across two opposing resistive films. The touch sensor 17 may also be external to the display device 20.

[0031] <About the function> The functions of the display device 20 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the functional configuration of the display device 20 according to the embodiment. The display device 20 has a touch sensor input acquisition unit 51, a touch data analysis unit 52, a gesture sensor input acquisition unit 53, a gesture data analysis unit 54, a gesture operation restriction unit 55, and a data output unit 56. These functional units of the display device 20 are functions or means realized by the CPU 10 in Fig. 2 executing instructions of a program expanded in the memory 14.

[0032] The touch sensor input acquisition unit 51 acquires data (touch data) detected by the touch sensor 17. Acquisition includes passive reception from the touch sensor 17 and reception of touch data as a result of the touch sensor input acquisition unit 51 instructing the touch sensor 17 to acquire the data. The touch sensor input acquisition unit 51 passes the touch data to the touch data analysis unit 52.

[0033] The touch data analysis unit 52 analyzes the touch data. The touch data analysis unit 52 determines whether the touch data is from the pen 23 or a finger, and also analyzes whether the touch data is a contact with the pen 23 or a finger, or a contact with the side of the palm. The touch data analysis unit 52 passes which of the finger or the pen 23 touched which coordinate (input means type) and the coordinate to the gesture operation restriction unit 55. If the pen 23 is an electronic pen, the input means is determined by radio waves output when the pen 23 touches the display. If the pen 23 is not an electronic pen, the input means is determined by the area of contact of the pen 23 or a finger with the display.

[0034] The gesture sensor input acquisition unit 53 acquires data (gesture data) detected by the gesture sensor 16. Acquisition includes passive reception from the gesture sensor 16 and reception of gesture data as a result of the gesture sensor input acquisition unit 53 instructing the gesture sensor 16 to acquire the data. The gesture sensor input acquisition unit 53 passes the gesture data to the gesture data analysis unit 54.

[0035] The gesture data analysis unit 54 analyzes the gesture data. Based on the gesture data, the gesture data analysis unit 54 determines whether the user has performed a gesture, and if so, analyzes the operation corresponding to the gesture. The gesture data analysis unit 54 passes the operation content determined by the analysis to the gesture operation restriction unit 55. Note that, because the analysis of the operation content is completed in a short time from the input of the gesture data, it is not necessary to strictly distinguish between the gesture data and the operation content. In other words, in this embodiment, either the gesture data or the operation content may be invalidated.

[0036] Note that methods using AI or machine learning are known as methods by which the gesture data analysis unit 54 analyzes operation content from gesture data. For example, methods that learn the correspondence between gesture data and operation content using neural networks, recurrent neural networks (RNNs), long short-term memories (LSTMs), etc. are known. Machine learning is a technology that allows computers to acquire human-like learning capabilities. The computer autonomously generates algorithms necessary for judgments such as data identification from previously acquired learning data, and applies these algorithms to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods. Any learning method for machine learning is acceptable.

[0037] The gesture operation restriction unit 55 restricts operations based on the gesture data during the period when the touch sensor input acquisition unit 51 is continuously acquiring touch data. Specifically, the gesture operation restriction unit 55 compares the time when the input means type and coordinates are received with the time when the gesture operation content is received. For example, if the gesture operation restriction unit 55 receives the gesture operation content immediately after receiving the input means type and coordinates, the gesture operation content is ignored. Furthermore, the gesture operation restriction unit 55 restricts operations based on the gesture data during the period from when the touch sensor input acquisition unit 51 acquires the touch data until a predetermined time has elapsed. For example, if the gesture operation restriction unit 55 receives the gesture operation content until a predetermined time has elapsed from when the gesture operation restriction unit 55 receives the input means type and coordinates, the gesture operation content is ignored. The gesture operation restriction unit 55 passes the input means type, coordinates, and the gesture operation content that was not ignored to the data output unit 56.

[0038] The data output unit 56 outputs drawing data to the display 11 based on the touch data. "Output" refers to "display." That is, the data output unit 56 displays drawing data based on the input means type and coordinates on the display. The data output unit 56 also reflects operations based on unrestricted gesture data on the screen. "Reflecting" refers to changing the screen display, such as switching pages or superimposing a marker on the drawing data. That is, the data output unit 56 reflects the operation content by the gesture on the screen or drawing data. Note that internal processing that is not reflected on the screen may be performed by the gesture.

[0039] <Action or Processing> FIG. 4 is a flowchart showing an example of processing when the display device 20 detects an input from the gesture sensor 16.

[0040] The gesture sensor input acquisition unit 53 acquires gesture data such as camera image data, which is then analyzed by the gesture data analysis unit 54 (step S1). For example, if the camera image data has 60 frames per second, the gesture data analysis unit 54 may analyze the data one frame at a time, or periodically, such as once per second. The gesture data analysis unit 54 analyzes the gesture data to determine whether the user has detected a predetermined operation that should be detected as an operation, or whether the detected movement is simply unrelated to an operation. In step S1, the description will be given assuming that a predetermined operation that should be detected as an operation has been detected. The gesture data analysis unit 54 passes the operation content to the gesture operation restriction unit 55.

[0041] When the gesture operation restriction unit 55 acquires the operation content based on the gesture data, it determines whether touch data (input means type and coordinates) is currently being received continuously or whether a predetermined time has passed since the last time touch data (input means type and coordinates) was received (step S2). Details of the determination in step S2 will be explained with reference to FIG. 5. If the determination in step S2 is Yes, the process proceeds to step S3, and if No, the process proceeds to step S4.

[0042] If the determination in step S2 is Yes, the gesture operation restriction unit 55 processes only the touch data (input means type and coordinates) and ignores the gesture data or the operation content by the gesture data (step S3). That is, the data output unit 56 draws the drawing data input from the touch sensor 17 on the display, but does not reflect the operation content of the gesture data on the drawing data or on the screen.

[0043] If the determination in step S2 is No, the gesture operation restriction unit 55 validates and processes the operation content by the gesture (step S4). That is, the data output unit 56 reflects the operation in the drawing data or on the screen according to the operation content from the gesture data analysis unit 54.

[0044] Fig. 5 is an explanatory diagram of step S2 in Fig. 4. Fig. 5 illustrates an example of a pattern in which the display device 20 recognizes gesture data. The input timing of touch data in the upper row of Fig. 5 is Case 1, and the input timing in the lower row is Case 2.

[0045] As in case 1, the gesture operation restriction unit 55 ignores gesture data from the time when handwriting 1 occurs until the gesture invalid period (a) has elapsed.

[0046] As in case 2, if handwriting 2 occurs after handwriting 1 occurs but before the gesture invalid period (a) has elapsed, the invalid period of the gesture data is extended by the period from handwriting 1 to handwriting 2 plus the gesture invalid period (a) from handwriting 2. In other words, a gesture invalid period (a) occurs for each new handwriting.

[0047] 5, a gesture invalid period (a) is provided in which gesture data or operation contents based on the gesture data are invalidated after handwriting occurs. However, for example, if handwritten touch data is input within an extremely short period after the input of gesture data, gesture operation restriction unit 55 may invalidate the gesture data or operation contents based on the gesture data going back in time from the timing of the handwriting.

[0048] FIG. 6 is an explanatory diagram showing an example of a gesture invalid period, showing a gesture invalid period (b) in which gesture data or operation content based on the gesture data is invalidated by tracing back in time from the timing of writing. In FIG. 6, in both Case 1 and Case 2, the start point of the gesture invalid period (b) is a time tracing back from the time when writing 1 occurred. Therefore, the gesture operation limiting unit 55 restricts operations based on the gesture data by tracing back a certain time from the time when touch data was received. This prevents a user who has started writing from being unable to write due to gesture data that arrives slightly earlier.

[0049] In order to realize this processing, it is possible to add processing such as the gesture data analysis unit 54 always delaying the gesture data for a certain period of time, or the gesture operation restriction unit 55 always delaying the invalidation decision based on the operation content for a certain period of time.

[0050] Furthermore, in the case of step S3 in FIG. 4, the display device 20 may display a message indicating that the gesture operation has been disabled. FIG. 7 is an explanatory diagram showing an example of a display when a gesture operation is disabled, and shows an example of a display displayed when the display device 20 disables a gesture operation. In FIG. 7, a message 25 saying "XX gesture operation has been disabled" is displayed at the top of the display 11. The data output unit 56 displays the message 25 on the display 11 before, after, or in parallel with the processing of step S3 in FIG. 4. The message 25 is automatically erased after being displayed for a certain period of time. Therefore, the user who performed the gesture operation can understand that their operation has been disabled. In other words, it is possible to suppress anxiety that the gesture operation has not been recognized.

[0051] Note that the fact that a gesture operation has been disabled may be displayed as an icon or mark instead of message 25. Furthermore, the reason why the gesture operation has been disabled (because writing is in progress) and what gesture operation has been disabled may be included in message 25. The user who performed the gesture operation can understand why the gesture operation has been disabled and what gesture operation has been disabled.

[0052] Furthermore, the display device 20 may execute the invalidated gesture operation later. For example, suppose that a user performs a gesture operation to open a new page, but the gesture operation is invalidated because the user is currently writing. In this case, the display device 20 executes the invalidated gesture operation to open a new page when the user finishes writing. The end of writing refers to the passage of a certain period of time since the user removed the pen 23 or finger from the display 11. At this time, the display device 20 displays the invalidated pending gesture operation on the display 11 to notify the user who is writing that the gesture operation is on hold. If the user wants to execute the pending gesture operation after finishing writing, the user can leave it as is, or if the user does not want to execute the pending gesture operation after finishing writing, the user can delete the pending gesture operation.

[0053] <Major Effects> The display device 20 of this embodiment disables gesture data detected by the gesture sensor 16 or operation content based on the gesture data during the period when the touch sensor that detects the writing operation detects writing, and for a certain period thereafter. This allows the display device 20 to prevent a user from interrupting the writing with a gesture operation from a distance. In other words, if the display device 20 has two or more sensors with different detection distances, it can prevent a user other than the user who is writing from operating the display device 20 with a gesture.

[0054] [Second embodiment] In the first embodiment, all gesture operations are disabled, but in this embodiment, a display device 20 will be described that enables some gesture operations. The gesture operation limiting unit 55 of this embodiment does not disable all gesture data or operation contents based on the gesture data during a gesture disabled period, but enables some operation contents by gestures.

[0055] In this embodiment, the hardware configuration diagram of FIG. 2 and the functional block diagram of FIG. 3 described in the first embodiment are assumed to be applicable.

[0056] <Process or Action> Fig. 8 is a flowchart showing an example of processing when the display device 20 detects an input from the gesture sensor 16 in a case where some operation contents by gestures are enabled. The explanation of Fig. 8 will mainly focus on the differences from Fig. 4. In Fig. 8, step S21 has been added.

[0057] If the determination in step S2 is Yes, the gesture operation limiting unit 55 determines whether the operation content of the gesture data corresponds to a predetermined operation (step S21). If the determination in step S21 is No, the process proceeds to step S3, and if Yes, the process proceeds to step S4. Therefore, the gesture operation limiting unit 55 can make some of the gesture data valid during the period when the touch sensor input acquiring unit 51 is continuously acquiring touch data, or during the period until a predetermined time has elapsed since the touch data was acquired.

[0058] Here, the "predetermined operation content" refers to an operation that does not interfere with writing, such as highlighting, moving the cursor, etc. This can improve usability.

[0059] The determination in step S21 may be limited to the case where the touch data and the gesture data are input with a very short time difference.

[0060] <Major Effects> According to this embodiment, in addition to the effects of the first embodiment, gesture operations are possible as long as the operations do not interfere with the writing of a user who is writing on the display device 20, so that a user who is not writing can operate using gestures from a distance, thereby improving usability.

[0061] [Third embodiment] In the first embodiment, the gesture sensor 16 is used as the long-distance sensor, but in this embodiment, a display device 20 will be described that can be used in combination with another sensor such as a human detection sensor 21. This allows the display device 20 to more effectively disable gesture data or operation content based on gesture data.

[0062] <Display device equipped with a human detection sensor> 9 is an external view showing an example of a display device 20 equipped with a human detection sensor 21. Examples of the human detection sensor 21 include a camera, an infrared camera, and a motion sensor. For example, while the human detection sensor 21 is detecting a person, the display device 20 invalidates gesture data detected by the gesture sensor 16 or operation content based on the gesture data.

[0063] Fig. 10 is a block diagram showing an example of the functional configuration of the display device 20 when the display device 20 is equipped with the human detection sensor 21. The explanation of Fig. 10 will mainly focus on the differences from Fig. 3. The display device 20 in Fig. 10 has a human detection sensor input acquisition unit 57 and a human data analysis unit 58.

[0064] The person detection sensor input acquisition unit 57 acquires data (person data) detected by the person detection sensor 21. The person data is, for example, image data of an image captured in front of the display device 20. The person detection sensor input acquisition unit 57 passes the person data to the person data analysis unit 58.

[0065] The person data analysis unit 58 analyzes the person data. Based on the person data, the person data analysis unit 58 determines whether or not a person is present within a certain distance from the display device 20, and if so, passes information indicating that the person has been detected to the gesture operation restriction unit 55. Note that, as an analysis method for the person data analysis unit 58 to detect a person, AI or machine learning may be used, as in the case where the gesture data analysis unit 54 analyzes operation content from gesture data. For example, an existing function for recognizing faces from image data is known. Furthermore, whether or not a person is within the certain distance range may be determined based on the size of the person's face or body, or distance information obtained by a stereo camera or LiDAR may be used. The certain distance is a distance at which operation by gesture is expected, and may be, for example, 3 to 5 meters. This certain distance may be set by the user.

[0066] The gesture operation restriction unit 55 restricts operations based on gesture data while the human detection sensor 21, which detects people separately from the gesture sensor, detects a person within a certain distance from the display device 20. This allows the gesture operation restriction unit 55 to disable gesture operations when a person is present within the certain distance from the display device 20, and more effectively prevents the reflection of operations based on gestures that the user does not intend while writing.

[0067] <Action or Processing> Fig. 11 is a flowchart showing an example of processing when display device 20 equipped with human detection sensor 21 detects input from gesture sensor 16. The explanation of Fig. 11 will mainly focus on the differences from Fig. 4. In Fig. 11, step S31 has been added.

[0068] In step S1, the gesture sensor input acquisition unit 53 acquires gesture data and receives the operation content, and then the gesture operation restriction unit 55 determines whether or not a person is present within a certain distance from the display device 20 (step S31). That is, the person data analysis unit 58 recognizes the person from the person data (image data), and if the size of the face is equal to or larger than a threshold, the person data analysis unit 58 notifies the gesture operation restriction unit 55 that a person is present within the certain distance, so that the gesture operation restriction unit 55 determines that a person is present within the certain distance from the display device 20.

[0069] If the determination in step S31 is Yes, the process proceeds to step S3, and if No, the process proceeds to step S2. The subsequent process may be the same as in FIG.

[0070] This embodiment can be suitably combined with either the first embodiment or the second embodiment.

[0071] <Major Effects> In addition to the effects of the first embodiment, the display device 20 of this embodiment disables gesture data or operation content based on gesture data when a person is within a certain distance from the display device 20, thereby more effectively preventing the reflection of operations due to unintended gestures while the user is writing.

[0072] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0073] For example, in this embodiment, the display device 20 has the touch sensor 17 and the gesture sensor 16, but may also have a function to accept operations from a remote control. In this case, the display device invalidates gesture data detected by the gesture sensor and operations from the remote control while the touch sensor that detects handwriting is continuously detecting handwriting, and for a certain period thereafter. Alternatively, the display device may invalidate only either the gesture data or operations from the remote control. The user can select whether to invalidate the gesture data or operations from the remote control.

[0074] Furthermore, in this embodiment, a user other than the user writing is prevented from operating the display device 20 using gestures. However, there are cases in which the user writing performs gesture operations. When the user writing performs gesture operations, it is not preferable for the display device 20 to invalidate gesture data or operation content based on the gesture data. Therefore, for example, the gesture operation restriction unit 55 does not invalidate gesture data or operation content based on the gesture data by the user holding the pen 23. A method using image recognition is known as a method for detecting the user holding the pen 23. Alternatively, the gesture operation restriction unit 55 does not invalidate gesture data of a user who approaches the display device 20. The determination of proximity is based on the distance of the user writing, which is, for example, about 50 cm. In this case, even if a user other than the user writing performs gesture operations, the user writing can recognize the operation, so gesture data by users other than the user writing does not need to be invalidated.

[0075] Furthermore, in the present embodiment, gesture data or operations based on the gesture data are disabled (restricted), but the display device 20 may disable the gesture sensor 16. That is, the display device 20 may turn off the power supply of the gesture sensor 16 or stop its operation.

[0076] In addition, in this embodiment, gesture data is invalidated both during the period when touch data is continuously acquired and during the period from when the touch data is acquired until a predetermined time has elapsed, but gesture data may be invalidated only during either of these periods. For example, in the case of a user who is writing, by invalidating gesture data only during the period when touch data is continuously acquired, it is possible to prevent the gesture of the user who is writing from being detected as unintended gesture data. Alternatively, if the user who is writing and the user who performs gesture operations are separate but both want to operate the display device alternately, operability (response) can be improved by invalidating gesture data only during the period when touch data is continuously acquired.

[0077] Furthermore, the display device 20 may have a function of converting the drawing data into language information (Japanese, English, Chinese, Hindi, Spanish, French, Arabic, Russian, etc.) in real time.

[0078] The display device 20 may also have a voice input function. In this case, gesture data or operation content based on gesture data is disabled while the user is inputting voice or for a certain period of time after the last voice input.

[0079] Furthermore, if the display device 20 has a human detection sensor 21, it may detect only gestures from people within a certain distance range from the display device 20. The certain distance range may be within a certain distance from the display device 20, more than a certain distance from the display device 20, or a certain range determined by upper and lower limits from the display device 20.

[0080] In addition, in this embodiment, it is assumed that data from the touch sensor 17 and the gesture sensor 16 is processed within an application executed by the display device 20, but the display device 20 may communicate with a server, and the server may execute processing such as that shown in Figure 4.

[0081] In addition, although the display device 20 has been described as an electronic whiteboard in this embodiment, the electronic whiteboard may also be called an electronic whiteboard, an electronic information board, or the like. The display device 20 of this embodiment can be suitably applied to any information processing device having a touch panel. Examples of information processing devices equipped with a touch panel include output devices such as projectors and digital signage, HUD (Head Up Display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, network home appliances, notebook PCs, mobile phones, smartphones, tablet terminals, game consoles, PDAs (Personal Digital Assistants), digital cameras, wearable PCs, and desktop PCs.

[0082] Furthermore, in this embodiment, the coordinates of the pen 23 are detected by detecting the coordinates of the pen tip using a touch panel, but the display device 20 may also detect the coordinates of the pen tip using ultrasonic waves. The pen 23 emits ultrasonic waves along with light emission, and a sensor on the frame of the display device 20 calculates the distance based on the arrival time of the ultrasonic waves, and a camera on the frame of the display device 20 captures the light emission. The display device 20 can identify the position of the pen 23 based on the direction and distance, and a projector draws (projects) the trajectory of the pen 23.

[0083] In addition, the configuration examples shown in Fig. 3 and the like are divided according to main functions to facilitate understanding of the processing by the display device 20. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing by the display device 20 can be divided into even more processing units depending on the processing content. Furthermore, it can also be divided so that one processing unit includes even more processing.

[0084] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0085] Embodiments of the present invention provide significant improvements in computer power and functionality. These improvements allow users to utilize computers that provide more efficient and robust interaction with tables, the way information is stored and presented in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of more efficient, powerful, and robust user interfaces. Such user interfaces provide better interaction between humans and machines.

[0086] The above description is one example, and other embodiments, additions, modifications, deletions, and other changes can be made within the scope of what a person skilled in the art can conceive. Any of the following aspects is within the scope of the present invention as long as it achieves the effects of the present invention.

[0087] <Aspect> [Aspect 1] Aspect 1 is a display device that outputs drawing data based on touch data, characterized in that it has a touch sensor input acquisition unit that acquires the touch data from a touch sensor, a gesture sensor input acquisition unit that acquires gesture data from a gesture sensor, and a gesture operation restriction unit that restricts operations based on the gesture data during the period when the touch sensor input acquisition unit is continuously acquiring the touch data. [Aspect 2] Aspect 2 is characterized in that, in aspect 1, the gesture operation restriction unit restricts operations based on the gesture data for a period from when the touch sensor input acquisition unit acquires the touch data until a predetermined time has elapsed. [Aspect 3] Aspect 3 is characterized in that, in aspect 1 or aspect 2, a person detection sensor that detects a person is provided separately from the gesture sensor, and the gesture operation restriction unit restricts operations based on the gesture data during a period when the person detection sensor detects the person within a certain distance from the display device. [Aspect 4] Aspect 4 is characterized in that, in aspect 2, the gesture operation restriction unit enables a portion of the gesture data during a period in which the touch sensor input acquisition unit is continuously acquiring the touch data, or during a period from the time the touch data is received until a predetermined time has elapsed. [Aspect 5] Aspect 5 is characterized in that, in any of aspects 1 to 4, the gesture operation restriction unit restricts operations based on the gesture data by going back a certain period of time from the time the touch data was received. [Aspect 6] Aspect 6 is characterized in that, in any of aspects 1 to 5, the gesture operation restriction unit has a data output unit that, when it restricts an operation based on the gesture data, displays that the operation based on the gesture data has been disabled. [Aspect 7] A seventh aspect is the sixth aspect, wherein the data output unit reflects the operation based on the gesture data that has not been restricted in the drawing data. [Aspect 8] Aspect 8 is a method for restricting operation of a display device that outputs drawing data based on touch data, characterized in that it includes a touch sensor input acquisition step for acquiring the touch data from a touch sensor, a gesture sensor input acquisition step for acquiring gesture data from a gesture sensor, and a gesture operation restriction step for restricting operations based on the gesture data during a period in which the touch sensor input acquisition step is continuously acquiring the touch data. [Aspect 9] Aspect 9 is a program to be executed by a display device that outputs drawing data based on touch data, and is characterized by including a touch sensor input acquisition process that acquires the touch data from a touch sensor, a gesture sensor input acquisition process that acquires gesture data from a gesture sensor, and a gesture operation restriction process that restricts operations based on the gesture data during the period when the touch sensor input acquisition process is continuously acquiring the touch data. [Explanation of symbols]

[0088] 16 Gesture Sensor 17 Touch Sensor 20 Display device 21 Person detection sensor [Prior art documents] [Patent documents]

[0089] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-110522

Claims

1. A display device that outputs drawing data based on touch data, a touch sensor input acquisition unit that acquires the touch data from the touch sensor; a gesture sensor input acquisition unit that acquires gesture data from a gesture sensor; a gesture operation limiting unit that limits an operation based on the gesture data during a period in which the touch sensor input acquiring unit continuously acquires the touch data; A display device comprising:

2. 2. The display device according to claim 1, wherein the gesture operation limiting unit limits operations based on the gesture data for a period from when the touch sensor input acquiring unit acquires the touch data until a predetermined time has elapsed.

3. a person detection sensor that detects a person separately from the gesture sensor; A period during which the person detection sensor detects the person within a certain distance from the display device, 3. The display device according to claim 1, wherein the gesture operation limiting unit limits operations based on the gesture data.

4. 3. The display device according to claim 2, wherein the gesture operation limiting unit enables a portion of the gesture data during a period in which the touch sensor input acquiring unit continuously acquires the touch data, or during a period from when the touch data is received until a predetermined time has elapsed.

5. The display device according to claim 1 , wherein the gesture operation restriction unit restricts operations based on the gesture data going back a certain time from when the touch data was received.

6. When the gesture operation limiting unit limits the operation based on the gesture data, 2. The display device according to claim 1, further comprising a data output unit that displays a message that the operation based on the gesture data has been invalidated.

7. 7. The display device according to claim 6, wherein the data output unit reflects the operation based on the gesture data that has not been restricted in the drawing data.

8. 1. A method for restricting operation of a display device that outputs drawing data based on touch data, comprising: a touch sensor input acquisition step of acquiring the touch data from the touch sensor; a gesture sensor input acquisition step of acquiring gesture data from the gesture sensor; a gesture operation limiting step of limiting an operation based on the gesture data during a period in which the touch sensor input acquiring step continuously acquires the touch data; An operation restriction method comprising:

9. A program executed by a display device that outputs drawing data based on touch data, a touch sensor input acquisition process for acquiring the touch data from the touch sensor; a gesture sensor input acquisition process for acquiring gesture data from the gesture sensor; a gesture operation limiting process that limits an operation based on the gesture data during a period in which the touch sensor input obtaining process is continuously obtaining the touch data; A program comprising:

Images