Viewpoint position indicator mark position maintaining device

Abstract

To provide a position holding device of a viewpoint position display mark that enables a viewer to keep on viewing wastelessly upon waking up by showing a viewpoint position, which is the position where the viewer was looking before sleeping with a viewpoint position display mark such as a cursor and the like so as to hold the viewpoint position when the viewer wakes up from a sleep.SOLUTION: In a position holding device 100 of a viewpoint position display mark, viewpoint position mark holding processing means 150 comprises: eyelid detection means 151 that detects an eyelid of a viewer from an image of a face from a camera 110 photographing the face of the viewer; eyelid opening / closing determination means 152 that determines an opening or closing of the detected eyelid of the viewer, and determines a closing time or a reciprocation time from the opening to the closing, and from the closing to the opening; and by-condition processing means 153 that performs processing of recovering and maintaining a viewpoint based on a result of the determination to display the viewpoint.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a position maintaining device for a viewpoint position display mark that maintains and recovers the viewpoint on a display screen that has become unclear due to drowsiness, using a viewpoint position display mark such as a cursor. [Background technology]

[0002] The prior arts shown in Patent Documents 1 to 14 all detect the viewpoint of a viewer looking at a display screen, place a cursor at the viewpoint position on the display screen, and move the cursor according to the movement of the viewpoint. In the early days of the technology, the cameras used for detection were expensive and had a large volume, so it seems that they were no longer used with the advancement of new technologies such as touch sensors. Due to this influence, even if the cursor is moved by a touch sensor, no device has been seen that moves the cursor according to the viewpoint. In such a background, when looking at a display such as text, drawings, or tables on a display screen, it is often the case that it is unclear where you were looking when you realize it, and as a result, you may have to reread it many times or waste a lot of time. This often happens when you are looking at difficult content or are tired. That is, you lose your memory due to falling asleep for a moment (starting to doze off), and even when you wake up, you do not know the original position of your viewpoint. No prior art was found that addresses this issue and improves browsing efficiency. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication 2018-73147 [Patent Document 2] Patent Publication No. 2018-13926 [Patent Document 3] Patent Publication 2015-118531 [Patent Document 4] Patent Publication No. 2014-235729 [Patent Document 5] Patent Publication 2014-6640 [Patent Document 6] Patent Publication No. 2013-255781 [Patent Document 7] Special table 2014-514661 [Patent Document 8] Special table 2013-506209 [Patent Document 9] Patent Publication No. 2000-187552 [Patent Document 10] Patent Publication H10-31549 [Patent Document 11] Patent H4-350715 [Patent Document 12] Patent H4-205617 [Patent Document 13] Patent H2-216520 [Patent Document 14] Patent application S62-196723 Summary of the Invention [Problem to be solved by the invention]

[0004] To provide a position maintaining device for a viewpoint position display mark, which indicates with a viewpoint position display mark such as a cursor that the viewpoint position before falling asleep is maintained when the viewer wakes up from falling asleep, thereby enabling the viewer to continue viewing efficiently when awake. [Means for solving the problem]

[0005] The following is described in accordance with the claims. The invention described in claim 1 is a position maintaining device for a viewpoint position indicating mark, A control means and a control device controlled by the control means. a camera which photographs the face of a viewer; viewpoint or gaze detection means which detects the position of the viewpoint or gaze on a display surface which the viewer is looking at from the photographed image of the face; viewpoint position indication mark position data conversion means which converts a detection result of the viewpoint or gaze detection means into position data of a viewpoint position indication mark; a viewpoint position indication mark position data memory which stores the position data of the viewpoint position indication mark; viewpoint position mark maintenance processing means; a display means; a display data memory which stores display data to be displayed on the display means; and data synthesis means which arranges the display data in the display data memory on the display surface of the display means and arranges the viewpoint position indication mark on the display surface of the display means based on the position data of the viewpoint position indication mark, The viewpoint position mark maintaining processing means has a condition-specific processing means for determining whether the viewer has fallen asleep or not from the facial image captured by the camera, and when it is determined that the viewer has not fallen asleep, the means performs processing for arranging the viewpoint position display mark so as to follow the position of the viewer's viewpoint or the movement of the line of sight, and when it is determined that the viewer has fallen asleep, the means performs processing for displaying the position of the viewpoint position display mark at the time before the viewer fell asleep, so as to show the viewer who is awake, The viewpoint position mark maintaining processing means has an eyelid detecting means for detecting eyelids from the image of the camera and an eyelid opening / closing determining means for determining whether the eyelids are open or closed, and causes the condition-based processing means to perform a discrimination process using the previous and current conditions of whether the eyelids are open or closed, or / and has a face detecting means for detecting a face from the image of the camera and a face movement determining means for determining the movement of the face, and causes the condition-based processing means to perform a discrimination process using the condition of the movement of the face, By updating and storing and maintaining position data of the viewpoint position display mark at a plurality of time points, when it is determined that the viewer has fallen asleep based on the previous and current conditions of whether the eyelids are open or closed, the position of the viewpoint position display mark at the time point before the viewer fell asleep is displayed to the awakened viewer based on the stored and maintained position data; When the position at the time point before falling asleep is shown to the awakened viewer, a trajectory from a position where the viewpoint position indication mark is removed to the position at the time point before falling asleep is displayed.

[0006] The invention described in claim 2 is 1 In the position maintaining device for the viewpoint position indicating mark described above, The condition-based processing means is characterized in that, in determining that the subject is asleep, the time threshold for the eyelids to be closed is set to 0.3 seconds or more.

[0007] The invention described in claim 3 is 1 In the position maintaining device for the viewpoint position indicating mark described above, The condition-based processing means determines whether the subject is asleep based on a condition of whether the eyelids are open or closed, This is characterized by the fact that it is performed using a threshold value set for the degree of black and white in the image of the eye area captured by the camera. Effect of the Invention

[0008] As configured as described above, the position maintaining device for the viewpoint position indication mark according to the present invention, when the viewer wakes up from falling asleep, indicates with a viewpoint position indication mark such as a cursor that the viewpoint position that was there before falling asleep is maintained, thereby enabling the viewer to continue viewing efficiently upon awakening. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing an embodiment of the configuration of a position maintaining device for a viewpoint position indicating mark according to the present invention; [Diagram 2] FIG. 2 is a diagram showing an embodiment of an operation flow of the position maintaining device for a viewpoint position indicating mark according to the present invention. [Diagram 3] 1 is a diagram for explaining an embodiment of the operation of a condition-specific processing means of the position maintaining device for a viewpoint position indicating mark according to the present invention; [Figure 4] 1A to 1C are diagrams illustrating the movement of a viewpoint position indication mark on a display surface due to the operation of a position maintaining device for a viewpoint position indication mark according to the present invention. [Diagram 5] 13 is a diagram showing another embodiment in which a part of the configuration of the position maintaining device for the viewpoint position indicating mark according to the present invention is modified. FIG. [Figure 6]7 is a diagram showing another embodiment corresponding to the example of FIG. 5 in the operational flow of the viewpoint position indicating mark position maintaining device according to the present invention. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1 is a diagram showing one embodiment of the configuration of a position maintaining device for a viewpoint position indication mark according to the present invention. The position maintaining device for a viewpoint position indication mark 100 includes a camera 110 for photographing the face of a viewer, a viewpoint or line of sight detection means 120 for detecting the position or line of sight on the display surface where the viewer's eyes are looking from the image of the face, a position data conversion means 130 for converting the detection result of the viewpoint or line of sight detection means 120 into position data for a viewpoint position indication mark, a position data memory 140 for storing the position data for the viewpoint position indication mark, a viewpoint position mark maintaining processing means 150, a display means 160, a display data memory 170 for storing display contents to be displayed on the display means 160, a data synthesis means 180 for arranging data in the display data memory on the display surface of the display means 160 and arranging a viewpoint position indication mark on the display surface of the display means 160 based on the position data for the viewpoint position indication mark, and a control means 190 for controlling these.

[0011] Here, many examples of the eye position or line of sight detection means 120 and the viewpoint position indication mark position data conversion means 130 can be found in Patent Documents 1 to 14, and since prior art can be used, detailed explanation will be omitted, but as one example, it is possible to configure the means by temporarily displaying viewpoint position indication marks at the center, both ends, and the top and bottom ends of the display screen as a preparation operation, having the viewer look at them, and determining the position of the viewpoint position indication mark for each actual facial position based on the positions of slight movements of facial feature points (eyes, mouth, nose, eyebrows, face itself, etc.) for each position photographed by camera 110. Advantageously, the position of the viewpoint position indication mark can be maintained by knowing what area the viewer was looking at as a feature, so slight positional errors do not pose a problem.

[0012] The viewpoint position mark maintenance processing means 150 comprises eyelid detection means 151 which detects the eyelids of the viewer from the image of the face taken by the camera 110, eyelid opening / closing judgment means 152 which judges whether the viewer's eyelids detected by the eyelid detection means 151 are open or closed and judges the closing time or the round trip time from open to closed to open, and has condition-specific processing means 153 perform judgment processing of the position of the viewpoint position indication mark using the judgment result of the eyelid opening / closing judgment means 152 and the conditions of the previous judgment result. Furthermore, the viewpoint position mark maintenance processing means 150 is equipped with face detection means 155 for detecting the viewer's face and face movement determination means 156, and as a result, uses the facial movement conditions to cause the condition-specific processing means 153 to perform processing for determining the position of the viewpoint position indication mark. The operation of the condition-specific processing means 153 will be described later together with the conditions. In the example of FIG. 1, the eyelid opening / closing determination data memory 154 is provided. Since data on whether the eyelids are "open" or "closed", which is the judgment result of the eyelid opening / closing judgment means 152, is stored, only one eyelid opening / closing judgment means 152 (or one process) is sufficient, and the stored data can be either "1" or "0" corresponding to opening or closing, so it is convenient to use the previous opening / closing data. Alternatively, FIG. 6, which will be described later, is also possible.

[0013] 3 is a diagram for explaining one embodiment of the operation of the condition-specific processing means 153 of the position maintaining device for the viewpoint position indicating mark according to the present invention. In 3-A, the condition-specific processing is performed only based on the opening and closing of the eyelids, and in 3-B, the condition-specific processing is performed based on the opening and closing of the eyelids and the movement of the face. Although not shown in the figure, it is of course also possible to carry out condition-specific processing based only on facial movements. In both cases, eyelid closure for less than 0.3 seconds is treated as a blink and the person is not asleep. Therefore, it is preferable to set the eyelid closure time threshold for determining whether the person is asleep to 0.3 seconds or more. Naturally, the camera takes pictures at appropriate time intervals (0.1 seconds, 1 second, etc.) or constantly (continuously) in consideration of the time threshold to determine whether the eyelids are open or closed. Instead of setting a time threshold, the following can be used. For example, the eyelid opening / closing can be determined by the degree of black and white (light and dark) due to the opening and closing of the eyelids in the eye area in the camera image. When the camera shutter is opened for an appropriate time interval and an image is taken, in the eye area, the iris is dominant and black prevails in the eyelid opening condition, and in the eyelid closing condition, the iris is hidden by the eyelid, so white prevails. As a result, the degree of black and white is greater in the case of falling asleep than in the case of simply blinking. By utilizing this, it is also possible to set a threshold (called a first black and white degree threshold) for the degree of black and white in the image of the eye area in the camera image, rather than a simple time threshold, and the same effect as the time threshold can be achieved. Similarly, if the shutter speed of a camera that takes pictures at appropriate time intervals is set faster than the blinking speed (e.g., 1 / 100 seconds), the blinking movement will be slower and the resolution will be insufficient (it will not be captured), but in the closed eyelid state of sleep, the closed state will provide sufficient resolution, so it is possible to set a threshold value (called a second black and white degree threshold value) different from the above for the black and white degree. Note that the threshold value set for the black and white degree described in the claims refers to either the first black and white degree threshold value or the second black and white degree threshold value. In 3-A, the right condition-specific processing is performed according to 1) to 4) based on the current and previous eyelid opening and closing (including time). For example, in 2), if the previous eyelid was open and the current eyelid is closed for 5 seconds, it is determined to be a sleep-initiated behavior that is not a blink (less than 0.3 seconds), and the position of the viewpoint position display mark is saved. This is to be used later to restore or maintain the position if the person wakes up. In 3-B, A. characteristic differences between when the gaze is shifted due to sleep onset (drowsiness) and when it is shifted intentionally are summarized. In addition, various research results are referred to for the characteristics of instantaneous sleep and onset of sleep. As a result, B shows the judgment conditions and processing by condition. It is natural that the judgment and processing of 3-B, which includes eyelid opening and closing as well as facial movement, can be handled more precisely than the judgment and processing of 3-A, which only involves the opening and closing of the eyelids. (2) The description of not determining that eyelid opening is a sleep behavior, except for the case where the face is suddenly dropped with a "jerk," the description of when the gaze is shifted by accidentally looking to the side, and the description of (3) are conditions that take facial movement into consideration. It is also possible to only require the face to suddenly drop with a "jerk" without including the opening and closing of the eyelids as a condition, but this would be less accurate since there are many cases where the face does not drop when falling asleep.

[0014] 2 is a diagram showing an embodiment of the operation flow of the position maintaining device for the viewpoint position indicating mark according to the present invention. #1 Based on a photograph of the face taken with a camera, #2a eyelid detection, #3a eyelid opening / closing judgment, #4a eyelid opening / closing judgment data storage, Similarly, based on #1 a photograph of the face taken with a camera, #2b detects the viewpoint and #3b converts the position of the viewpoint position display mark, #4b stores the position data of the viewpoint position display mark, When facial movement is taken into consideration, similarly, #2c face detection and #3c facial movement judgment are performed based on #1 face camera photography, Depending on the results, #5 condition-specific processing (position display and maintenance processing of the viewpoint position display mark by condition judgment) is performed. This flow is repeated at appropriate intervals.

[0015] FIG. 4 is a diagram for explaining the movement of the viewpoint position display mark on the display surface by the operation of the viewpoint position display mark position maintaining device according to the present invention. 4-A, 4-B, and 4-C are examples of horizontally written text, and 4-A corresponds to the case where the technology of the present application is not applied. The viewpoint position display mark surrounded by a circle is shown with a dotted line, but in reality nothing is displayed. The cursor is located regardless of the viewpoint. All other than 4-A correspond to the case where the present application is applied. 4-B is an example of the case where, as shown by the viewpoint position display mark in normal times, when looking along a line, the line finally moves down. If the viewer intentionally moves down to look down, the viewer is conscious, so he or she can move the viewpoint anywhere of his or her own will and return. In addition, the viewpoint position display mark also returns to anywhere corresponding to the viewpoint by moving the viewpoint. However, if the deviation from the normal line is due to unintentional sleep, the patient is unconscious and, as such, traditionally it is not possible to know where to return to. It is wasteful to read here and there without knowing where you were looking. In order to easily return to such a case, the technology of the present application shows the return destination of the gaze point (including the transit point, which makes it easy to recognize). In particular, in the case of an intentional movement of the gaze point and an unintentional movement due to falling asleep, the former may be slower than the latter, which moves suddenly and "jerks", and can be self-restored, so no special measures are required. However, since the latter cannot be self-restored, it is useful to show the position of the gaze point position display mark in normal times stored in the position data memory 140 of the gaze point position display mark and provide it to the viewer. It is also possible to reserve and maintain the position of the gaze point position display mark in normal times as in 4-C, and not display the movement of the gaze point position display mark when falling asleep. Furthermore, if there is no movement of the face, the position of the gaze point position display mark is maintained as in 4-C, so that self-restore is possible without any problems.

[0016] 4-D and 4-E are examples of vertically written text. 4-D corresponds to 4-B, and 4-E corresponds to 4-C. In 4-D and 4-E, The difference with horizontal writing is that the fulcrum moves vertically under normal circumstances, but the sudden, rapid downward movement that occurs when falling asleep is the same, and other explanations are the same, so we will not go into detail here. In order to maintain the position of the viewpoint position indication mark, it is preferable to constantly store and maintain the position data of the viewpoint position indication mark at multiple points in time while updating it, so that it can be restored at any time if the user falls asleep. Furthermore, the viewpoint position display mark may be a so-called cursor for performing some operation on the display surface, or it may be something separate from the cursor since its purpose is to display the viewpoint. Figure 4 shows a case where it is separate.

[0017] Fig. 5 is a diagram showing another embodiment in which a part of the configuration of the position maintaining device of the viewpoint position indication mark according to the present invention is changed. In the configuration of the viewpoint position mark maintaining processing means 150, 5-A is the same as the configuration shown in Fig. 1, and 5-B is a modified version of that part. Instead of the eyelid open / close judgment data memory 154, there is an eyelid data memory 154b, which stores the results of the eyelid detection means 151 as they are. Comparing the two, 5-A has the advantage that 5-B performs the same processing as the eyelid open / close judgment means 152 on the previous data as in the previous eyelid open / close judgment means 152, and the data stored in the memory is a value of 1 or 0 in 5-A, while in 5-B it is eyelid data, so the amount of data is slightly larger.

[0018] Fig. 6 is a diagram showing another embodiment of the operation flow of the position maintaining device for the viewpoint position indication mark according to the present invention, which corresponds to the example of Fig. 5. The difference from Fig. 2 is that after eyelid detection in #2a, there is storage of eyelid data in #3d, and eyelid opening / closing judgment / previous / current in #4d.

[0019] Furthermore, when a viewer leaves his / her seat for some reason, the position maintaining device for the viewpoint position display mark of the present invention maintains the viewpoint, closes the eyelids for a period of time corresponding to falling asleep (allowing the device to virtually determine that the viewer has fallen asleep), and then removes his / her eyes, leaves his / her seat, and when the viewer returns, the viewpoint position before leaving the seat is restored, so that it can be expected to be effective in situations other than when falling asleep. Furthermore, like a personal computer (PC) or tablet PC, if a smartphone is provided with a camera on the display surface side or a means for inverting a camera facing the back side of the display surface, this camera can be used as camera 110. [Industrial Applicability]

[0020] As described above, the position maintaining device for a viewpoint position indication mark of the present invention is extremely advantageous for industrial use because, when waking up from falling asleep, the viewpoint position indicating mark, such as a cursor, is used to maintain the viewpoint position that was there before the viewer fell asleep, allowing the viewer to continue viewing efficiently upon waking up.

[0021] 100 Position maintaining device for indicating viewpoint position 110 Camera 120 Point of view or line of sight detection means 130 Position data conversion means for viewing position indication mark 140 Position data memory for viewpoint position indication mark 150 Viewpoint position mark maintenance processing means 151 Eyelid detection means 152 Eyelid opening / closing determination means 152b Last eyelid opening / closing determination means 153 Condition-specific processing methods 154 Eyelid opening / closing judgment data memory 154b Eyelid Data Memory 155 Face detection means 156 Facial movement judgment means 160 Display means 170 Display Data Memory 180 Data synthesis methods 190 Control Means

Claims

1. A control means and a control device controlled by the control means. a camera which photographs the face of a viewer; viewpoint or gaze detection means which detects the position of the viewpoint or gaze on a display surface which the viewer is looking at from the photographed image of the face; viewpoint position indication mark position data conversion means which converts a detection result of the viewpoint or gaze detection means into position data of a viewpoint position indication mark; a viewpoint position indication mark position data memory which stores the position data of the viewpoint position indication mark; viewpoint position mark maintenance processing means; a display means; a display data memory which stores display data to be displayed on the display means; and data synthesis means which arranges the display data in the display data memory on the display surface of the display means and arranges the viewpoint position indication mark on the display surface of the display means based on the position data of the viewpoint position indication mark, The viewpoint position mark maintaining processing means has a condition-specific processing means for determining whether the viewer has fallen asleep or not from the facial image captured by the camera, and when it is determined that the viewer has not fallen asleep, the means performs processing for arranging the viewpoint position display mark so as to follow the position of the viewer's viewpoint or the movement of the line of sight, and when it is determined that the viewer has fallen asleep, the means performs processing for displaying the position of the viewpoint position display mark at the time before the viewer fell asleep, so as to show the viewer who is awake, The viewpoint position mark maintaining processing means has an eyelid detecting means for detecting eyelids from the image of the camera and an eyelid opening / closing judging means for judging whether the eyelids are open or closed, and causes the condition-based processing means to perform a discrimination process using the previous and current conditions of whether the eyelids are open or closed, or / and has a face detecting means for detecting a face from the image of the camera and a face movement judging means for judging the movement of the face, and causes the condition-based processing means to perform a discrimination process using the condition of the movement of the face, By updating and storing and maintaining position data of the viewpoint position display mark at a plurality of time points, when it is determined that the viewer has fallen asleep based on the previous and current conditions of whether the eyelids are open or closed, the position of the viewpoint position display mark at the time point before the viewer fell asleep is displayed to the awakened viewer based on the stored and maintained position data; A position maintaining device for a viewpoint position indication mark, characterized in that, when showing the position at the time before falling asleep to the awake viewer, a trajectory from the position where the viewpoint position indication mark was removed to the position at the time before falling asleep is displayed.

2. 2. The apparatus for maintaining the position of a viewpoint position indicating mark according to claim 1, wherein said condition-based processing means sets the eyelid closure time threshold value for determining that the subject is asleep to 0.3 seconds or more.

3. The condition-based processing means determines whether the subject is asleep based on a condition of whether the eyelids are open or closed, 2. The apparatus for maintaining the position of the viewpoint position indicating mark according to claim 1, wherein said step is carried out by using a threshold value set for the degree of black and white in an image of the eye area captured by a camera.

Images