Information processing device, information processing method, and program

Abstract

An information processing device, an information processing method, and a program are provided. [Solution] In an information processing device, a processing unit 200 includes a viewing direction information acquisition unit that acquires information indicating the viewing direction of a person obtained from an image of the person captured by an imaging device that captures the surroundings of one or more display devices, a display position information acquisition unit that acquires information indicating a display position, which is a position at which one or more display devices displayed a specific object that can be gazed upon by the person at a timing corresponding to the timing at which the image was captured, and an estimation unit that estimates at least one of the position and orientation of one or more display devices using the information indicating the viewing direction and the information indicating the display position.

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses an information display device that automatically sets itself to prevent others from peeking when it is determined that there is a risk of others peeking. Patent Document 2 discloses an electronic device that performs anti-peeking control when it detects the line of sight of a person other than the user. Patent Document 3 discloses a technology that acquires line-of-sight information of a user's gaze on a screen image for each user and displays which display area in the screen image the user has gazed on. (Prior art document) (Patent Documents) (Patent Document 1) JP 2012-129701 A (Patent Document 2) Patent No. 6474921 (Patent Document 3) Patent No. 6485522 Summary of the Invention

[0003] In a first aspect of the present invention, there is provided an information processing device. The information processing device includes a viewing direction information acquisition unit that acquires information indicating a viewing direction of a person acquired from an image of the person captured by an imaging device that captures an image of the surroundings of one or more display devices. The information processing device includes a display position information acquisition unit that acquires information indicating a display position, which is a position at which the one or more display devices display a specific object that can be gazed upon by the person at a timing corresponding to a timing at which the image is captured. The information processing device includes an estimation unit that estimates at least one of a position and an orientation of the one or more display devices using the information indicating the viewing direction and the information indicating the display position.

[0004] The information processing device may include a determination unit that determines whether the specific object is being gazed at by a person captured in the image based on a change over time in the viewing direction and a change over time in the display position. The information processing device may include a storage unit that stores information indicating the viewing direction and the display position at a plurality of times at which the determination unit determines that the specific object is being gazed at by the person captured in the image. The estimation unit may estimate at least one of a position and an orientation of the one or more display devices using the information indicating the viewing direction and the information indicating the display position at the plurality of times stored in the storage unit.

[0005] In any of the information processing devices described above, the specific object may be a cursor displayed on the one or more display devices.

[0006] In any of the above-mentioned information processing devices, the judgment unit may determine that the specific object is being gazed upon by a person captured in the image when the amount of change over time in the viewing direction is smaller than a predetermined value and the amount of change over time in the display position is smaller than a predetermined value.

[0007] In any of the information processing devices described above, the estimation unit may estimate at least one of a position and an orientation of the one or more display devices by further using a positional relationship between the one or more display devices and the imaging device, and geometric attribute information of the display area of ​​the one or more display devices.

[0008] In any of the above information processing devices, the estimation unit may assume a plurality of combinations of positions and orientations of the one or more display devices, calculate for each of the plurality of combinations a distance between a position of an intersection between the viewing direction and the one or more display devices in three-dimensional space and a display position of the specific object, and weight the position and orientation of a combination for which the calculated distance is shorter with a greater weight and use the weighted value for estimating the position and orientation of the one or more display devices.

[0009] In a second aspect of the present invention, there is provided an information processing method. The information processing method includes a step of acquiring information indicating a viewing direction of a person acquired from an image of the person captured by an imaging device that captures an image of the periphery of one or more display devices. The information processing method includes a step of acquiring information indicating a display position, which is a position at which the one or more display devices displayed a specific object that may be gazed upon by the person at a timing corresponding to a timing at which the image was captured. The information processing method includes a step of estimating at least one of a position and an orientation of the one or more display devices using the information indicating the viewing direction and the information indicating the display position.

[0010] In a third aspect of the present invention, there is provided a program which, when executed by a computer, causes the computer to function as any one of the information processing devices described above.

[0011] The above summary of the invention does not list all of the features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]

[0012] [Figure 1] 1 illustrates an example of how an information processing system 100 is used in accordance with an embodiment. [Diagram 2] 2 shows a schematic diagram of functional blocks of an information processing device 30. [Diagram 3]A situation is shown in which a person 80a and a person 80b are gazing at a pointer 300 displayed on the display device 20a. [Figure 4] 2 shows an example of a data structure of gaze information data stored in the storage unit 280. [Diagram 5] 13 is an example of a flowchart relating to an information processing method executed in the information processing device 30. [Figure 6] FIG. 13 is a diagram for roughly explaining an example of a technique for estimating the number of display devices 20. [Figure 7] 2 shows a schematic diagram of an example of initial position and orientation values ​​for the display device 20. [Figure 8] 2 shows a schematic diagram of an example of initial position and orientation values ​​for the display device 20. [Figure 9] 2 shows a schematic diagram of an example of initial position and orientation values ​​for the display device 20. [Figure 10] 10A and 10B show schematic examples of setting multiple combinations of positions and orientations of the display device 20. [Figure 11] FIG. 11 is a diagram for explaining a calculation process of information used to calculate an evaluation value. [Figure 12] FIG. 11 is a diagram for explaining a calculation process of information used to calculate an evaluation value. [Figure 13] 13 shows an example of a clustering result of a combination of positions and orientations. [Figure 14] 13 shows an example of a clustering result of a combination of positions and orientations. [Figure 15] 13A and 13B show schematic diagrams of an example of concealment processing by the display control unit 250. [Figure 16] An example of a computer 2000 is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0014] 1 is a schematic diagram showing a usage state of an information processing system 100 according to an embodiment. The information processing system 100 includes an information processing device 30, a display device 20b, and a display device 20c. The information processing device 30 includes a display device 20a and an imaging device 40.

[0015] The information processing device 30 is, for example, a user terminal such as a personal computer or a mobile terminal. Examples of the mobile terminal include a mobile phone, a smartphone, a PDA (registered trademark), a tablet, a notebook computer or laptop computer, and a wearable computer. Person 80a is a user of the information processing system 100. Person 80b is a different user from person 80a. Person 80a and person 80b may be collectively referred to as "person 80".

[0016] The imaging device 40 captures an image of the surroundings of the display device 20a. The imaging device 40 captures, for example, an image of at least a space on the display area 10a side of the display device 20a. The imaging device 40 is fixed to the information processing device 30. The imaging device 40 may have a function of capturing an image using visible light. The imaging device 40 may have a function of capturing an image using infrared light.

[0017] The display device 20b and the display device 20c are display devices separate from the information processing device 30. For example, the display device 20b and the display device 20c are display devices external to the information processing device 30. The display device 20b has a display area 10b, and the display device 20c has a display area 10c. The display areas 10a, 10b, and 10c may be collectively referred to as "display area 10." In the embodiment of the information processing system 100, the display device 20a is a display device included in the information processing device 30 itself, whereas the display device 20b and the display device 20c are display devices external to the information processing device 30.

[0018] The information processing device 30 presents information to the person 80 by displaying the information on the display devices 20a, 20b, and 20c. In this embodiment, the display device 20b is disposed to the right of the display device 20a as viewed from the person 80, and the display device 20c is disposed above the display device 20a as viewed from the person 80. In the embodiment of the information processing system 100, the display devices 20a, 20b, and 20c may be collectively referred to as "display devices 20." Examples of the display devices 20a, 20b, and 20c include displays, projectors, and televisions. The display devices 20a, 20b, and 20c may be display devices that display images that are recognized as three-dimensional images.

[0019] The information processing device 30 performs a process for suppressing information displayed on the display devices 20a, 20b, and 20c from leaking to the outside. As an example, the information processing device 30 determines whether or not there is a person other than the person 80a registered as a user of the information processing device 30 from an image captured by the imaging device 40, and when there is a person other than the person 80a, conceals the information displayed on the display devices 20a, 20b, and 20c, thereby suppressing information from leaking to the outside. Note that the information processing device 30 may conceal the information displayed on the display devices 20a, 20b, and 20c on the condition that an administrator of the information processing system 100 or the person 80a has instructed the information to be concealed.

[0020] The concealment process is, for example, a process of converting an image into another image, a process of superimposing an image into another image, etc. Examples of the process of converting an image into another image include a mosaic process, a blurring process, a mixing process, etc. The process of superimposing an image into another image includes a graying process.

[0021] Since the display device 20b and the display device 20c are separate display devices from the information processing device 30, the display device 20b and the display device 20c can be arranged freely. Depending on the positions and / or orientations of the display device 20b and the display device 20c, there may be cases where the imaging device 40 cannot capture an image of a position where the display device 20b and the display device 20c are visible. Therefore, depending on the positions and / or orientations of the display device 20b and the display device 20c, there may be cases where the imaging device 40 cannot monitor whether or not there are people other than the person 80a.

[0022] The information processing device 30 identifies the gaze direction of the person 80 from the image captured by the imaging device 40, and identifies the display position of a specific object that is likely to be viewed by the person 80 among the objects displayed on the display device 20. The information processing device 30 estimates at least one of the position and orientation of the display device 20 based on the gaze direction of the person 80 and the display position of the specific object. The information processing device 30 determines whether or not to perform a concealment process on at least any area in the display region 10 based on at least one of the estimated position and orientation of the display device 20. For example, when it is determined that the imaging device 40 cannot monitor a position where the display device 20b can be viewed, the information processing device 30 conceals information displayed on the display device 20b.

[0023] In the description of this embodiment, the position, orientation, etc. of the display device 20 may be described using a three-dimensional coordinate system. In this embodiment, the z-axis direction of the coordinate system is parallel to the imaging direction of the imaging device 40. The imaging direction of the imaging device 40 is substantially the same as the normal direction of the display area 10a of the display device 20a. The x-axis direction of the coordinate system is the same as the horizontal direction of the display area 10a, and the y-axis direction of the coordinate system is the same as the vertical direction of the display area 10a.

[0024] 2 shows a schematic functional block diagram of the information processing device 30. The information processing device 30 includes a processing unit 200, an imaging device 40, and a display device 20a. The processing unit 200 has a function of controlling the display of the display device 20a, as well as a function of controlling the display of the display devices 20b and 20c connected to the information processing device 30.

[0025] The processing unit 200 includes a display position information acquisition unit 210 , a viewing direction information acquisition unit 220 , a determination unit 230 , an estimation unit 240 , a display control unit 250 , and a storage unit 280 .

[0026] The viewing direction information acquisition unit 220 acquires information indicating the viewing direction of the person 80, which is acquired from an image of the person 80 captured by the imaging device 40 that captures the surroundings of the display device 20. The viewing direction information acquisition unit 220 may acquire the viewing direction of the person 80 based on an image of the face of the person 80. For example, the viewing direction information acquisition unit 220 may identify the direction of the face or head of the person 80 based on an image of the head of the person 80 captured by the imaging device 40, and acquire the identified direction as the viewing direction of the person 80. The viewing direction information acquisition unit 220 may identify the position of the iris of the person 80 from the image of the face of the person 80 captured by the imaging device 40, and identify the viewing direction of the person 80 based on the identified position of the iris. The viewing direction information acquisition unit 220 may identify the position of the iris of the person 80 from the image of the face of the person 80 captured by the imaging device 40, and identify the viewing direction of the person 80 based on the identified position of the iris and the direction of the head of the person 80. The viewing direction information acquisition section 220 may continuously input images captured by the imaging device 40, and continuously acquire the viewing direction of the person 80 based on the continuously input images.

[0027] The display position information acquisition unit 210 acquires information indicating a display position, which is a position at which the display device 20 displays a specific object that may be gazed upon by the person 80 at a timing corresponding to a timing at which an image is captured by the imaging device 40. For example, the specific object may be a pointer displayed on the display device 20. The display position information acquisition unit 210 may acquire the display position of the pointer through an operating system. The estimation unit 240 estimates at least one of the position and orientation of one or more display devices using the information indicating the viewing direction and the information indicating the display position.

[0028] The specific object may be the above-mentioned pointer (e.g., a mouse pointer, etc.), as well as an object that is displayed by flashing, an object that is displayed in a highlighted manner, an object that is displayed in an enlarged manner, an object that is displayed in a pop-up manner, a portion of a moving image display where the fluctuation in the display content is greater than a reference value, a display object for guiding the user (e.g., a display object for giving instructions to the user such as "Look at the right edge of the display"), etc.

[0029] Specifically, the determination unit 230 determines whether or not a specific object is being gazed at by a person captured in an image, based on a time change in the viewing direction and a time change in the display position. The storage unit 280 stores information indicating the viewing direction and the display position at multiple times when the determination unit 230 determines that the specific object is being gazed at by a person captured in an image. The estimation unit 240 estimates at least one of the position and the orientation of the display device 20, using the information indicating the viewing direction and the information indicating the display position at multiple times stored in the storage unit 280.

[0030] When the display device 20 is a display device that displays a two-dimensional image, the "information indicating the display position at which the display device 20 displays a specific object that can be gazed at by the person 80" acquired by the display position information acquisition unit 210 may be information indicating the position on the display surface of the display device 20 at which the display device 20 displays the specific object. The estimation unit 240 may estimate at least one of the position and the orientation of the display device 20 based on the assumption that the specific object is displayed at an intersection position where the viewing direction of the person 80 and the display surface of the display device 20 intersect, or in the vicinity of the intersection position. In this case, the "information indicating the display position" stored by the storage unit 280 together with the information indicating the viewing direction may be information indicating the display position of the specific object displayed on the display surface of the display device 20.

[0031] In the case where the display device 20 is a display device that displays an image recognized as a three-dimensional image (also called a stereoscopic image), the "information indicating the display position where the display device 20 displays a specific object that can be gazed at by the person 80" acquired by the display position information acquisition unit 210 and the "information indicating the display position" stored by the storage unit 280 together with the information indicating the viewing direction may be information indicating the position in three-dimensional space of an image recognized as a specific object in three-dimensional space, rather than a position on the display surface of the display device 20. In addition, the "information indicating the viewing direction of the person 80" acquired by the viewing direction information acquisition unit 220 may include information indicating the viewing direction of the right eye of the person 80 and information indicating the viewing direction of the left eye of the person 80. The estimation unit 240 may estimate at least one of the position and orientation of the display device 20 based on the assumption that a specific object in three-dimensional space recognized by the person 80 is (i) located at an intersection position where the viewing direction of the right eye of the person 80 and the viewing direction of the left eye of the person 80 substantially intersect, or (ii) located in the vicinity of the intersection position.

[0032] As an example, when the display device 20 is a display device that displays an image recognized as a three-dimensional image by displaying parallax images (e.g., a right-eye image presented to the right eye of the person 80 and a left-eye image presented to the left eye of the person 80), the "information indicating the display position at which the display device 20 displays a specific object that can be gazed upon by the person 80" acquired by the display position information acquisition unit 210 may be information indicating the position of the specific object in the right-eye image displayed by the display device 20 on the display surface of the display device 20, and the "information indicating the viewing direction of the person 80" acquired by the viewing direction information acquisition unit 220 may include at least information indicating the viewing direction of the right eye of the person 80. In this case, the "information indicating the display position" stored by the storage unit 280 together with the information indicating the viewing direction may be information indicating the position of a specific object in the right-eye image. The estimation unit 240 may estimate at least one of the position and the orientation of the display device 20, assuming that a specific object in the right-eye image is displayed at or near an intersection position where the viewing direction of the right eye of the person 80 and the display surface of the display device 20 intersect. On the other hand, the "information indicating the display position where the display device 20 displays a specific object that can be gazed upon by the person 80" acquired by the display position information acquisition unit 210 may be information indicating the position of a specific object in the left-eye image displayed by the display device 20 on the display surface of the display device 20, and the "information indicating the viewing direction of the person 80" acquired by the viewing direction information acquisition unit 220 may include at least information indicating the viewing direction of the left eye of the person 80. In this case, the "information indicating the display position" stored by the storage unit 280 together with the information indicating the viewing direction may be information indicating the position of a specific object in the left-eye image. The estimation unit 240 may estimate at least one of the position and orientation of the display device 20 based on the assumption that a specific object in the left eye image is displayed at or near the intersection position where the viewing direction of the left eye of the person 80 and the display surface of the display device 20 intersect.

[0033] In addition, when the display device 20 is a display device that displays an image recognized as a three-dimensional image, the "information indicating the viewing direction of the person 80" acquired by the viewing direction information acquisition unit 220 may be information indicating the viewing direction of the person 80 specified based on the direction of the face or head of the person 80, rather than information indicating the viewing direction of a specific eye of the person 80, and the "information indicating the display position, which is the position at which the display device 20 displays a specific object that can be gazed upon by the person 80" acquired by the display position information acquisition unit 210 may be information indicating the position of a specific object in an image displayed by the display device 20 on the display surface of the display device 20. The "information indicating the position of a specific object in an image displayed on the display surface of the display device 20" and the "information indicating the display position" stored by the storage unit 280 together with the information indicating the viewing direction may be information indicating one of (i) the position of a specific object in an image for the right eye on the display surface of the display device 20, and (ii) the position of a specific object in an image for the left eye on the display surface of the display device 20. The "information indicating the position of a specific object in an image displayed on the display surface of the display device 20" and the "information indicating the display position" stored by the memory unit 280 together with the information indicating the viewing direction may be the average position of (i) the position of a specific object in an image for the right eye on the display surface of the display device 20 and (ii) the position of a specific object in an image for the left eye on the display surface of the display device 20.

[0034] The judgment unit 230 may determine that a particular object is being gazed upon by a person captured in an image when the amount of change in the viewing direction over time is smaller than a predetermined value and the amount of change in the display position over time is less than a predetermined value.

[0035] The estimation section 240 may estimate at least one of the position and orientation of the display device 20 by further using the positional relationship between the display device 20 and the imaging device 40 and geometric attribute information of the display area of ​​the display device 20.

[0036] The estimation unit 240 may (i) assume a number of combinations of the position and orientation of the display device 20, (ii) calculate, for each of the multiple combinations, the distance between the position of the intersection of the viewing direction in three-dimensional space with the display device 20 and the display position of a specific object, and (iii) weight the position and orientation of the combination for which the shorter distance is calculated with a larger weight and use the weight for estimating the position and orientation of the display device 20.

[0037] The display control unit 250 determines a target area, which is an area to be concealed on the display device 20, based on the arrangement of the display device 20a identified by the estimation unit 240, and conceals a portion of the image to be displayed on the display device 20 based on the determined target area, and displays the image on the display device 20.

[0038] FIG. 3 shows a situation in which person 80a and person 80b are gazing at a pointer 300 displayed on the display device 20a. The pointer 300 is, for example, a mouse pointer. The position of the pointer 300 can be moved by the person 80 operating a pointing device such as a mouse, a touch pad, or a trackball. The pointer 300 is an example of a specific object that can be gazed at by the person 80. The pointer 300 is an example of a cursor displayed on the display device 20. The cursor is, for example, an object displayed on the display device 20 to indicate a position where character input is performed. The display position information acquisition unit 210 can acquire the position of the pointer 300 at least in pixel units from the operating system.

[0039] The viewing direction information acquisition unit 220 continuously acquires the viewing direction of the person 80 based on images continuously captured by the imaging device 40. The determination unit 230 determines whether or not the pointer 300 is being gazed at by the person 80 based on the time change in the viewing direction acquired by the viewing direction information acquisition unit 220 and the time change in the display position of the pointer 300 acquired by the display position information acquisition unit 210. For example, the determination unit 230 may determine that the pointer 300 is being gazed at by the person 80 when the time change in the viewing direction acquired by the viewing direction information acquisition unit 220 is smaller than a predetermined value and the time change in the display position of the pointer 300 is smaller than a predetermined value.

[0040] In addition, the determination unit 230 may determine that the pointer 300 is not gazed upon by the person captured in the image when the period during which the amount of change over time in the display position of the pointer 300 can be considered to be zero is longer than a predetermined value. The determination unit 230 may determine that the pointer 300 is gazed upon by the person 80 when the change in the display position of the pointer 300 starts from a state in which the amount of change over time in the display position of the pointer 300 can be considered to be zero, and the amount of change over time in the viewing direction acquired by the viewing direction information acquisition unit 220 is smaller than a predetermined value. The determination unit 230 may determine that the pointer 300 is gazed upon by the person 80 when the amount of change over time in the display position of the pointer 300 decreases from a state in which the amount of change over time in the display position of the pointer 300 is equal to or greater than a predetermined value to a state in which the amount of change over time in the viewing direction can be considered to be zero, and the amount of change over time in the viewing direction acquired by the viewing direction information acquisition unit 220 is smaller than a predetermined value.

[0041] For example, when a person 80 operates a pointing device to move the pointer 300 onto a button and clicks it, when the pointer 300 stops, it is highly likely that the person 80 is gazing at the position of the pointer 300. At this time, it is highly likely that the line of sight of the person 80 has not substantially changed. In this way, when the amount of change over time in the line of sight is smaller than a predetermined value and the amount of change over time in the display position of the pointer 300 is smaller than a predetermined value, it can be determined that the pointer 300 is being gazed at by the person 80.

[0042] In addition, in general, when a person is viewing an outdoor signboard or the like, if the person's gaze stays on one place for a minimum of about 0.3 seconds, the person can read up to about 15 characters. Therefore, it is considered that the optimal quantization unit in the time axis direction for determining whether the pointer 300 is being gazed upon by the person 80 captured in the image is set to a time interval of about 0.25 seconds, and by averaging and aggregating the coordinates of the viewpoint over that time interval, it is possible to determine whether the pointer 300 is being gazed upon by the person 80. If the time interval for determining whether the pointer 300 is being gazed upon by the person 80 is too narrow, the influence of noise becomes large, making it difficult to determine whether the pointer 300 is being gazed upon by the person 80, and therefore it is preferable to set the time interval to about 0.25 seconds.

[0043] In the real world, the person 80 and the pointer 300 are often aligned on a nearly horizontal plane when viewed from a bird's-eye view. In such cases, the height direction (height direction from the horizontal plane) of the monitor and the person's eyes are often all densely located within a width of about 1 to 2 m. Considering that the vertical (height direction) error of the gaze estimation technology is structurally likely to be large, it is easier to specify the degree of synchronization between the movement of the pointer 300 and the gaze in the height direction by quantizing and approximating the coordinate amount in the height direction into several meaningful stages rather than precisely analyzing the coordinate amount in the height direction, and when the degree of synchronization is high, it can be considered that the person 80 is gazing at the pointer 300. Furthermore, if the movement of the pointer 300 and the movement of the gaze direction stop, it is possible to determine with a higher probability that the gaze direction and the pointer are aligned, so that it is possible to determine with a higher accuracy whether the person 80 is gazing at the pointer 300 or not. For example, if the variation in the height direction is classified by quantizing and convoluting the height in units of 10 cm (however, consideration may be given to not processing information of the viewing direction outside the vertical width of 1 to 2 m by the viewing direction information acquisition unit 220), the position in the height direction of the pointer 300 will have about three stages of displacement in the width of the height of a general display device. This leads to quantization that has the meaning of whether a person is looking at the upper part, the middle part, or the lower part of the display device 20. For example, in a general user interface screen of the Windows (registered trademark) OS, looking at the upper part corresponds to operating the upper toolbar of the software interface, looking at the middle part corresponds to editing near the center, and looking at the lower part corresponds to operating the task bar or writing a comment in chat. In this way, by quantizing the height position to approximately 10 cm, it becomes easier to grasp the timing of meaningful synchronization between the pointer 300 and the gaze (synchronization between the up and down eye movement and the up and down movement of the pointer 300), and if the amount of displacement between their respective movements approaches zero while they are determined to be synchronized (for example, if the amount of displacement between the movements becomes less than a predetermined reference value), this can be regarded as an event action such as pressing a mouse click button, i.e., a gaze event, and the judgment unit 230 can judge that the pointer 300 is being gazed at by the person 80.

[0044] 4 shows an example of a data structure of gaze information data stored in the storage unit 280. The gaze information data associates a "display position," a "gaze vector," and "user identification information." The gaze information data is stored in the storage unit 280.

[0045] The "display position" indicates the display position on the display device 20 of the pointer 300 that is determined to be gazed upon by the person 80. The "display position" may include, for example, pixel-unit coordinates in the display area 10. In a configuration in which multiple display devices 20 are provided as in this embodiment, the "display position" may include pixel-unit coordinates for each display device 20 and identification information of the display device. Note that, for example, when the display areas 10 of multiple display devices 20 are expanded and the display devices 20 can be identified by their coordinates in the expanded coordinate system, or in other cases where it is possible to specify the display devices 20 from the coordinates, it is not necessary for the "display position" to include identification information of the display devices 20.

[0046] The "line-of-sight vector" is a vector that originates from the eye position of the person 80 and is parallel to the viewing direction of the person 80. The "line-of-sight vector" may be a vector that uses the position of the imaging device 40 as a reference position. The eye position of the person 80 may be identified from an image of the person 80 included in an image captured by the imaging device 40. The viewing direction information acquisition unit 220 identifies a relative positional relationship between the position of the imaging device 40 and the eye position of the person 80 (e.g., a distance in the z-axis direction and a position in a plane parallel to xy) from the size of the face of the person 80 in the image of the person 80. The viewing direction information acquisition unit 220 may identify a relative positional relationship between the position of the imaging device 40 and the eye position of the person 80, taking into account the imaging conditions of the imaging device 40 (e.g., focal length, etc.).

[0047] When the determination unit 230 determines that the pointer 300 is gazed at by the person 80, the storage unit 280 stores a set of the display position of the pointer 300, the line-of-sight vector of the person 80 based on the viewing direction, and user identification information as a record of gaze information data. Here, the person 80 may be identified by extracting a feature amount from an image of the face of the person 80 in an image captured by the imaging device 40, and identification information may be assigned to each person 80. In this way, the storage unit 280 can store information indicating the viewing direction at multiple times when it is determined that the pointer 300 is gazed at by the person 80 and information indicating the display position of the pointer 300.

[0048] 5 is an example of a flowchart relating to an information processing method executed in the information processing device 30. In S502, gaze information data is accumulated. Specifically, when the determination unit 230 determines that the pointer is gazed at by the person 80, the storage unit 280 stores a set of the display position of the pointer, the gaze vector, and the identification information of the person 80.

[0049] In S504, the estimation unit 240 estimates the number of display devices 20. For example, the estimation unit 240 estimates the number of display devices 20 based on gaze information data. With reference to FIG. 6, an example of a method for estimating the number of display devices 20 will be outlined. Based on gaze information data, the estimation unit 240 extracts a plurality of gaze vectors associated with displays at the same display position, and calculates the coordinates of an intersection on an extension line of the extracted plurality of gaze vectors (referred to as "intersection coordinates"), or the coordinates of a position where the extension line of the extracted plurality of gaze vectors approaches within a predetermined distance (referred to as "approaching point coordinates"). The approaching point coordinates may be, for example, a point on an extension line of one of the extracted plurality of gaze vectors, and a point where the distance from the point to the extension line of the other gaze vector is within a predetermined distance. Black circles (for example, black circle 650, etc.) shown in FIG. 6 indicate intersection coordinates and approaching point coordinates.

[0050] The estimation unit 240 groups the approach point coordinates and the intersection coordinates in the three-dimensional space to estimate the number of display devices 20. For example, as shown in Fig. 6, when the approach point coordinates and the intersection coordinates can be grouped into three groups (group 600, group 610, and group 620) based on the mutual positional relationships of the approach point coordinates and the intersection coordinates, the estimation unit 240 estimates the number of display devices 20 to be three.

[0051] Returning to the explanation of the flowchart of FIG. 5, in S510, a loop related to the number of display devices is started. The loop related to the number of display devices that starts in S510 repeats the processing between S510 and S540. The loop related to the number of display devices is executed assuming that the number of display devices 20 is a number between Mm and M+m, where M is the number of display devices 20 estimated in S504. However, if Mm is 0 or less, the loop is executed assuming that the number of display devices 20 is a number between 1 and M+m. Note that if the number of display devices is known (for example, if it is set in advance), the known number of display devices may be used. In this case, the loop processing related to the number of display devices from S510 to S540 (or a part of the loop processing) can be omitted.

[0052] In S512, the estimation unit 240 sets initial values ​​of the position and orientation of the display device 20. An example of the initial values ​​of the position and orientation of the display device 20 is shown in FIG. 7, FIG. 8, and FIG. 9. Note that the "position of the display device 20" may indicate the position of the display area 10 of the display device 20, and the "orientation of the display device 20" may indicate the orientation of the display area 10 of the display device 20. In FIG. 7 to FIG. 9, the reference numeral 700 indicates the position and orientation of the display device 20 corresponding to the group 600, the reference numeral 710 indicates the position and orientation of the display device 20 corresponding to the group 610, and the reference numeral 720 indicates the position and orientation of the display device 20 corresponding to the group 620. Note that FIG. 8 and FIG. 9 show different planes in the y-axis direction.

[0053] The estimation unit 240 may set the initial values ​​of the position and orientation of the display device 20 in consideration of the distribution of the approach point coordinates and the intersection coordinates. The estimation unit 240 may set the initial values ​​of the position and orientation of the display device 20 in consideration of the geometric attribute information of the display device 20 (for example, the actual size of the display area 10, specifically, the height and width of the display area 10, etc.). The estimation unit 240 may set the initial value of the position of the display device 20 in the z-axis direction to a position at a predetermined distance (for example, 50 cm) from the position of the person 80. The estimation unit 240 may set the initial value of the orientation of the display device 20 to a direction opposite to any of the line-of-sight vectors included in the gaze information data. The estimation unit 240 may acquire the actual size of the display area 10 of each display device 20 from, for example, an operating system running on the information processing device 30.

[0054] The estimation unit 240 may specify the direction of the face of the person 80 captured in the image from the image captured by the imaging device 40, and set the direction to the direction opposite to the specified direction. When a state in which the time change amount of the viewing direction of the person 80 is shorter than a predetermined value continues for a time longer than a predetermined value, the estimation unit 240 may set the initial value of the orientation of the display device 20 to the direction opposite to the viewing direction. When the number of times that the viewing direction of the person 80 turns to a specific direction is greater than a predetermined value, the estimation unit 240 may set the initial value of the orientation of the display device 20 to the direction opposite to the viewing direction. The estimation unit 240 may refer to the gaze information data, and based on the gaze vector associated with the identification information of the person 80, group the coordinates of the positions that the person 80 has looked at more than a predetermined number of times based on the gaze vector associated with the identification information of a specific person 80 (e.g., person 80a), thereby estimating the number of display devices 20 and setting the initial values ​​of the positions and orientations of the display devices 20.

[0055] Returning to the explanation of the flowchart in Fig. 5, in S514, a plurality of combinations of positions and orientations are set as candidates for the position and orientation of the display device 20. Fig. 10 shows a schematic setting example of a plurality of combinations of the position and orientation of the display device 20. Fig. 10 shows a schematic setting example of a plurality of combinations corresponding to the initial values ​​of the position and orientation of the display device 20 shown by reference numeral 720. In order to prevent the explanation from becoming redundant, a setting example of a plurality of combinations corresponding to reference numeral 720 will be explained here, and detailed explanations of setting examples of a plurality of combinations corresponding to reference numerals 700 and 710 will be omitted.

[0056] As shown in Fig. 10, 32 positions are set as the positions of the display device 20 around the initial value of the position of the display device 20 indicated by reference numeral 720. The 32 positions are arranged in a sector shape centered on a representative base point 1010 corresponding to the position of the head of the person 80. Three types of orientations are set for each of the 32 positions. For example, as indicated by reference numeral 1020, a first orientation 1021, a second orientation 1022, and a third orientation 1023 are set. The closer to reference numeral 720, the more positions the estimation unit 240 may set as the positions of the display device 20.

[0057] Returning to the description of the flowchart in Fig. 5, in S520, a loop for a plurality of combinations of positions and orientations is started. The loop started in S520 repeats the processes between S520 and S530. The loop for the combinations is executed assuming that the position and orientation of the display device 20 is each of the plurality of combinations set in S514.

[0058] The processes of S522, S524, and S526 will be described with reference to Fig. 11 and Fig. 12 as appropriate. In S522, the estimation unit 240 calculates an intersection between a line-of-sight vector in a three-dimensional space and the display area 10 of the display device 20. For example, as shown in Fig. 11 and Fig. 12, the estimation unit 240 calculates an intersection 1140 between the display area 1110 of the display device 20 assumed to be oriented in the first direction 1021 and an extension line of a line-of-sight vector 1120 included in the gaze information data and the display area 1110.

[0059] In S524, the estimation unit 240 calculates the display position of the pointer on the display area 10 in the three-dimensional space. For example, as shown in Fig. 11 and Fig. 12, the estimation unit 240 converts the position indicated by the coordinates of the display position included in the gaze information data into a display position 1150 on the display area 1110 based on the position and orientation of the display area 1110.

[0060] In S526, the estimation unit 240 calculates a distance D between an intersection point between the line of sight vector and the display area 10 of the display device 20, and a display position of the pointer on the display area 10 in the three-dimensional space. For example, as shown in Fig. 11 and Fig. 12, the estimation unit 240 calculates a distance D between the intersection point 1140 and a display position 1150.

[0061] The estimation unit 240 performs the processes of S522, S524, S526, and S528 for each record of the gaze information data.

[0062] In S528, the estimation unit 240 calculates a first evaluation value for the combination of the position and orientation. The first evaluation value is an example of a "weight". Here, the magnitude of the first evaluation value indicates the degree of reliability with which the position and orientation indicated by the current combination in this loop process can be trusted as the position and orientation of the display device 20. The estimation unit 240 may calculate a higher first evaluation value as the distance D is shorter. The estimation unit 240 may calculate a higher evaluation value for each of the multiple records included in the gaze information data as the distance D is shorter, and may use the sum of the evaluation values ​​calculated for each of the multiple records as the first evaluation value. Thereby, a high first evaluation value is calculated for a combination with many records having a high degree of coincidence between the display position of the pointer and the position of the viewing direction. As described later, a combination of a position and orientation with a high first evaluation value is used with a high weight to estimate the position and orientation of the display device 20.

[0063] The estimation unit 240 may select a record in which the distance D is less than a predetermined distance from among a plurality of records included in the gaze information data, calculate a deviation vector between the intersection of the extension of the gaze vector and the display area and the display position of the pointer for the selected record, and calculate an average value of the deviation vectors calculated for the selected record. The "predetermined distance" may be a distance of about 1 / 2 the size (e.g., width) of the display area 10 of the display device 20. The estimation unit 240 may correct the position of the display device 20 using the average value of the deviation vector. For example, the estimation unit 240 may determine the position of the display device 20 in the current combination as a correction vector, which is a vector in the opposite direction to the average value of the correction vector, and determine the position of the display device 20 shifted using the correction vector as the corrected position in the combination of the display devices 20.

[0064] In S532, the estimation unit 240 clusters a plurality of combinations of the positions and orientations of the display device 20 into the number of display devices in the current loop. For example, the estimation unit 240 may extract a plurality of combinations for which a first evaluation value higher than a predetermined value is calculated, and cluster the extracted plurality of combinations. An example of a clustering result of the combinations of positions and orientations is shown in Fig. 13 and Fig. 14. By clustering a plurality of combinations of the positions and orientations of the display device 20, the combinations of positions and orientations are clustered into cluster 1300, cluster 1310, and cluster 1320.

[0065] In S534, the estimation unit 240 calculates the position and orientation of the display device 20 by calculating the average value of the positions and orientations of the display device 20 clustered into each cluster. For example, the estimation unit 240 may determine the average value of the positions of the display device 20 clustered into the cluster 1300 shown in FIG. 14 as the position of the display device 20 estimated in the cluster 1300. Furthermore, the estimation unit 240 may determine the average value of the orientations of the display device 20 clustered into the cluster 1300 as the orientation of the display device 20 estimated in the cluster 1300. Similarly, the estimation unit 240 may determine the average value of the positions of the display device 20 clustered into the cluster 1310 shown in FIG. 14 as the position of the display device 20 estimated in the cluster 1310, and may determine the average value of the orientations of the display device 20 clustered into the cluster 1310 as the orientation of the display device 20 estimated in the cluster 1310. Similarly, the average value of the positions of the display devices 20 clustered into cluster 1320 shown in FIG. 13 may be determined as the position of the display device 20 estimated in cluster 1320, and the average value of the orientations of the display devices 20 clustered into cluster 1320 may be determined as the orientation of the display device 20 estimated in cluster 1320.

[0066] In S536, the estimation unit 240 calculates a second evaluation value for the position and orientation of the display device 20 calculated in S534. For example, the estimation unit 240 calculates the second evaluation value by performing the same processing as the processing of S522, S534, S526, and S528. As with the first evaluation value, the magnitude of the second evaluation value may indicate the degree of reliability with which the current number of display devices can be trusted as the position and orientation of the display device 20. The estimation unit 240 may calculate a higher second evaluation value as the distance D is shorter. The estimation unit 240 may calculate a higher evaluation value for each of the multiple records included in the gaze information data as the distance D is shorter, and may use the sum of the evaluation values ​​calculated for each of the multiple records as the second evaluation value.

[0067] When the loop process on the number of display devices in S510 to S540 is completed, the estimated position and orientation of the display device 20 is determined in S542. For example, the estimation unit 240 determines the estimated position and orientation of the display device 20 based on the second evaluation value calculated in S536. Specifically, the estimation unit 240 determines the number of display devices corresponding to the loop in which the largest second evaluation value is calculated among the second evaluation values ​​calculated in the process of each loop as the number of display devices 20, and determines the position and orientation of the display device 20 calculated in S534 in the loop in which the largest second evaluation value is calculated as the estimated position and orientation of the display device 20. In this way, the estimation unit 240 can estimate the position and orientation of the display device 20 using the viewing direction of the person 80 and the display position of the pointer when the person 80 is looking in the viewing direction.

[0068] 15 is a schematic diagram showing an example of the concealment process by the display control unit 250. In the information processing device 30, a predetermined range 110 is set in which the imaging device 40 can monitor whether or not another person is present in a position where the display areas of the display devices 20b and 20c are visible, on the condition that the display devices 20b and 20c are arranged in a predetermined positional relationship. In other words, if the display devices 20b and 20c are arranged in a predetermined normal positional relationship, the imaging device 40 can monitor whether or not another person is present in a position where the range 110 is visible, and therefore there is no need to perform concealment process on the image in the display area 10 in the range 110.

[0069] However, if the display device 20b and the display device 20c are not arranged in a predetermined regular positional relationship, there is a possibility that the imaging device 40 cannot monitor whether or not another person is present in a position where the range 110 can be viewed. Therefore, the display control unit 250 determines an area to be concealed in the display area 10b based on the position and orientation of the display area 10b specified by the estimation unit 240, and conceals at least a part of the image to be displayed on the display device 20b based on the determined area, and displays it on the display device 20b. As an example, when the angle between the normal direction of the display area 10b of the display device 20b and the imaging direction of the imaging device 40 exceeds a predetermined value, the display control unit 250 subjects the entire image area of ​​the display area 10b of the display device 20b to the concealment process as shown in FIG. 15. This makes it possible to suppress information leakage through the display devices 20b and 20c, which can be arranged particularly freely.

[0070] The area to be subjected to the concealment process does not have to be the entire display area 10, and may be a part of the display area 10. For example, the area to be subjected to the concealment process may be an area outside the range 110 of the entire image area of ​​the display area 10. The area to be subjected to the concealment process may include an area outside the range 110 of the entire image area of ​​the display area 10 and a part of an area within the range 110. The part of the area within the range 110 that may be subjected to the concealment process may be an area near the boundary line of the range 110. Furthermore, the concealment may be limited to one display device 20, or may be performed on multiple display devices 20, or different concealment processes may be performed for each display device 20.

[0071] In FIG. 15, the display control unit 250 determines, based on the position and orientation of the display device 20c estimated by the estimation unit 240, an area 1510 and an area 1520 that are outside the range 110 in the display area 10c as target areas for the concealment process. When performing the concealment process, the display control unit 250 may increase the strength of the concealment process for an area that is far from the position where the imaging device 40 is provided. Examples of the strength of the concealment process include the degree of mixing of other images, the degree of superimposition of other images, the transmittance of the superimposed image, the display size of the presented information (e.g., the size of characters), the contrast of the presented information, the color of the presented information and the background, and the brightness of the screen or a part of the screen. Examples of the degree of mixing of other images include the fineness of the mosaic and the strength of blurring.

[0072] 8 to 14 show examples in which the display area 10 is planar, but the present invention is also applicable to a non-planar display area (e.g., a curved display area). The estimation unit 240 may identify information indicating whether the display surface of the display device is curved and information indicating the degree of curvature of the display area (e.g., curvature information, etc.) based on model information of the display device held by the operating system, and may estimate the position and orientation of the display device 20 based on the identified information.

[0073] In the above, a pointer is exemplified as a "specific object", but the "specific object" may be any object that a person can focus on. The "specific object" may be a warning object for the user of the information processing device 30. The "specific object" may be a region of interest (e.g., ROI) in a video such as a movie. The position of the "specific object" may be acquired from an operating system, but may also be identified by analyzing an image displayed on the display device 20. For example, the position with the highest score calculated by a predetermined method for calculating a score as the position of a specific object among (i) the position of the center of gravity of an area where the density of feature points extracted by feature point analysis is higher than a reference value and the density of the feature points is gradually increasing or growing, (ii) the position of a face extracted by face detection technology that is larger in size than a reference value, (iii) the position of the center of gravity of an area where the color change is larger than a reference value, and (iv) a specific position on the display surface (e.g., the center position of the screen, etc.) may be the position of the "specific object". In addition, a form in which the information processing device 30 actively displays the "specific object" on the display device 20 may be adopted.

[0074] In the above, the estimation unit 240 of the information processing device 30 estimates the position and direction of the display device 20 by using the viewing direction of the person 80 captured by the imaging device 40. However, the estimation unit 240 may estimate at least one of the position and direction of the display device 20 by using the viewing direction of the person 80 captured by the imaging device 40. For example, the estimation unit 240 may estimate the mutual positional relationship of the display devices 20 by referring to the setting information, and estimate the direction of the display devices 20 by using the viewing direction of the person 80 captured by the imaging device 40. For example, the estimation unit 240 may estimate the mutual positional relationship of the display devices 20 (for example, the positional relationship in the x-axis direction and the y-axis direction) by referring to the arrangement information of the extended display set in advance in the operating system, and estimate the position of the display area 10 in the z-axis direction to be a position that is a predetermined distance (for example, a general 50 cm) away from the person 80. As another example, the estimation unit 240 may estimate the orientation of the display device 20 to be a predetermined orientation (e.g., the same orientation as the imaging direction of the imaging device 40) and estimate the position of the display device 20 using the viewing direction of the person 80 imaged by the imaging device 40.

[0075] In the above, a configuration in which three display devices 20 are present has been described. However, the number of display devices 20 is not limited to three. As other configurations, any of a configuration in which one display device 20 is present, a configuration in which two display devices 20 are present, and a configuration in which three or more display devices 20 are present may be adopted.

[0076] In the above, a configuration in which the information processing device 30 includes the imaging device 40 has been described. However, a configuration in which the imaging device 40 is not provided in the information processing device 30 may be adopted. In this case, it is preferable that the positional relationship between the imaging device and the display device 20a and the imaging direction of the imaging device are known or can be estimated by some means (e.g., communication means, etc.) in the information processing device 30. Furthermore, the number of imaging devices is not limited to one, and a configuration in which multiple imaging devices are included may be adopted. When multiple imaging devices are included, it is preferable that the positional relationship between the imaging devices and the imaging direction of each imaging device are known or can be estimated by some means (e.g., communication means, etc.).

[0077] A part of the processing of the processing unit 200 may be realized by processing using machine learning.

[0078] According to the information processing system 100 described above, at least one of the position and the direction of the display device 20 can be estimated using the viewing direction of the person 80 imaged by the imaging device 40. Therefore, the information processing device 30 can perform various display controls based on at least one of the estimated position and the direction of the display device 20. For example, the display contents of the display device 20 can be controlled so that information is not leaked through the display device 20.

[0079] 16 shows an example of a computer 2000 in which multiple embodiments of the present invention can be embodied in whole or in part. A program installed in the computer 2000 can cause the computer 2000 to function as an information processing system 100 according to an embodiment or each part of the system, or an apparatus such as an information processing apparatus 30, or each part of the apparatus, execute operations associated with the system or each part of the system or the apparatus or each part of the apparatus, and / or execute a process or a step of the process according to an embodiment. Such a program can be executed by the CPU 2012 to cause the computer 2000 to execute specific operations associated with some or all of the blocks of the processing procedures and block diagrams described herein.

[0080] The computer 2000 according to this embodiment includes a CPU 2012 and a RAM 2014, which are connected to each other by a host controller 2010. The computer 2000 also includes a ROM 2026, a flash memory 2024, a communication interface 2022, and an input / output chip 2040. The ROM 2026, the flash memory 2024, the communication interface 2022, and the input / output chip 2040 are connected to the host controller 2010 via the input / output controller 2020.

[0081] The CPU 2012 operates according to the programs stored in the ROM 2026 and RAM 2014, thereby controlling each unit.

[0082] The communication interface 2022 communicates with other electronic devices via a network. The flash memory 2024 stores programs and data used by the CPU 2012 in the computer 2000. The ROM 2026 stores a boot program and the like executed by the computer 2000 upon activation, and / or a program that depends on the hardware of the computer 2000. The input / output chip 2040 may also connect various input / output units, such as a keyboard, mouse, and monitor, to the input / output controller 2020 via input / output ports, such as a serial port, a parallel port, a keyboard port, a mouse port, a monitor port, a USB port, an HDMI port, and the like.

[0083] The programs are provided via a computer-readable storage medium such as a CD-ROM, a DVD-ROM, or a memory card, or a network. The RAM 2014, the ROM 2026, or the flash memory 2024 are examples of computer-readable storage media. The programs are installed in the flash memory 2024, the RAM 2014, or the ROM 2026, and executed by the CPU 2012. Information processing described in these programs is read by the computer 2000, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing an operation or processing of information according to the use of the computer 2000.

[0084] For example, when communication is performed between the computer 2000 and an external device, the CPU 2012 may execute a communication program loaded in the RAM 2014 and instruct the communication interface 2022 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 2012, the communication interface 2022 reads transmission data stored in a transmission buffer processing area provided in a recording medium such as the RAM 2014 and the flash memory 2024, transmits the read transmission data to a network, and writes reception data received from the network to a reception buffer processing area or the like provided on the recording medium.

[0085] Furthermore, the CPU 2012 may cause all or a necessary portion of a file or database stored on a recording medium such as the flash memory 2024 to be read into the RAM 2014, and perform various types of processing on the data on the RAM 2014. The CPU 2012 then writes back the processed data to the recording medium.

[0086] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and subjected to information processing. The CPU 2012 may execute various types of processing, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., described herein and specified by the instruction sequence of the program, on the data read from the RAM 2014, and write back the results to the RAM 2014. The CPU 2012 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 2012 may search for an entry that matches a condition in which an attribute value of the first attribute is specified from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0087] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 2000 or in the vicinity of the computer 2000. A recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as the computer-readable storage medium. The programs stored in the computer-readable storage medium may be provided to the computer 2000 via a network.

[0088] A program installed in the computer 2000 and causing the computer 2000 to function as the information processing device 30 may, when executed by the computer, act on the CPU 2012 and the like to cause the computer 2000 to function as each part of the information processing device 30. When the information processing described in these programs is read into the computer 2000, it functions as each part of the information processing device 30, which is a specific means in which the software and the above-mentioned various hardware resources work together. Then, by implementing calculation or processing of information according to the intended use of the computer 2000 in this embodiment by these specific means, a specific information processing device 30 according to the intended use is constructed.

[0089] Various embodiments have been described with reference to block diagrams or the like. In the block diagrams, each block may represent (1) a stage of a process where an operation is performed or (2) a portion of an apparatus responsible for performing an operation. Particular stages and portions may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuitry and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuitry including logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, memory elements such as flip-flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like.

[0090] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon constitutes at least a portion of a product that includes instructions that can be executed to provide a means for performing the operations specified in a process or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0091] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0092] The computer readable instructions may be provided to a processor or programmable circuitry of a programmable data processing apparatus, either locally or over a local area network (LAN), a wide area network (WAN) such as the Internet, etc., and the computer readable instructions may be executed to provide means for performing operations specified in the process steps or block diagrams described.

[0093] Here, the computer may be a computer such as a PC (personal computer), a tablet computer, a smartphone, a workstation, a server computer, or a general-purpose computer, or may be a computer system in which multiple computers are connected. Such a computer system in which multiple computers are connected is also called a distributed computing system, and is a computer in the broad sense. In a distributed computing system, each of the multiple computers executes a part of a program, and the multiple computers collectively execute a program by transferring data during program execution between the computers as necessary.

[0094] Examples of processors include computer processors, central processing units (CPUs), processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc. A computer may have one processor or multiple processors. In a multiprocessor system with multiple processors, each processor executes a part of a program, and the multiple processors collectively execute a program by passing data during program execution between the processors as necessary. For example, in executing multitasks, each of the multiple processors may execute a part of each task in small chunks by switching tasks for each time slice. In this case, which part of a program each processor executes changes dynamically. Also, which part of a program each of the multiple processors executes may be statically determined by programming that takes the multiprocessor into consideration.

[0095] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0096] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and may be realized in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is essential to perform the process in this order. [Explanation of symbols]

[0097] 10 Display area 20 Display device 30 Information processing device 40 Imaging device 80 people 100 Information Processing Systems 110 Range 200 Processing section 210 Display position information acquisition unit 220 Viewing direction information acquisition unit 230 Judgment Department 240 Estimation part 250 Display control unit 280 Storage section 300 Pointer 600, 610, 620 Group 650 Black Circle 700, 710, 720, 1020 code 1110 Display area 1120 Gaze Vector 1140 intersection 1150 Display position 1300 Cluster 1310 Cluster 1320 Cluster 1510 area 1520 areas 2000 Computer 2010 Host Controller 2012 CPU 2014 RAM 2020 Input / Output Controller 2022 Communication Interface 2024 Flash Memory 2026 ROM 2040 Input / Output Chip

Claims

1. a viewing direction information acquisition unit that acquires information indicating a viewing direction of a person acquired from an image of the person captured by an imaging device that captures an image of the surroundings of one or more display devices; a display position information acquisition unit that acquires information indicating a display position, which is a position at which the one or more display devices display a specific object that may be gazed upon by a person at a timing corresponding to a timing at which the image is captured; an estimation unit that estimates at least one of a position and an orientation of the one or more display devices using the information indicating the viewing direction and the information indicating the display position; Equipped with The estimation unit is Assuming a plurality of combinations of positions and orientations of the one or more display devices; For each of the plurality of combinations, a distance between a position of an intersection of the viewing direction and the one or more display devices in a three-dimensional space and a display position of the specific object is calculated; The position and orientation of the combination for which the distance is calculated to be shorter among the plurality of combinations is weighted with a larger weight and used to estimate the position and orientation of the one or more display devices. Information processing device.

2. a determination unit that determines whether or not the specific object is being gazed at by a person captured in the image based on the change over time in the viewing direction and the change over time in the display position; a storage unit that stores information indicating the viewing direction and information indicating the display position at a plurality of times when the determination unit has determined that the specific object is being gazed at by the person captured in the image; Further equipped with The estimation unit estimates at least one of a position and an orientation of the one or more display devices using information indicating the viewing direction and information indicating the display position at the plurality of times stored in the storage unit. The information processing device according to claim 1 .

3. The particular object is a cursor displayed on the one or more display devices.

3. The information processing device according to claim 1 or 2.

4. The determination unit determines that the specific object is being gazed at by a person captured in the image when the amount of change over time in the viewing direction is smaller than a predetermined value and the amount of change over time in the display position is smaller than a predetermined value. The information processing device according to claim 2 .

5. The estimation unit estimates at least one of a position and an orientation of the one or more display devices by further using a positional relationship between the one or more display devices and the imaging device and geometric attribute information of a display area of ​​the one or more display devices.

3. The information processing device according to claim 1 or 2.

6. obtaining information indicative of a viewing direction of a person, the information being obtained from an image of the person captured by an imaging device that captures an image of a surrounding of the one or more display devices; acquiring information indicating a display position at which the one or more display devices displayed a specific object that may be gazed upon by a person at a timing corresponding to a timing at which the image was captured; estimating at least one of a position and an orientation of the one or more display devices using the viewing direction information and the display position information; Equipped with The estimating step includes: assuming a plurality of combinations of positions and orientations of the one or more display devices; calculating, for each of the plurality of combinations, a distance between a position of an intersection of the viewing direction and the one or more display devices in a three-dimensional space and a display position of the specific object; weighting the position and orientation of a combination for which the distance is shorter among the plurality of combinations with a larger weight and using the weighted position and orientation of the one or more display devices; An information processing method comprising the steps of:

7. A program for causing a computer to function as the information processing device according to claim 1 or 2 when executed by the computer.

Images