Information processing device

Abstract

The information processing device includes: a viewing direction information acquisition unit that acquires information indicating the viewing direction of a person obtained from an image of a person captured by an imaging device that images the area around one or more display devices; a display position information acquisition unit that acquires information indicating the display position, which is the position where one or more display devices display a specific object that can be viewed by the person at a timing corresponding to the timing when the image was captured; and an estimation unit that estimates at least one of the position and orientation of one or more of the one or more display devices using the viewing direction information and the display position information.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Patent Document 1 discloses an information display device that automatically makes settings so as not to be peeked at by others when it is determined that there is a risk of being peeked at by others. Patent Document 2 discloses an electronic device that performs anti-peeping control when the line of sight of a person other than the user is detected. Patent Document 3 discloses a technique for acquiring, for each user, line-of-sight information indicating the line of sight that the user has gazed at in a screen image, and displaying which display area within the screen image the user has gazed at. (Prior Art Documents) (Patent Documents) (Patent Document 1) Japanese Unexamined Patent Application Publication No. 2012-129701 (Patent Document 2) Japanese Patent No. 6474921 (Patent Document 3) Japanese Patent No. 6485522

Summary of the Invention

[0003] In a first aspect of the present invention, an information processing apparatus is provided. The information processing apparatus includes a viewing direction information acquisition unit that acquires information indicating the viewing direction of a person, which is acquired from an image of the person captured by an imaging device that captures the surroundings of one or more display devices. The information processing apparatus includes a display position information acquisition unit that acquires information indicating a display position, which is a position where a specific object that can be gazed at by a person is displayed on the one or more display devices at a timing corresponding to the timing when the image is captured. The information processing apparatus includes an estimation unit that estimates at least one of the position and orientation of the one or more display devices by using the information indicating the viewing direction and the information indicating the display position.

[0004] The above-described information processing device may include a determination unit that determines whether the specific object is being gazed upon by a person captured in the image, based on the time change in the viewing direction and the time change in the display position. The information processing device may include a storage unit that stores information indicating the viewing direction and information indicating the display position at multiple timings in which the determination unit determines that the specific object is being gazed upon by a person captured in the image. The estimation unit may use the information indicating the viewing direction and information indicating the display position at multiple timings stored in the storage unit to estimate at least one of the position and orientation of the one or more display devices.

[0005] In any of the above-described information processing devices, the specific object may be a cursor displayed on one or more of the display devices.

[0006] In any of the above-described information processing devices, the determination unit may determine that the specific object is being gazed upon by the person captured in the image if the amount of change in the viewing direction over time is less than a predetermined value and the amount of change in the display position over time is less than a predetermined value.

[0007] In any of the above-described information processing devices, the estimation unit may further use the positional relationship between the one or more display devices and the imaging device, and the geometric attribute information of the display areas of the one or more display devices to estimate at least one of the position and orientation of the one or more display devices.

[0008] In any of the above-described information processing devices, the estimation unit may assume a plurality of combinations of the position and orientation of the one or more display devices, calculate the distance between the position of the intersection point of the viewing direction in three-dimensional space and the one or more display devices and the display position of the specific object for each of the plurality of combinations, and use the combination for which the shorter distance is calculated among the plurality of combinations to estimate the position and orientation of the one or more display devices by weighting it with a larger weight.

[0009] A second embodiment of the present invention provides an information processing method. The information processing method comprises the step of acquiring information indicating the viewing direction of a person, obtained from an image of a person captured by an imaging device that images the area around one or more display devices. The information processing method comprises the step of acquiring information indicating a display position, which is the position where the one or more display devices displayed a specific object that can be viewed by the person at a timing corresponding to the timing when the image was captured. The information processing method comprises the step of estimating at least one of the position and orientation of the one or more display devices using the viewing direction information and the display position information.

[0010] In a third aspect of the present invention, a program is provided. When the program is executed by a computer, it causes the computer to function as one of the information processing devices described above.

[0011] It should be noted that the above summary of the invention does not enumerate all of its features. Furthermore, subcombinations of these features may also constitute an invention. [Brief explanation of the drawing]

[0012] [Figure 1] A schematic representation of how the information processing system 100 is used in one embodiment is shown below. [Figure 2] The functional blocks of the information processing device 30 are shown schematically. [Figure 3]This shows a situation where person 80a and person 80b are looking at the pointer 300 displayed on the display device 20a. [Figure 4] An example of the data structure of gaze information data stored in the memory unit 280 is shown. [Figure 5] This is an example of a flowchart illustrating an information processing method executed in the information processing device 30. [Figure 6] This diagram provides a schematic explanation of one example of a method for estimating the number of display devices 20. [Figure 7] An example of the initial values ​​for the position and orientation of the display device 20 is schematically shown. [Figure 8] An example of the initial values ​​for the position and orientation of the display device 20 is schematically shown. [Figure 9] An example of the initial values ​​for the position and orientation of the display device 20 is schematically shown. [Figure 10] A schematic example of setting multiple combinations of the position and orientation of the display device 20 is shown. [Figure 11] This diagram illustrates the calculation process for the information used to determine the evaluation value. [Figure 12] This diagram illustrates the calculation process for the information used to determine the evaluation value. [Figure 13] An example of clustering results based on combinations of location and orientation is shown. [Figure 14] An example of clustering results based on combinations of location and orientation is shown. [Figure 15] An example of the concealment process performed by the display control unit 250 is schematically shown. [Figure 16] An example using Computer 2000 is shown. [Modes for carrying out the invention]

[0013] The present invention will be described below through embodiments of the invention, but these embodiments are not intended to limit the invention as defined in the claims. Furthermore, not all combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0014] FIG. 1 schematically shows the usage mode of the information processing system 100 in one embodiment. The information processing system 100 includes an information processing apparatus 30, a display device 20b, and a display device 20c. The information processing apparatus 30 includes a display device 20a and an imaging device 40.

[0015] The information processing apparatus 30 is, for example, a user terminal such as a personal computer or a mobile terminal. Examples of the mobile terminal include a mobile phone, a smartphone, a PDA (registered trademark), a tablet, a notebook computer or a laptop computer, and a wearable computer. Person 80a is a user of the information processing system 100. Person 80b is a user different from person 80a. Persons 80a and 80b may be collectively referred to as "person 80".

[0016] The imaging device 40 images the periphery of the display device 20a. The imaging device 40, for example, images at least the space on the display area 10a side of the display device 20a. The imaging device 40 is fixed to the information processing apparatus 30. The imaging device 40 may have a function of imaging by visible light. The imaging device 40 may have a function of imaging by infrared rays.

[0017] The display devices 20b and 20c are display devices separate from the information processing apparatus 30. For example, the display devices 20b and 20c are external display devices of the information processing apparatus 30. The display device 20b has a display area 10b, and the display device 20c has a display area 10c. The display area 10a, the display area 10b, and the display area 10c may be collectively referred to as "display area 10". In the embodiment related to the information processing system 100, the display device 20a is a display device provided in the information processing apparatus 30 itself, while the display devices 20b and 20c are external display devices of the information processing apparatus 30.

[0018] The information processing device 30 presents information to the person 80 by displaying information on the display devices 20a, 20b, and 20c. In this embodiment, the display device 20b is positioned to the right of the display device 20a as viewed from the person 80, and the display device 20c is positioned above the display device 20a as viewed from the person 80. In embodiments relating to the information processing system 100, the display devices 20a, 20b, and 20c may be collectively referred to as "display device 20". Examples of display devices 20a, 20b, and 20c include displays, projectors, televisions, etc. The display devices 20a, 20b, and 20c may be display devices that display images that are recognized as three-dimensional images.

[0019] The information processing device 30 performs processing to prevent the information displayed on the display devices 20a, 20b, and 20c from being leaked to the outside. For example, the information processing device 30 determines from the image captured by the imaging device 40 whether there are any persons other than person 80a who are registered as users of the information processing device 30, and if there are persons other than person 80a, it prevents the information from being leaked to the outside by concealing the information displayed on the display devices 20a, 20b, and 20c. The information processing device 30 may conceal the information displayed on the display devices 20a, 20b, and 20c, provided that it has been instructed to do so by the administrator of the information processing system 100 or person 80a.

[0020] Concealment processing includes, for example, conversion to another image or overlaying another image. Examples of conversion to another image include mosaic processing, blurring processing, and blending processing. Overlaying another image includes grayscale processing.

[0021] Since the display devices 20b and 20c are separate display devices from the information processing device 30, they can be freely positioned. Depending on the position and / or orientation of the display devices 20b and 20c, it may not be possible for the imaging device 40 to capture an image of a position where the display devices 20b and 20c can be viewed. Therefore, depending on the position and / or orientation of the display devices 20b and 20c, it may not be possible for the imaging device 40 to monitor whether or not there are people other than person 80a.

[0022] The information processing device 30 identifies the direction of the person 80's gaze from the image captured by the imaging device 40, and identifies the display position of a specific object among the objects displayed on the display device 20 that is highly likely to be viewed by the person 80. Based on the direction of the person 80's gaze and the display position of the specific object, the information processing device 30 estimates at least one of the position and orientation of the display device 20. Based on at least one of the estimated position and orientation of the display device 20, the information processing device 30 determines whether or not to perform concealment processing on at least one area within the display area 10. For example, if the information processing device 30 determines that the position from which the display device 20b can be viewed cannot be monitored by the imaging device 40, it conceals the information displayed on the display device 20b.

[0023] In describing this embodiment, the position and orientation of the display device 20 may be described using a three-dimensional coordinate system. In this embodiment, the z-axis direction of the coordinate system is assumed to be parallel to the imaging direction of the imaging device 40. The imaging direction of the imaging device 40 is assumed to substantially coincide with the normal direction of the display area 10a of the display device 20a. The x-axis direction of the coordinate system coincides with the horizontal direction of the display area 10a, and the y-axis direction of the coordinate system coincides with the vertical direction of the display area 10a.

[0024] Figure 2 schematically shows the functional blocks of the information processing device 30. The information processing device 30 comprises a processing unit 200, an imaging device 40, and a display device 20a. The processing unit 200 has the function of controlling the display of the display device 20a, as well as the function of controlling the displays of the display devices 20b and 20c connected to the information processing device 30.

[0025] The processing unit 200 includes a display position information acquisition unit 210, a viewing direction information acquisition unit 220, a determination unit 230, an estimation unit 240, a display control unit 250, and a storage unit 280.

[0026] The viewing direction information acquisition unit 220 acquires information indicating the viewing direction of a person 80 from an image of the person 80 captured by the imaging device 40 that images the area around the display device 20. The viewing direction information acquisition unit 220 may acquire the viewing direction of person 80 based on an image of the person 80's face. For example, the viewing direction information acquisition unit 220 may identify the orientation of the person 80's face or head based on an image of the person 80's head captured by the imaging device 40, and acquire the identified orientation as the viewing direction of person 80. The viewing direction information acquisition unit 220 may identify the position of the person 80's pupils from an image of the person 80's face captured by the imaging device 40, and identify the viewing direction of person 80 based on the identified position of the pupils. The viewing direction information acquisition unit 220 may identify the position of the person 80's pupils from an image of the person 80's face captured by the imaging device 40, and identify the viewing direction of person 80 based on the identified position of the pupils and the orientation of person 80's head. The viewing direction information acquisition unit 220 may continuously acquire the viewing direction of the person 80 based on the images continuously input by the imaging device 40, which are images captured by the imaging device 40.

[0027] The display position information acquisition unit 210 acquires information indicating the display position, which is the position where the display device 20 displays a specific object that can be viewed by the person 80 at a timing corresponding to the timing when the image was captured by the imaging device 40. For example, the specific object may be a pointer displayed on the display device 20. The display position information acquisition unit 210 may acquire the display position of the pointer through the operating system. The estimation unit 240 estimates at least one of the position and orientation of one or more display devices using information indicating the viewing direction and information indicating the display position.

[0028] Specific objects may include, in addition to the pointers mentioned above (e.g., mouse pointers), objects that are displayed as blinking objects, objects that are displayed as highlighted objects, objects that are displayed as enlarged objects, objects that are displayed as pop-ups, parts of a video where the variation in the displayed content is greater than a baseline value, and display objects that guide the user (e.g., display objects that give instructions to the user, such as "Look at the right edge of the display").

[0029] Specifically, the determination unit 230 determines whether a particular object is being gazed at by a person captured in the image, based on the temporal changes in the viewing direction and the temporal changes in the display position. The storage unit 280 stores information indicating the viewing direction and information indicating the display position at multiple timings in which the determination unit 230 determines that the particular object is being gazed at by a person captured in the image. The estimation unit 240 uses the information indicating the viewing direction and information indicating the display position at multiple timings stored in the storage unit 280 to estimate at least one of the position and orientation of the display device 20.

[0030] If the display device 20 is a display device that displays a two-dimensional image, the "information indicating the display position, which is the position where the display device 20 displays a specific object that can be viewed by the person 80," acquired by the display position information acquisition unit 210 may be information indicating the position on the display surface of the display device 20 where the display device 20 displayed the specific object. The estimation unit 240 may estimate at least one of the position and orientation of the display device 20 based on the assumption that the specific object is displayed at the intersection point, which is the position where the viewing direction of the person 80 intersects with the display surface of the display device 20, or in the vicinity of said intersection point. In this case, the "information indicating the display position," which the storage unit 280 stores together with the information indicating the viewing direction, may be information indicating the display position of the specific object displayed on the display surface of the display device 20.

[0031] If the display device 20 is a display device that displays an image that is recognized as a three-dimensional image (also called a stereoscopic image), then the "information indicating the display position, which is the position where the display device 20 displays a specific object that can be gazed upon by the person 80" acquired by the display position information acquisition unit 210, and the "information indicating the display position" stored by the storage unit 280 together with the information indicating the viewing direction, may not be a position on the display surface of the display device 20, but rather information indicating the position in three-dimensional space of an image that is recognized as a specific object in three-dimensional space. Furthermore, the "information indicating the viewing direction of the person 80" acquired by the viewing direction information acquisition unit 220 may include information indicating the viewing direction of the person 80's right eye and information indicating the viewing direction of the person 80's left eye. The estimation unit 240 may estimate at least one of the position and orientation of the display device 20 based on the assumption that a specific object in three-dimensional space recognized by the person 80 is located at an intersection point where (i) the viewing direction of the person 80's right eye and the viewing direction of the person 80's left eye substantially intersect, or (ii) in the vicinity of said intersection point.

[0032] For example, if the display device 20 is a display device that displays an image that is recognized as a three-dimensional image by displaying a parallax image (for example, a right-eye image presented to the right eye of person 80 and a left-eye image presented to the left eye of person 80), the "information indicating the display position, which is the position where the display device 20 displays a specific object that can be gazed upon by person 80" acquired by the display position information acquisition unit 210 may be information indicating the position of a specific object in the right-eye image displayed on the display surface of the display device 20, and the "information indicating the viewing direction of person 80" acquired by the viewing direction information acquisition unit 220 may include at least information indicating the viewing direction of person 80's right eye. In this case, the "information indicating the display position" stored by the storage unit 280 together with the viewing direction information may be information indicating the position of a specific object in the right-eye image. The estimation unit 240 may estimate at least one of the position and orientation of the display device 20 by assuming that a specific object in the right-eye image is displayed at or near the intersection point, which is the position where the viewing direction of the person 80's right eye intersects with the display surface of the display device 20. On the other hand, the "information indicating the display position, which is the position where the display device 20 displays a specific object that can be gazed upon by the person 80," acquired by the display position information acquisition unit 210 may be information indicating the position of a specific object in the left-eye image displayed by the display device 20 on its display surface, and the "information indicating the viewing direction of the person 80," acquired by the viewing direction information acquisition unit 220 may include at least information indicating the viewing direction of the person 80's left eye. In this case, the "information indicating the display position," which the storage unit 280 stores together with the information indicating the viewing direction, may be information indicating the position of a specific object in the left-eye image. The estimation unit 240 may estimate at least one of the position and orientation of the display device 20 based on the assumption that a specific object in the left-eye image is displayed at or near the intersection point where the viewing direction of the left eye of the person 80 intersects with the display surface of the display device 20.

[0033] Furthermore, in the case where the display device 20 is a display device that displays an image recognized as a three-dimensional image, the "information indicating the viewing direction of the person 80" acquired by the viewing direction information acquisition unit 220 may not be information indicating the viewing direction of a specific eye of the person 80, but rather information indicating the viewing direction of the person 80 identified based on the orientation of the person 80's face or head, etc. The "information indicating the display position, which is the position where the display device 20 displays a specific object that can be gazed upon by the person 80" acquired by the display position information acquisition unit 210 may be information indicating the position of a specific object in the image displayed on the display surface of the display device 20. The "information indicating the position of a specific object in the image displayed on the display surface of the display device 20" and the "information indicating the display position" stored by the storage unit 280 together with the viewing direction information may be information indicating either (i) the position of a specific object in the right-eye image on the display surface of the display device 20, or (ii) the position of a specific object in the left-eye image on the display surface of the display device 20. The "information indicating the position of a specific object in an image displayed on the display surface of the display device 20" and the "information indicating the display position" stored in the storage unit 280 together with the information indicating the viewing direction may be the average position of (i) the position of a specific object in the right-eye image on the display surface of the display device 20 and (ii) the position of a specific object in the left-eye image on the display surface of the display device 20.

[0034] The determination unit 230 may determine that a particular object is being gazed upon by a person captured in the image if the amount of change in the viewing direction over time is less than a predetermined value, and the amount of change in the display position over time is less than a predetermined value.

[0035] The estimation unit 240 may further use the positional relationship between the display device 20 and the imaging device 40, and the geometric attribute information of the display area of ​​the display device 20, to estimate at least one of the position and orientation of the display device 20.

[0036] The estimation unit 240 may (i) assume a plurality of combinations of the position and orientation of the display device 20, (ii) calculate the distance between the position of the intersection of the viewing direction in three-dimensional space and the display device 20 and the display position of a specific object for each of the plurality of combinations, and (iii) weight the position and orientation of the combination for which the shorter distance was calculated among the plurality of combinations with a larger weight, and use this for estimating the position and orientation of the display device 20.

[0037] The display control unit 250 determines a target area, which is an area to be concealed on the display device 20, based on the arrangement of the display device 20a identified by the estimation unit 240. Based on the determined target area, it conceals a portion of the image to be displayed on the display device 20 and displays it on the display device 20.

[0038] Figure 3 shows a situation in which persons 80a and 80b are looking at a pointer 300 displayed on the display device 20a. The pointer 300 is, for example, a mouse pointer. The position of the pointer 300 can be moved by persons 80 operating a pointing device such as a mouse, touchpad, and trackball. The pointer 300 is an example of a specific object that persons 80 may be looking at. The pointer 300 is an example of a cursor displayed on the display device 20. A cursor is an object displayed to indicate, for example, the position where character input will be performed on the display device 20. The display position information acquisition unit 210 can acquire the position of the pointer 300 from the operating system, at least in pixels.

[0039] The viewing direction information acquisition unit 220 continuously acquires the viewing direction of the person 80 based on images continuously captured by the imaging device 40. The determination unit 230 determines whether the pointer 300 is being gazed at by the person 80 based on the time change of the viewing direction acquired by the viewing direction information acquisition unit 220 and the time change of the display position of the pointer 300 acquired by the display position information acquisition unit 210. For example, the determination unit 230 may determine that the pointer 300 is being gazed at by the person 80 if the amount of time change of the viewing direction acquired by the viewing direction information acquisition unit 220 is smaller than a predetermined value, and the amount of time change of the display position of the pointer 300 is smaller than a predetermined value.

[0040] Furthermore, the determination unit 230 may determine that the pointer 300 is not being gazed at by the person captured in the image if the period during which the amount of time change of the display position of the pointer 300 can be considered zero is longer than a predetermined value. The determination unit 230 may determine that the pointer 300 is being gazed at by the person 80 if the change in the display position of the pointer 300 begins from a state where the amount of time change of the display position of the pointer 300 can be considered zero, and the amount of time change of the viewing direction acquired by the viewing direction information acquisition unit 220 is smaller than a predetermined value. The determination unit 230 may determine that the pointer 300 is being gazed at by the person 80 if the amount of time change of the display position of the pointer 300 decreases from a state where the amount of time change of the display position of the pointer 300 is greater than or equal to a predetermined value to a state where the amount of time change of the display position of the pointer 300 can be considered zero, and the amount of time change of the viewing direction acquired by the viewing direction information acquisition unit 220 is smaller than a predetermined value.

[0041] For example, when person 80 operates a pointing device to move pointer 300 over a button and clicks it, when pointer 300 stops, it is highly likely that person 80 is fixated on the position of pointer 300. At this time, it is highly likely that person 80's line of sight has not substantially changed. Thus, when the amount of change in the line of sight over time is smaller than a predetermined value, and the amount of change in the display position of pointer 300 over time is smaller than a predetermined value, it can be determined that pointer 300 is being fixed on by person 80.

[0042] Furthermore, it is generally believed that when a person views an outdoor sign, they can read up to 15 characters if their gaze remains in one place for a minimum of 0.3 seconds. Therefore, it is considered that the optimal quantization unit in the time axis direction for determining whether the pointer 300 is being gazed at by the person 80 captured in the image is a time interval of approximately 0.25 seconds, and by averaging and aggregating the coordinates of the viewpoint over this time interval, it is possible to determine whether or not the pointer 300 is being gazed at by the person 80. If the time interval for determining whether or not the pointer 300 is being gazed at by the person 80 is too narrow, the influence of noise will increase, making it difficult to determine whether or not the pointer 300 is being gazed at by the person 80. Therefore, it is desirable to set the time interval to approximately 0.25 seconds.

[0043] In the real world, when viewed from a bird's-eye perspective, person 80 and pointer 300 are often aligned almost horizontally. In such cases, the actual positions of monitors and people at eye level (height from the horizontal plane) are often densely packed within a width of about 1-2 meters. Considering that vertical (height) errors in gaze estimation technology tend to be structurally large, quantizing and approximating the height coordinates into meaningful steps is more effective than precisely analyzing them. This makes it easier to identify the degree of synchronization between the height movement of pointer 300 and the gaze, and when this degree of synchronization is high, it can be assumed that person 80 is fixated on pointer 300. Furthermore, if the movement of pointer 300 and the movement of the gaze direction stop, the coincidence between the gaze direction and the pointer can be determined with even higher probability, allowing for a more accurate determination of whether or not person 80 is fixated on pointer 300. For example, if the variation in height is classified by quantizing and convolving the height in 10cm units (however, consideration may be given to not processing viewing direction information outside the vertical range of 1-2m in the viewing direction information acquisition unit 220), the position of the pointer 300 in the height direction will generally be in about three stages of displacement within the height range of a typical display device. This leads to quantization that has meaning in terms of whether the person is looking at the upper, middle, or lower section of the display device 20. For example, in a typical user interface screen of the Windows® OS, looking at the upper section corresponds to operating the upper toolbar of the software interface, looking at the middle section corresponds to editing near the center, and looking at the lower section corresponds to operating the taskbar or writing comments in chat. In this way, by quantizing the position in the height direction to about 10 cm, it becomes easier to capture the timing of meaningful synchronization between the pointer 300 and the gaze (synchronization of the up and down movement of the eyes and the up and down movement of the pointer 300). If the displacement of their movements approaches zero while they are judged to be synchronized (for example, if the displacement of their movements falls below a predetermined reference value), the judgment unit 230 can consider this to be an event action such as a mouse click, i.e., a gaze event, and determine that the pointer 300 is being gazed at by the person 80.

[0044] Figure 4 shows an example of the data structure of gaze information data stored in the memory unit 280. The gaze information data associates "display position," "gaze vector," and "user identification information." The gaze information data is stored in the memory unit 280.

[0045] The "display position" indicates the display position on the display device 20 where the pointer 300, which is determined to be being viewed by the person 80, is located. The "display position" may include, for example, the coordinates in pixels within the display area 10. In a configuration with multiple display devices 20, as in this embodiment, the "display position" may include the coordinates in pixels for each display device 20 and the identification information of the display device. However, if, for example, the display area 10 of multiple display devices 20 is extended, and the display device 20 can be identified by its coordinates in the extended display coordinate system, or if it is possible to identify the display device 20 from its coordinates, then it is not necessary to include the identification information of the display device 20 in the "display position".

[0046] The "line of sight vector" is a vector that originates from the position of the person 80's eyes and is parallel to the direction of the person 80's view. The "line of sight vector" may be a vector with the position of the imaging device 40 as the reference position. The position of the person 80's eyes may be determined from the image of the person 80 included in the image captured by the imaging device 40. The viewing direction information acquisition unit 220 determines the relative positional relationship between the position of the imaging device 40 and the position of the person 80's eyes (for example, the distance in the z-axis direction and the position in a plane parallel to x and y) from the size of the person 80's face in the image of the person 80. The viewing direction information acquisition unit 220 may determine the relative positional relationship between the imaging device 40 and the position of the person 80's eyes by considering the imaging conditions of the imaging device 40 (for example, the focal length).

[0047] When the determination unit 230 determines that the pointer 300 is being gazed upon by the person 80, the storage unit 280 stores a set of information as gaze information data, including the display position of the pointer 300, the gaze vector of the person 80 based on the viewing direction, and user identification information. Here, the person 80 is identified by extracting feature quantities from the image of the person 80's face in the image captured by the imaging device 40, and identification information may be assigned to each person 80. As a result, the storage unit 280 can store information indicating the viewing direction and information indicating the display position of the pointer 300 at multiple timings when it is determined that the pointer 300 is being gazed upon by the person 80.

[0048] Figure 5 is an example of a flowchart of an information processing method performed in the information processing device 30. In S502, gaze information data is stored. Specifically, when the determination unit 230 determines that the pointer is being gazed upon by the person 80, the storage unit 280 stores a set of the pointer's display position, gaze vector, and identification information of the person 80.

[0049] In S504, the estimation unit 240 estimates the number of display devices 20. For example, the estimation unit 240 estimates the number of display devices 20 based on gaze information data. An example of a method for estimating the number of display devices 20 will be schematically explained with reference to Figure 6. Based on the gaze information data, the estimation unit 240 extracts multiple gaze vectors associated with displays at the same display position and calculates the coordinates of the intersection point on the extensions of the extracted gaze vectors (referred to as the "intersection coordinates"), or the coordinates of the position where the extensions of the extracted gaze vectors approach each other within a predetermined distance (referred to as the "proximity point coordinates"). The proximity point coordinates may be, for example, a point on the extension of one of the extracted gaze vectors where the distance from that point to the extension of the other gaze vector is within a predetermined distance. The black circles shown in Figure 6 (for example, black circle 650, etc.) indicate the intersection coordinates and proximity point coordinates.

[0050] The estimation unit 240 estimates the number of display devices 20 by grouping the proximity point coordinates and intersection point coordinates in three-dimensional space. For example, as shown in Figure 6, if the proximity point coordinates and intersection point coordinates can be grouped into three groups (group 600, group 610, and group 620) based on their relative positional relationships, the estimation unit 240 estimates 3 as the number of display devices 20.

[0051] Returning to the flowchart in Figure 5, in S510, a loop regarding the number of display devices is started. The loop regarding the number of display devices started in S510 repeats the processing between S510 and S540. The loop regarding the number of display devices is executed assuming that the number of display devices 20 is between Mm and M+m, where M is the number of display devices 20 estimated in S504. However, if Mm is 0 or less, the loop is executed assuming that the number of display devices 20 is between 1 and M+m. Note that if the number of display devices is known (for example, if it is set in advance), the known number of display devices may be used. In this case, the loop processing regarding the number of display devices from S510 to S540 (or part of said loop processing) can be omitted.

[0052] In S512, the estimation unit 240 sets initial values ​​for the position and orientation of the display device 20. Figures 7, 8, and 9 schematically show examples of initial values ​​for the position and orientation of the display device 20. Note that "position of the display device 20" may refer to the position of the display area 10 of the display device 20, and "orientation of the display device 20" may refer to the orientation of the display area 10 of the display device 20. In Figures 7 to 9, reference numeral 700 indicates the position and orientation of the display device 20 corresponding to group 600, reference numeral 710 indicates the position and orientation of the display device 20 corresponding to group 610, and reference numeral 720 indicates the position and orientation of the display device 20 corresponding to group 620. Note that Figures 8 and 9 show different planes in the y-axis direction.

[0053] The estimation unit 240 may set initial values ​​for the position and orientation of the display device 20 by considering the distribution of proximity point coordinates and intersection point coordinates. The estimation unit 240 may set initial values ​​for the position and orientation of the display device 20 by considering the geometric attribute information of the display device 20 (for example, the actual size of the display area 10, specifically the height and width of the display area 10). The estimation unit 240 may set the initial value for the position of the display device 20 in the z-axis direction to a predetermined distance (for example, 50 cm) from the position of the person 80. The estimation unit 240 may set the initial value for the orientation of the display device 20 to an orientation opposite to any of the gaze vectors included in the gaze information data. The estimation unit 240 may obtain the actual size of the display area 10 of each display device 20, for example, from the operating system running on the information processing device 30.

[0054] The estimation unit 240 may identify the orientation of the face of the person 80 captured in the image from the image captured by the imaging device 40 and set the orientation of the display device 20 to the orientation opposite to the identified orientation. The estimation unit 240 may set the initial orientation of the display device 20 to the orientation opposite to the direction of the person 80's gaze if the amount of change in the person 80's gaze direction over a predetermined value continues for a longer period of time than a predetermined value. The estimation unit 240 may set the initial orientation of the display device 20 to the orientation opposite to the direction of the person 80's gaze if the number of times the person 80 has looked in a particular direction is greater than a predetermined value. The estimation unit 240 may estimate the number of display devices 20 by referring to gaze information data and, based on the gaze vector associated with the identification information of the person 80, grouping the coordinates of positions that the person 80 has looked at more than a predetermined number of times based on the gaze vector associated with the identification information of a specific person 80 (for example, person 80a), and set the initial position and orientation of the display device 20.

[0055] Returning to the flowchart in Figure 5, in S514, multiple combinations of position and orientation are set as candidates for the position and orientation of the display device 20. Figure 10 schematically shows an example of setting multiple combinations of position and orientation of the display device 20. Figure 10 schematically shows an example of setting multiple combinations corresponding to the initial values ​​of position and orientation of the display device 20 indicated by reference numeral 720. To avoid redundancy in the explanation, the example of setting multiple combinations corresponding to reference numeral 720 is explained here, and detailed explanations of the example of setting multiple combinations corresponding to reference numerals 700 and 710 are omitted.

[0056] As shown in Figure 10, 32 positions are set as the position of the display device 20 around the initial value of the position of the display device 20 indicated by reference numeral 720. The 32 positions are arranged in a fan shape around a representative base point 1010 corresponding to the position of the head of the person 80. Three types of orientations are set for each of the 32 positions. For example, as indicated by reference numeral 1020, a first orientation 1021, a second orientation 1022, and a third orientation 1023 are set. The estimation unit 240 may set more positions for the display device 20 the closer it is to reference numeral 720.

[0057] Returning to the flowchart in Figure 5, in S520, a loop is started for multiple combinations of position and orientation. The loop started in S520 repeats the processing between S520 and S530. The loop for combinations is executed assuming that the position and orientation of the display device 20 are each of the multiple combinations set in S514.

[0058] The processes in S522, S524, and S526 will be explained with reference to Figures 11 and 12 as appropriate. In S522, the estimation unit 240 calculates the intersection point between the line of sight vector in three-dimensional space and the display area 10 of the display device 20. For example, as shown in Figures 11 and 12, the estimation unit 240 calculates the intersection point 1140 between the display area 1110 of the display device 20, which is assumed to be facing the first orientation 1021, and the extension of the line of sight vector 1120 included in the gaze information data and the display area 1110.

[0059] In S524, the estimation unit 240 calculates the display position of the pointer on the display area 10 in three-dimensional space. For example, as shown in Figures 11 and 12, the estimation unit 240 converts the position indicated by the coordinates of the display position included in the gaze information data into a display position 1150 on the display area 1110, based on the position and orientation of the display area 1110.

[0060] In S526, the estimation unit 240 calculates the distance D between the intersection point of the line of sight vector and the display area 10 of the display device 20 and the display position of the pointer on the display area 10 in three-dimensional space. For example, as shown in Figures 11 and 12, the estimation unit 240 calculates the distance D between the intersection point 1140 and the display position 1150.

[0061] The estimation unit 240 performs the processes in S522, S524, S526, and S528 for each record of the attention information data.

[0062] In S528, the estimation unit 240 calculates a first evaluation value for a combination of position and orientation. The first evaluation value is an example of a "weight". Here, the height of the first evaluation value indicates the level of confidence that the position and orientation shown by the current combination in this loop processing can be trusted as the position and orientation of the display device 20. The estimation unit 240 may calculate a higher first evaluation value the shorter the distance D. The estimation unit 240 may calculate a higher evaluation value for each of the multiple records included in the gaze information data the shorter the distance D, and the sum of the evaluation values ​​calculated for each of the multiple records may be used as the first evaluation value. As a result, a high first evaluation value is calculated for combinations in which there are many records with a high degree of agreement between the pointer display position and the viewing direction position. As will be described later, position and orientation combinations with a high first evaluation value are used with a high weight to estimate the position and orientation of the display device 20.

[0063] The estimation unit 240 may select records from among multiple records included in the gaze information data in which the distance D is less than a predetermined distance, calculate a shift vector between the intersection point of the extension of the gaze vector and the display area and the display position of the pointer for the selected records, and calculate the average value of the shift vectors calculated for the selected records. The "predetermined distance" may be a distance of about half the size (e.g., width) of the display area 10 of the display device 20. The estimation unit 240 may use the average value of the shift vector to correct the position of the display device 20. For example, the estimation unit 240 may determine the position of the display device 20 in the current combination as a correction vector which is a vector in the opposite direction of the average value of the correction vector, and determine the position of the display device 20 shifted using this correction vector as the corrected position in the combination of display devices 20.

[0064] In S532, the estimation unit 240 clusters multiple combinations of the position and orientation of the display device 20 to the number of display devices in the current loop. For example, the estimation unit 240 may extract multiple combinations for which a first evaluation value higher than a predetermined value is calculated, and cluster the extracted multiple combinations. Figures 13 and 14 show examples of clustering results for position and orientation combinations. By clustering multiple combinations of the position and orientation of the display device 20, the position and orientation combinations are clustered into cluster 1300, cluster 1310, and cluster 1320.

[0065] In S534, the estimation unit 240 calculates the position and orientation of the display device 20 by calculating the average values ​​of the position and orientation of the display device 20 clustered in each cluster. For example, the estimation unit 240 may determine the average value of the position of the display device 20 clustered in cluster 1300 as the estimated position of the display device 20 in cluster 1300. Alternatively, the estimation unit 240 may determine the average value of the orientation of the display device 20 clustered in cluster 1300 as the estimated orientation of the display device 20 in cluster 1300. Similarly, the average value of the position of the display device 20 clustered in cluster 1310 as shown in Figure 14 may be determined as the estimated position of the display device 20 in cluster 1310, and the average value of the orientation of the display device 20 clustered in cluster 1310 may be determined as the estimated orientation of the display device 20 in cluster 1310. Similarly, the average of the positions of the display devices 20 clustered into cluster 1320 shown in Figure 13 may be determined as the estimated position of the display devices 20 in cluster 1320, and the average of the orientations of the display devices 20 clustered into cluster 1320 may be determined as the estimated orientation of the display devices 20 in cluster 1320.

[0066] In S536, the estimation unit 240 calculates a second evaluation value for the position and orientation of the display device 20 calculated in S534. For example, the estimation unit 240 calculates the second evaluation value by performing the same processing as in S522, S534, S526, and S528. Similar to the first evaluation value, the height of the second evaluation value can indicate the level of confidence that the current number of display devices can be trusted as the position and orientation of the display device 20. The estimation unit 240 may calculate a higher second evaluation value as the distance D decreases. The estimation unit 240 may calculate a higher evaluation value as the distance D decreases for each of the multiple records included in the gaze information data, and the sum of the evaluation values ​​calculated for each of the multiple records may be used as the second evaluation value.

[0067] Once the loop processing related to the number of display devices in S510 to S540 is completed, the estimated position and orientation of the display device 20 is determined in S542. For example, the estimation unit 240 determines the estimated position and orientation of the display device 20 based on the second evaluation value calculated in S536. Specifically, the estimation unit 240 determines the number of display devices corresponding to the loop in which the largest second evaluation value among the second evaluation values ​​calculated in each loop processing is calculated as the number of display devices 20, and determines the position and orientation of the display device 20 calculated in S534 in the loop in which the largest second evaluation value was calculated as the estimated position and orientation of the display device 20. In this way, the estimation unit 240 can estimate the position and orientation of the display device 20 using the viewing direction of the person 80 and the display position of the pointer when the person 80 is looking in that direction.

[0068] Figure 15 schematically shows an example of the concealment process performed by the display control unit 250. The information processing device 30 has a predetermined range 110 set, which the imaging device 40 can monitor to see whether or not another person is present in a position where the display areas of the display devices 20b and 20c are visible, provided that the display devices 20b and 20c are arranged in a predetermined positional relationship. In other words, if the display devices 20b and 20c are arranged in a predetermined and correct positional relationship, the imaging device 40 can monitor whether or not another person is present in a position where the range 110 is visible, so there is no need to perform concealment processing on the image within the display area 10 within the range 110.

[0069] However, if the display devices 20b and 20c are not arranged in a predetermined regular positional relationship, the imaging device 40 may not be able to monitor whether or not another person is present in a position where the range 110 is visible. Therefore, the display control unit 250 determines the area in the display area 10b that should be concealed based on the position and orientation of the display area 10b identified by the estimation unit 240, and based on the determined area, conceals at least a portion of the image to be displayed on the display device 20b and displays it on the display device 20b. As an example, if the angle between the normal direction of the display area 10b of the display device 20b and the imaging direction of the imaging device 40 exceeds a predetermined value, the display control unit 250 makes the entire image area of ​​the display area 10b of the display device 20b subject to concealment processing, as shown in Figure 15. This makes it possible to suppress information leakage, especially through the display devices 20b and 20c, which can be freely arranged.

[0070] The area subject to the concealment process does not have to be the entire display area 10, but may be a part of the display area 10. For example, the area subject to the concealment process may be the area outside the range 110 of the entire image area of ​​the display area 10. The area subject to the concealment process may include the area outside the range 110 and a part of the range 110 of the entire image area of ​​the display area 10. The part of the range 110 that may be subject to the concealment process may be an area near the boundary of the range 110. Furthermore, the concealment may be limited to one display device 20, or it may be performed on multiple display devices 20, or different concealment processes may be applied to each display device 20.

[0071] In Figure 15, the display control unit 250 determines that areas 1510 and 1520, which are outside the range 110 of the display area 10c, are areas to be concealed, based on the position and orientation of the display device 20c estimated by the estimation unit 240. The display control unit 250 may increase the intensity of the concealment process for areas that are farther from the position where the imaging device 40 is installed when performing the concealment process. Examples of the intensity of the concealment process include the degree of mixing with other images, the degree of superimposition of other images, the transmittance of the superimposed images, the display size of the presented information (e.g., the size of the characters), the contrast of the presented information, the colors of the presented information and the background, and the brightness of the screen or a part of the screen. Examples of the degree of mixing with other images include the fineness of the mosaic and the strength of the blur.

[0072] Figures 8 to 14 show examples where the display area 10 is flat, but the method can also be applied to non-planar display areas (for example, curved display areas). The estimation unit 240 may identify information indicating whether the display surface of the display device is curved, and information indicating the degree of curvature of the display area (for example, curvature information, etc.) based on the display device model information held by the operating system, and estimate the position and orientation of the display device 20 based on the identified information.

[0073] In the above, a pointer was used as an example of a "specific object," but a "specific object" can be any object that a person can focus on. A "specific object" may be a warning object for the user of the information processing device 30. A "specific object" may be a region of interest (e.g., ROI) in a video such as a movie. The location of a "specific object" may be obtained from the operating system, or it may be identified by analyzing the image displayed on the display device 20. For example, the location of a "specific object" may be the position with the highest score calculated by a predetermined method for calculating the score of a specific object from among (i) the centroid of a region in which the density of feature points extracted by feature point analysis is higher than a reference value and the density of said feature points is increasing or growing, (ii) the position of a face larger than a reference value extracted by face detection technology, (iii) the centroid of a region in which the color change is greater than a reference value, or (iv) a specific position on the display surface (e.g., the center of the screen).

[0074] In the above, an embodiment was described in which the estimation unit 240 of the information processing device 30 estimates the position and direction of the display device 20 using the viewing direction of the person 80 captured by the imaging device 40. However, the estimation unit 240 may estimate at least one of the position and direction of the display device 20 using the viewing direction of the person 80 captured by the imaging device 40. For example, the estimation unit 240 may estimate the relative positions of the display devices 20 by referring to setting information and estimate the direction of the display devices 20 using the viewing direction of the person 80 captured by the imaging device 40. For example, the estimation unit 240 may estimate the relative positions of the display devices 20 (for example, the positional relationship in the x-axis and y-axis directions) by referring to the arrangement information of the extended display set in advance in the operating system, and estimate that the position of the display area 10 in the z-axis direction is a predetermined distance (for example, 50 cm) away from the person 80. As another example, the estimation unit 240 may estimate the orientation of the display device 20 to be a predetermined orientation (for example, the same orientation as the imaging direction of the imaging device 40), and estimate the position of the display device 20 using the viewing direction of the person 80 captured by the imaging device 40.

[0075] The above describes a configuration in which there are three display devices 20. However, the number of display devices 20 is not limited to three. Other configurations include a configuration with one display device 20, a configuration with two display devices 20, and a configuration with three or more display devices 20.

[0076] The above describes an configuration in which the information processing device 30 is equipped with an imaging device 40. However, a configuration in which the imaging device 40 is not provided in the information processing device 30 can also be adopted. In this case, it is desirable that the positional relationship between the imaging device and the display device 20a, and the imaging direction of the imaging device, are known or can be estimated by some means (e.g., communication means). Furthermore, the number of imaging devices is not limited to one, and a configuration in which multiple imaging devices are equipped can also be adopted. When multiple imaging devices are equipped, it is desirable that the relative positional relationships of the imaging devices and the imaging direction of each imaging device are known or can be estimated by some means (e.g., communication means).

[0077] Some of the processing in the processing unit 200 may be implemented using machine learning.

[0078] According to the information processing system 100 described above, the position and orientation of the display device 20 can be estimated using the viewing direction of the person 80 captured by the imaging device 40. Therefore, the information processing device 30 can perform various display controls based on at least one of the estimated position and orientation of the display device 20. For example, the display content of the display device 20 can be controlled to prevent information leakage through the display device 20.

[0079] Figure 16 shows an example of a computer 2000 in which multiple embodiments of the present invention may be embodied in whole or in part. A program installed on the computer 2000 can cause the computer 2000 to function as an information processing system 100 or parts of said system, or a device such as an information processing device 30 or parts of said device, to perform operations associated with said system or parts of said system or devices or parts of said device, and / or to perform processes or stages of said processes according to the embodiment. Such a program may be executed by the CPU 2012 to cause the computer 2000 to perform specific operations associated with some or all of the processing procedures and blocks of the block diagram described herein.

[0080] The computer 2000 according to this embodiment includes a CPU 2012 and RAM 2014, which are interconnected by a host controller 2010. The computer 2000 also includes a ROM 2026, flash memory 2024, communication interface 2022, and input / output chip 2040. The ROM 2026, flash memory 2024, communication interface 2022, and input / output chip 2040 are connected to the host controller 2010 via an input / output controller 2020.

[0081] CPU2012 operates according to the programs stored in ROM2026 and RAM2014, thereby controlling each unit.

[0082] The communication interface 2022 communicates with other electronic devices via a network. The flash memory 2024 stores programs and data used by the CPU 2012 in the computer 2000. The ROM 2026 stores boot programs and / or hardware-dependent programs of the computer 2000, such as those executed by the computer 2000 upon activation. The input / output chip 2040 may also connect various input / output units, such as keyboards, mice, and monitors, to the input / output controller 2020 via input / output ports such as serial ports, parallel ports, keyboard ports, mouse ports, monitor ports, USB ports, and HDMI® ports.

[0083] The program is provided via a computer-readable storage medium such as a CD-ROM, DVD-ROM, or memory card, or via a network. RAM2014, ROM2026, or flash memory 2024 are examples of computer-readable storage media. The program is installed in flash memory 2024, RAM2014, or ROM2026 and executed by CPU2012. The information processing described within these programs is read by computer 2000, resulting in coordination between the program and the various types of hardware resources described above. The apparatus or method may be configured to realize the operation or processing of information in accordance with the use of computer 2000.

[0084] For example, when communication is performed between computer 2000 and an external device, CPU 2012 may execute a communication program loaded into RAM 2014 and, based on the processing described in the communication program, instruct the communication interface 2022 to perform communication processing. Under the control of CPU 2012, the communication interface 2022 reads the transmission data stored in the transmit buffer processing area provided in the recording medium such as RAM 2014 and flash memory 2024, sends the read transmission data to the network, and writes the received data received from the network to the receive buffer processing area provided on the recording medium.

[0085] Furthermore, CPU2012 may read all or necessary parts of a file or database stored on a recording medium such as flash memory 2024 into RAM2014, and perform various types of processing on the data in RAM2014. CPU2012 then writes the processed data back to the recording medium.

[0086] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and subjected to information processing. The CPU2012 may perform various types of processing on the data read from RAM2014, including various types of operations, information processing, conditional judgments, conditional branching, unconditional branching, information retrieval / replacement, etc., as described herein and specified by the program's instruction sequence, and write the results back to RAM2014. The CPU2012 may also retrieve information in files, databases, etc., within the recording medium. For example, if multiple entries are stored in the recording medium, each having an attribute value of a first attribute associated with an attribute value of a second attribute, the CPU2012 may search among the multiple entries for an entry that matches the condition where the attribute value of the first attribute is specified, read the attribute value of the second attribute stored in that entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies the predetermined condition.

[0087] The programs or software modules described above may be stored on or near computer-readable storage media on computer 2000. Recording media such as hard disks or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as computer-readable storage media. Programs stored on computer-readable storage media may be provided to computer 2000 via the network.

[0088] A program installed on computer 2000, which causes computer 2000 to function as an information processing device 30, may, when executed by the computer, interact with the CPU 2012, etc., to cause computer 2000 to function as each part of the information processing device 30. The information processing described in these programs is read by computer 2000 and functions as each part of the information processing device 30, which is a concrete means of cooperation between software and the various hardware resources described above. Then, by realizing the calculation or processing of information according to the purpose of use of computer 2000 in this embodiment, a specific information processing device 30 suited to the purpose of use is constructed.

[0089] Various embodiments have been described with reference to block diagrams, etc. In a block diagram, each block may represent (1) a stage in a process in which an operation is performed, or (2) a part of a device that has the role of performing an operation. A particular stage and part may be implemented by a dedicated circuit, a programmable circuit supplied with computer-readable instructions stored on a computer-readable storage medium, and / or a processor supplied with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuit may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuit may include reconfigurable hardware circuits, including logic AND, logic OR, logic XOR, logic NAND, logic NOR, and other logic operations, flip-flops, registers, memory elements such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc.

[0090] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, and as a result, a computer-readable storage medium having instructions stored therein constitutes at least part of a product containing instructions that can be executed to provide a means for performing an operation specified in a processing procedure or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital multipurpose disc (DVD), Blu-ray® disc, memory stick, integrated circuit card, etc.

[0091] Computer-readable instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, Java®, C++, and traditional procedural programming languages ​​such as the C programming language or similar programming languages.

[0092] Computer-readable instructions may be provided to the processor or programmable circuit of a programmable data processing device locally or via a wide area network (WAN) such as a local area network (LAN) or the Internet, and may be executed to provide a means for performing the described processing procedure or the operation specified in the block diagram.

[0093] Here, "computer" can refer to a personal computer (PC), tablet computer, smartphone, workstation, server computer, or general-purpose computer, and may also refer to a computer system in which multiple computers are connected. Such a computer system in which multiple computers are connected is also called a distributed computing system and is a computer in a broad sense. In a distributed computing system, each of the multiple computers executes a part of the program, and the multiple computers execute the program collectively by passing data from the computers during program execution as needed.

[0094] Examples of processors include computer processors, central processing units (CPUs), processing units, microprocessors, digital signal processors, controllers, and microcontrollers. A computer may have one or more processors. In a multiprocessor system with multiple processors, each processor executes a portion of the program, and the processors collectively execute the program by passing program execution data between them as needed. For example, in the execution of multitasking, each of the multiple processors may execute a portion of each task in small chunks by switching tasks at each time slice. In this case, which part of a program each processor executes changes dynamically. Alternatively, which part of a program each of the multiple processors executes may be statically determined by multiprocessor-aware programming.

[0095] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. It will be clear from the claims that such modified or improved forms may also be included in the technical scope of the present invention.

[0096] It should be noted that the execution order of operations, procedures, steps, and stages in the apparatus, systems, programs, and methods shown in the claims, specifications, and drawings is not explicitly stated as "before," "prior to," etc., and that these can be implemented in any order unless the output of a previous process is used in a later process. Even if the operation flow in the claims, specifications, and drawings is described using phrases such as "first," "next," etc. for convenience, it does not mean that it is essential to perform the operations in that order. [Explanation of Symbols]

[0097] 10 Display area 20 Display device 30 Information Processing Devices 40 Imaging device 80 people 100 Information Processing Systems 110 range 200 Processing Units 210 Display position information acquisition unit 220 Viewing direction information acquisition unit 230 Judgment Department 240 Estimation part 250 Display Control Unit 280 Storage section 300 pointers Groups 600, 610, and 620 650 Black Circle 700, 710, 720, 1020 code 1110 Display area 1120 Line of sight vector 1140 intersection 1150 Display position 1300 clusters 1310 clusters 1320 clusters 1510 area 1520 areas 2000 Computer 2010 Host Controller 2012 CPU 2014 RAM 2020 Input / Output Controller 2022 Communication Interface 2024 Flash Memory 2026 ROM 2040 Input / Output Chip

Claims

[Claim 1] A viewing direction information acquisition unit acquires information indicating the viewing direction of a person, obtained from an image of a person captured by an imaging device that images the area around one or more display devices, A display position information acquisition unit acquires information indicating the display position, which is the position where one or more display devices display a specific object that can be viewed by a person at a timing corresponding to the timing when the image was captured. An estimation unit that estimates at least one of the position and orientation of the one or more display devices using the information indicating the viewing direction and the information indicating the display position, A determination unit that determines whether the specific object is being gazed upon by the person captured in the image, based on the time change in the viewing direction and the time change in the display position, A storage unit that stores information indicating the viewing direction and information indicating the display position at multiple timings in which the determination unit determines that the specific object is being gazed upon by the person captured in the image. Equipped with, The estimation unit estimates at least one of the position and orientation of the one or more display devices using the information indicating the viewing direction and the information indicating the display position at the multiple timings stored in the storage unit. The aforementioned specific object is a cursor displayed on one or more of the aforementioned display devices, The determination unit determines that the specific object is being gazed upon by the person captured in the image if the amount of change in the viewing direction over time is less than a predetermined value, and the amount of change in the display position over time is less than a predetermined value. Information processing device.

Images