Information processing device, information processing method, and computer-readable recording medium

Abstract

An information processing device according to one embodiment of the present technology comprises an acquisition unit, a determination unit, and a control unit. The acquisition unit acquires: a viewpoint position of a subject captured by using a screen for chromakey composition, provided in a real space; a position of interest to be viewed by the subject; and virtual space information indicating the position of the screen in a virtual three-dimensional space. The determination unit determines a virtual intersection between the screen indicated by the virtual space information, and a virtual line connecting the viewpoint position of the subject and the position of interest in the virtual three-dimensional space. The control unit controls the irradiation direction of an irradiation device, which is provided in the real space, so as to point a guide point on the screen corresponding to the virtual intersection in the real space.

Description

Information processing device, information processing method, and computer-readable recording medium

[0001] The present technology relates to an information processing device, an information processing method, and a computer-readable recording medium used for guiding the gaze of a subject, etc.

[0002] Patent Document 1 describes a display imaging device in which a transparent display is disposed between a camera and a user, and a video call system using the device. In this system, a user looking at a display showing the other party is photographed by a camera behind the display. This allows the other party to see an image of the user facing forward, allowing the user and the other party to align their gazes (see paragraphs

[0012] ,

[0060] ,

[0061] , Figures 1 and 7, etc., of the specification of Patent Document 1).

[0003] International Publication No. 2019 / 207922

[0004] In recent years, technology that synthesizes live-action footage of a subject and recreates it in a virtual space has been attracting attention. When shooting footage for compositing, a chromakey screen is used, but such screens generally have few features, and the subject may not know where to focus during shooting. For example, if a display or other device is used to guide the subject's line of sight, the display or other device may be reflected in the video of the subject, making it difficult to properly extract the subject's image.

[0005] In view of the above circumstances, an object of the present technology is to provide an information processing device, an information processing method, and a computer-readable recording medium that are capable of guiding the subject's gaze without reducing the subject image extraction performance.

[0006] To achieve the above object, an information processing device according to one embodiment of the present technology includes an acquisition unit, a determination unit, and a control unit. The acquisition unit acquires a viewpoint position of a subject photographed using a screen for chromakey compositing installed in real space, a focus position at which the subject should look, and virtual space information representing the position of the screen in a virtual three-dimensional space. The determination unit determines a virtual intersection between a virtual line connecting the viewpoint position and the focus position of the subject in the virtual three-dimensional space and the screen represented by the virtual space information. The control unit controls the irradiation direction of an irradiation device installed in the real space so as to point to a guidance point on the screen corresponding to the virtual intersection in the real space.

[0007] This information processing device determines the virtual intersection point where a virtual line connecting the subject's viewpoint and the focal point where the subject should look intersects with a screen for chromakey compositing. An illumination device then points to an induction point on the actual screen that corresponds to the virtual intersection point. Pointing to the induction point makes it possible to present the direction in which the subject should look. This makes it possible to guide the subject's line of sight without reducing the subject image extraction performance.

[0008] The focus position may be a viewpoint position of a viewer of a display device that displays an image of the subject extracted from an image of the subject captured against the screen.

[0009] The acquisition unit may acquire the viewpoint position of the subject based on images of the subject captured by a plurality of cameras installed in the real space such that the screen is a background of a capture range.

[0010] The acquisition unit may generate the virtual space information based on images captured by the plurality of cameras.

[0011] The acquisition unit may capture images of spots projected onto the screen by the irradiation device using the plurality of cameras, and calculate three-dimensional coordinates of the spots based on the images captured by the plurality of cameras.

[0012] The control unit may calculate the irradiation direction based on a position of the irradiation device and a position of the induction point in the real space.

[0013] The control unit may calculate a rotation angle of the irradiation direction relative to a reference irradiation direction that serves as a reference for the irradiation direction in the irradiation device.

[0014] The control unit may calibrate the position of the illumination device and the reference illumination direction based on a rotation angle of the illumination direction when the illumination device projects a spot onto each of the multiple cameras and installation positions of the multiple cameras.

[0015] The irradiation device may have a position detection marker. In this case, the control unit may calculate the position of the irradiation device and the reference irradiation direction based on images of the position detection marker of the irradiation device captured by the multiple cameras.

[0016] The control unit may determine whether an illumination path connecting the position of the illumination device and the induction point in the real space passes through a non-illuminated area where pointing of the induction point should be avoided, and if it is determined that the illumination path passes through the non-illuminated area, may stop pointing of the induction point.

[0017] The illumination device may be a laser pointer that is rotatable in elevation and azimuth.

[0018] The irradiation device may be a laser beam scanning projector that scans a laser beam using a movable mirror. In this case, the control unit may control the projector to project a pattern including at least one of a character and an image onto the guidance point.

[0019] The screen may be a full-circle screen that surrounds the entire periphery of a shooting space in which the subject is shot. In this case, the control unit may control the irradiation direction of the irradiation device so as to point the guidance point on a region of the full-circle screen that is forward as viewed from the subject.

[0020] The focus position may be a shooting position of a virtual camera set in the virtual three-dimensional space.

[0021] The attention position may be a position of an attention target that the subject should look at in a background that is combined with an image of the subject extracted from an image of the subject.

[0022] An information processing method according to one embodiment of the present technology is an information processing method executed by a computer system, and includes acquiring a viewpoint position of a subject photographed using a screen for chromakey compositing provided in a real space, a focus position to be viewed by the subject, and virtual space information representing the position of the screen in a virtual three-dimensional space. A virtual intersection between a virtual line connecting the viewpoint position of the subject and the focus position in the virtual three-dimensional space and the screen represented by the virtual space information is determined. An irradiation direction of an irradiation device provided in the real space is controlled so as to point to a guidance point on the screen corresponding to the virtual intersection in the real space.

[0023] A computer-readable recording medium according to one embodiment of the present technology stores a program for causing a computer system to execute the following steps: a step of acquiring a viewpoint position of a subject to be photographed using a screen for chromakey compositing provided in real space, a focus position at which the subject should look, and virtual space information representing the position of the screen in a virtual three-dimensional space; a step of determining a virtual intersection between a virtual line connecting the viewpoint position and the focus position of the subject in the virtual three-dimensional space and the screen represented by the virtual space information; and a step of controlling an irradiation direction of an irradiation device provided in the real space so as to point to a guidance point on the screen that corresponds to the virtual intersection in the real space.

[0024] FIG. 1 is a schematic diagram showing an example of the configuration of a telepresence system according to the present embodiment. FIG. 1 is a schematic diagram showing an example of the configuration of an imaging system. FIG. 2 is a schematic diagram showing an example of the configuration of an illumination device. FIG. 3 is a block diagram showing an example of the functional configuration of a gaze guidance controller. FIG. 2 is a schematic diagram illustrating a basic operation of the gaze guidance system. FIG. 3 is a flowchart illustrating an example of the operation of the gaze guidance system. FIG. 4 is a flowchart illustrating an example of three-dimensional space measurement processing. FIG. 4 is a schematic diagram illustrating an example of three-dimensional space measurement processing. FIG. 5 is a flowchart illustrating an example of subject viewpoint measurement processing. FIG. 5 is a schematic diagram illustrating an example of coordinate conversion processing. FIG. 6 is a schematic diagram illustrating an example of pointing operation by the illumination device. FIG. 7 is a schematic diagram illustrating another example of pointing operation by the illumination device. FIG. 7 is a schematic diagram illustrating a method of calibrating the illumination device. FIG. 8 is a schematic diagram illustrating another method of calibrating the illumination device. FIG. 8 is a schematic diagram showing another example of the configuration of the illumination device. FIG. 9 is a schematic diagram illustrating another application example of the gaze guidance system. FIG. 9 is a schematic diagram illustrating another application example of the gaze guidance system.

[0025] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0026] [Outline of Telepresence System] Fig. 1 is a schematic diagram showing an example of the configuration of a telepresence system according to this embodiment. The telepresence system 100 is a system that displays video of a user 1 captured at a video capture site at a viewing site in real time. In other words, the telepresence system 100 functions as a live distribution system that provides live video of user 1 captured at a video capture site to user 2 at a viewing site.

[0027] Hereinafter, user 1 who is photographed at the filming location will be referred to as subject 1, and user 2 who watches the video at the viewing location will be referred to as viewer 2. Furthermore, the video showing subject 1 displayed at the viewing location will be referred to as content video 5. In this embodiment, content video 5 is live video that displays the state of subject 1 in real time.

[0028] An example configuration of a telepresence system 100 at a filming location is schematically illustrated in the lower part of Fig. 1, and an example configuration of a telepresence system 100 at a viewing location is schematically illustrated in the upper part of Fig. 1. The telepresence system 100 includes a filming system 10, a display system 20, and a visual guidance system 30.

[0029] The imaging system 10 is provided at a location on the imaging side, and is a system that images a subject 1 and generates content data of the subject 1. The content data is data for constructing a content video 5 that is displayed at a location on the viewing side. The display system 20 is provided at the location on the viewing side, and is a system that displays the content video 5 based on the content data generated by the imaging system 10.

[0030] The line-of-sight guidance system 30 is installed at the base of the filming side and is a system that guides the line of sight L1 of the subject 1 while the subject 1 is being filmed. The line of sight L1 of the subject 1 is controlled so that the line of sight L1 of the subject 1 displayed as the content video 5 is directed toward the viewpoint P2 of the viewer 2. By guiding the line of sight L1 of the subject 1 in this way, it is possible to align the line of sight between the viewer 2 and the subject 1. This makes it possible to provide the viewer 2 with a natural video experience in which the line of sight between the viewer 2 and the subject 1 is aligned.

[0031] [Photography System] As shown in the lower part of Fig. 1, the photography system 10 includes a green screen 11, a photography unit 12, and a photography-side controller 14. The green screen 11 and the photography unit 12 are photography equipment used in real space. Here, real space refers to a three-dimensional space that constitutes the real world. A photography studio using these pieces of equipment is configured at the photography-side base (real space). The photography-side controller 14 is, for example, a computer installed at the photography-side base, but may also be, for example, a server device installed at a location separate from the photography studio.

[0032] The green screen 11 is a screen for chromakey compositing that is installed in real space. The green screen 11 has a green surface (screen surface) also called a green back, and functions as a background for capturing images for chromakey compositing. In addition to the green screen 11, any other capture screen that can be used for chromakey compositing, such as a blue screen, may be used as appropriate. Note that in FIG. 1 , the green screen 11 for chromakey compositing that is located behind the subject 1 is not shown.

[0033] Chroma key compositing is a technique for separating, for example, a homogeneous color area from a photographed image and compositing another image into that separated area. By using this technique, it is possible to separate the green background from an image of subject 1 photographed against a green screen 11, for example, and extract the foreground subject 1. This image in which the foreground subject 1 is extracted from the background area is referred to as subject image 3.

[0034] The photographing unit 12 has a plurality of cameras 13 that photograph the subject 1. As the plurality of cameras 13, for example, any digital cameras equipped with image sensors such as CMOS (Complementary Metal-Oxide Semiconductor) sensors or CCD (Charge Coupled Device) sensors are used.

[0035] The multiple cameras 13 capture images of the subject 1. In this disclosure, the term "image" includes both moving images (video) that change over time and still images that do not change over time. The following mainly describes a case where the multiple cameras 13 capture video of the subject 1, but the present technology can also be applied to a case where a still image of the subject 1 is captured.

[0036] The multiple cameras 13 are installed in real space so that the green screen 11 forms the background of the shooting range. Here, the shooting range refers to, for example, the range of the angle of view of the cameras 13. Since the image of the subject 1 shot by the multiple cameras 13 has the green screen 11 as the background, it becomes an image from which the subject image 3 can be extracted with high accuracy. Furthermore, the multiple cameras 13 are positioned so as to shoot the subject 1 from different directions. This makes it possible to shoot the subject image 3 as seen from various directions.

[0037] The shooting-side controller 14 extracts a subject image 3 from an image of the subject 1 captured by the shooting unit 12 (multiple cameras 13), and generates content data using the subject image 3. For example, a 3D model 6 of the subject 1 can be constructed using the subject image 3. In this case, the data of the 3D model 6 becomes the content data. Furthermore, an image showing only the subject image 3 (extracted image), or an image in which the subject image 3 is combined with an arbitrary background (composite image) may be generated as content data. The content data generated by the shooting-side controller 14 is transmitted to a viewing location via a predetermined network using a protocol such as UDP (User Datagram Protocol).

[0038] 1, the display system 20 includes a display device 21, a viewpoint sensor 22, and a viewer-side controller 23. The display device 21 and the viewpoint sensor 22 are display devices used in real space. The viewer-side controller 23 is, for example, a computer installed at the viewer's base, but may also be, for example, a server device installed at another location.

[0039] Display device 21 is a device that displays content video 5. As described above, content video 5 is an image generated from subject image 3, and the image includes subject image 3. In other words, display device 21 can also be said to be a device that displays subject image 3 extracted from an image of subject 1 captured with green screen 11 as the background.

[0040] The display device 21 may be, for example, a display device having a plurality of LEDs (Light Emitting Diodes) arranged on a display surface (so-called LED Wall). The display device 21 shown in Fig. 1 has display surfaces using LEDs on both the floor surface and a vertical surface arranged perpendicular to the floor surface. For example, by combining an image display on the vertical surface and an image display on the floor surface, it is possible to realize a display in which the subject 1 appears to be protruding from the vertical surface or the floor surface.

[0041] The configuration of the display device 21 is not limited, and a display equipped with display elements such as a liquid crystal panel or an organic EL panel may be used. The display device 21 may be a display that displays the same image to both eyes of the viewer 2, or a display that is capable of stereoscopic display that displays images with parallax independently to the right and left eyes of the viewer 2, allowing the viewer 2 to perceive a three-dimensional image. The display device 21 may also be an HMD (Head Mounted Display) that is worn on the head of the viewer 2.

[0042] The viewpoint sensor 22 is a sensor that detects the viewpoint P2 of the viewer 2. Specifically, the viewpoint sensor 22 is a sensor that can detect the three-dimensional position of the viewpoint P2 of the viewer 2 in real space. The processing of the viewpoint detection P2 may be performed by a computing device included in the viewpoint sensor 22, or may be performed by the viewer-side controller 23, which will be described later.

[0043] In the example shown in FIG. 1 , the viewer 2 watches the display device 21 while wearing eyeglasses 26 equipped with markers 25 for position detection. In this case, the position of the eyeglasses 26 is detected as the viewer 2's viewpoint P2. The markers 25 provided on the eyeglasses 26 are, for example, reflective markers that reflect infrared light. In this case, the viewpoint sensor 22 is an infrared camera or the like that irradiates an imaging range with infrared light and detects the reflected infrared light. For example, the markers 25 are detected from an image of the viewer 2 captured by the viewpoint sensor 22, and the position and orientation of the eyeglasses 26 are calculated from the positional relationship of the markers 25. This makes it possible to detect not only the position of the viewer 2's viewpoint P2, but also the direction of the viewer 2's line of sight L2.

[0044] The type of viewpoint sensor 22 is not limited, and a sensor capable of detecting depth, such as a ToF camera or a stereo camera, may be used. In this case, the face of the viewer 2 is detected from an image captured by the camera, and the three-dimensional position of the viewpoint P2 of the viewer 2 is calculated. Any other sensor capable of detecting the three-dimensional position of the viewpoint P2 may be used. Furthermore, when an HMD is used as the display device 21, the three-dimensional position of the HMD within the viewer's base may be calculated as the viewpoint P2 of the viewer 2.

[0045] The viewer-side controller 23 receives content data generated at the shooting site via a predetermined network. Based on the content data, the viewer-side controller 23 generates a content video 5 to be displayed on the display device 21 and outputs it to the display device 21. In this embodiment, the content video 5 is generated according to the viewpoint P2 of the viewer 2 detected using the viewpoint sensor 22. This makes it possible to generate a content video 5 in which the appearance of the subject image 3 changes in accordance with the movement of the viewpoint P2 of the viewer 2, for example.

[0046] 2 is a schematic diagram showing an example of the configuration of the photography system 10. In the photography system 10 (photography studio), a green screen 11 and multiple cameras 13 constituting a photography unit 12 are arranged around a photography space 16 for photographing the subject 1. The photography space 16 is set to be a relatively large space so that the subject 1 can move freely, for example.

[0047] The green screen 11 is a full-circle screen that surrounds the entire shooting space 16 in which the subject 1 is shot. Therefore, the shooting studio shown in FIG. 2 is a full-circle green screen shooting studio. By using the full-circle screen, for example, the green screen 11 can be used as the background no matter from which direction the subject 1 is shot. Furthermore, the shape of the green screen 11 directly becomes the shape of the shooting space 16. Note that the full-circle screen only needs to be positioned behind and in front of the subject 1, and does not necessarily have to surround the subject 1 in all directions (360°). For example, the green screen 11 configured as a full-circle screen may have a non-shooting area that includes the entrance and exit of the shooting space 16.

[0048] In the example shown in Figure 2, a cylindrical green screen 11 is constructed. In this case, the side walls and floor of the cylinder are green. The cylindrical space surrounded by the green screen 11 becomes the shooting space 16. Note that lighting equipment and the like are appropriately placed above the shooting space 16 (ceiling portion). The shape of the green screen 11 is not limited to this, and for example, a rectangular parallelepiped or polygonal prism-shaped green screen 11 may be used.

[0049] The multiple cameras 13 are arranged to surround the shooting space 16. That is, the subject 1 is photographed by the multiple cameras 13 arranged to surround the shooting space 16. Each camera 13 is arranged, for example, along the side wall of the green screen 11, facing the center of the shooting space 16. In this embodiment, the multiple cameras 13 (photographing units 12) are configured as omnidirectional cameras that photograph the subject 1 from all around. This makes it possible to photograph the subject 1 from all directions without any blind spots. The multiple cameras 13 may be embedded in the green screen 11, or may be placed inside the green screen 11 using a predetermined fixture.

[0050] It is also assumed that the internal and external parameters of each of the multiple cameras 13 are known in advance. The internal parameters are parameters such as the focal position, image center, and lens characteristics of the camera 13. The external parameters are parameters that represent the position (shooting position) and attitude (shooting direction) of the camera 13. These parameters are recorded as data.

[0051] In this embodiment, live-action volumetric video is shot using a photography studio equipped with a 360° screen (green screen 11) and 360° cameras (multiple cameras 13) as shown in Fig. 2. Live-action volumetric video is a technique for constructing a live-action 3D model 6 of a subject 1 from, for example, images of the subject 1. Therefore, live-action volumetric video is an image that can display, for example, a 3D model 6 that moves in the same way as the subject 1 from any viewpoint or scale.

[0052] When shooting live-action volumetric video, first, subject 1 is photographed from different directions by multiple cameras 13 against the background of green screen 11. Next, a shooting-side controller 14 extracts subject images 3, with the background separated, from the images captured by each camera 13. Next, the shooting-side controller 14 uses the data of the subject images 3 to create three-dimensional data of subject 1, and a 3D model 6 of subject 1 is constructed.

[0053] The data of the 3D model 6 includes, for example, shape data representing the three-dimensional shape of the model, skeletal data as a human model, texture data representing surface information, etc. It may also include data on the background of the 3D model 6. These data are transmitted to the viewing-side controller 23 as content data.

[0054] At the viewing location, a viewing controller 23 generates a content video 5 (live video) based on the content data. The content video 5 is, for example, a video obtained by rendering a 3D model 6 at an arbitrary viewpoint and scale. In this embodiment, the 3D model 6 is rendered according to the position of the viewpoint P2 and the direction of the line of sight L2 of the viewer 2. This enables video display with 6 DoF (6 Degrees of Freedom). In this way, the use of live-action volumetric technology makes it possible to achieve video expression with a high degree of freedom.

[0055] [Gaze Guidance System] As shown in the lower part of Fig. 1, the gaze guidance system 30 includes an illumination device 31 and a gaze guidance controller 32. The illumination device 31 is a device that is placed in real space when used. Specifically, the illumination device 31 is appropriately placed at a location at the filming site where it can project a spot 33 onto the green screen 11 to guide the line of sight L1 of the subject 1. The gaze guidance controller 32 is, for example, a computer installed at the filming site. Note that the gaze guidance controller 32 may also be, for example, a server device installed at a location separate from the filming studio.

[0056] 3 is a schematic diagram showing an example of the configuration of the irradiation device 31. The irradiation device 31 is a device that irradiates a laser beam LB to project a spot 33 on the green screen 11. The irradiation device 31 is a movable laser pointer that has an azimuth angle rotation unit 34, an elevation angle rotation unit 35, and a laser pointer 36, and the attitude of the laser pointer 36 is controlled by the azimuth angle rotation unit 34 and the elevation angle rotation unit 35.

[0057] The azimuth angle rotation unit 34 is a rotation stage provided below the irradiation device 31, and rotates around a rotation axis C1. The azimuth angle rotation unit 34 is disposed, for example, so that the rotation axis C1 is parallel to the vertical direction. The elevation angle rotation unit 35 is a rotation mechanism mounted on the azimuth angle rotation unit 34, and rotates around a rotation axis C2. The elevation angle rotation unit 35 is disposed on the azimuth angle rotation unit 34 so that the rotation axis C2 is parallel to the horizontal direction, for example. The azimuth angle rotation unit 34 and the elevation angle rotation unit 35 are driven, for example, using a stepping motor or the like.

[0058] The laser pointer 36 is a light-emitting element that emits a laser beam LB. When the laser beam LB emitted from the laser pointer 36 hits an object, a spot 33, which is a bright spot of the laser beam LB, is formed at that position. This makes it possible to project the spot 33 onto the green screen 11. This spot 33 serves as a mark indicating the position to which the subject 1 should direct their line of sight L1, and functions as gaze guidance information for the subject 1. In this way, the irradiation device 31 functions as a presentation device that presents the position to which the subject 1 should direct their line of sight L1 (gaze guidance information).

[0059] A light-emitting element using a laser diode or the like as a light source is used as the laser pointer 36. The color and intensity of the laser beam LB are appropriately set so that the subject 1 can easily perceive the spot 33 when it is irradiated onto the green screen 11, for example. Other than this, the specific configuration of the laser pointer 36 is not limited.

[0060] The laser pointer 36 is attached to the elevation angle rotation unit 35 so as to rotate together with the elevation angle rotation unit 35. For example, by driving the elevation angle rotation unit 35, the irradiation direction of the laser pointer 36 (the irradiation direction of the laser beam LB) is rotated in the elevation angle direction. This corresponds to angle control in the tilt direction, which moves the position of the spot 33 up and down. Furthermore, by driving the azimuth angle rotation unit 34, the laser pointer 36 is rotated together with the elevation angle rotation unit 35. As a result, the irradiation direction of the laser pointer 36 is rotated in the azimuth angle direction. This corresponds to angle control in the pan direction, which moves the position of the spot 33 left and right.

[0061] In this way, the irradiation device 31 is a laser pointer 36 that can rotate in the elevation angle direction and the azimuth angle direction. This makes it possible to set the irradiation direction of the laser pointer 36 in a desired direction, for example, by controlling the rotation angles in the elevation angle direction and the azimuth angle direction, i.e., the tilt angle and pan angle.

[0062] 3, the rotation axis C1 of the azimuth angle rotation unit 34 and the rotation axis C2 of the elevation angle rotation unit 35 are configured to intersect. As a result, the intersection of the rotation axis C1 and the rotation axis C2 is a point that does not move even when the azimuth angle rotation unit 34 and the elevation angle rotation unit 35 are driven, and can be treated as the origin when controlling, for example, the laser pointer 36. The configuration of the irradiation device 31 is not limited to this, and, for example, the rotation axis C1 and the rotation axis C2 do not have to intersect. Even in such a case, it is possible to appropriately calculate the rotation angles in the elevation angle direction and the azimuth angle direction in accordance with the structure of the irradiation device 31, and it is possible to align the irradiation direction with the desired direction.

[0063] 4 is a block diagram showing an example of the functional configuration of the gaze guidance controller 32. The gaze guidance controller 32 is a device that controls the operation of the irradiation device 31 to guide the gaze L1 of the subject 1, and is configured using, for example, a PC (Personal Computer) or the like. The gaze guidance controller 32 has a storage unit 40 and a control device 41.

[0064] The storage unit 40 is a non-volatile storage device, such as a solid state drive (SSD) or a hard disk drive (HDD). A control program is stored in the storage unit 40. The control program is, for example, a program that controls the overall operation of the gaze guidance controller 32. In this embodiment, the storage unit 40 corresponds to a computer-readable recording medium on which a program is recorded. The control program corresponds to the program recorded on the recording medium.

[0065] The control device 41 is a computing device that controls the operation of the line-of-sight guidance controller 32. The control device 41 has hardware components necessary for a computer, such as a CPU and memory (RAM, ROM). The CPU loads a control program stored in the storage unit 40 into the RAM and executes it, thereby performing various processes. In this embodiment, the control device 41 corresponds to an information processing device.

[0066] The control device 41 may be a programmable logic device (PLD) such as a field programmable gate array (FPGA), or another device such as an application specific integrated circuit (ASIC). Alternatively, the control device 41 may be a graphics processing unit (GPU).

[0067] In this embodiment, the CPU of the control device 41 executes a program (control program) according to this embodiment, thereby realizing functional blocks including a focus position acquisition unit 42, a captured image processing unit 43, and a gaze guidance processing unit 44. These functional blocks then execute the information processing method according to this embodiment. Note that dedicated hardware such as an IC (integrated circuit) may be used as appropriate to realize each functional block.

[0068] The attention position acquisition unit 42 acquires the attention position Pin that the subject 1 should look at. The attention position Pin is, for example, a position to which the line of sight L1 of the subject 1 should be guided, that is, a position to which it is desired to guide the line of sight of the subject 1. Note that the attention position Pin is a position to which the line of sight L1 of the subject 1 is guided as a result of the subject 1 looking at the spot 33 of the irradiation device 31, and is a position different from the position of the projection point where the spot 33 is projected.

[0069] In this embodiment, the attention position Pin is the viewpoint position of the viewer 2 on the display device 21 provided at the viewer's base. The viewpoint position of the viewer 2 is the position of the viewpoint P2 of the viewer 2 detected, for example, by the viewpoint sensor 22 and the viewer-side controller 23 described with reference to FIG. 1 . Hereinafter, the viewpoint position of the viewer 2 may be referred to as the viewpoint position P2 using the same reference symbol as the viewpoint P2 of the viewer 2. The viewpoint position P2 of the viewer 2 is detected, for example, as a coordinate position in real space set at the viewer's base. In this case, the attention position acquisition unit 42 reads coordinate information of the viewpoint position P2 of the viewer 2 via a predetermined network. Note that the attention position acquisition unit 42 may also read the detection result of the viewpoint sensor 22 to detect the viewpoint position P2 of the viewer 2.

[0070] The captured image processing unit 43 processes images from the capture unit 12 (plurality of cameras 13) installed at the capture-side base and acquires information for guiding the line of sight L1 of the subject 1. Specifically, the captured image processing unit 43 acquires the viewpoint position of the subject 1 based on images of the subject 1 captured by the plurality of cameras 13 installed in real space so that the green screen 11 serves as the background of the capture range. The viewpoint position of the subject 1 is the position of the viewpoint P1 of the subject 1. Hereinafter, the viewpoint position of the subject 1 may be referred to as viewpoint position P1 using the same symbol as the viewpoint P1 of the subject 1. The viewpoint position P1 of the subject 1 is detected as a coordinate position in real space set at the capture-side base, for example. A method for calculating the viewpoint position P1 of the subject 1 will be described later with reference to FIG. 9 etc.

[0071] The captured image processing unit 43 also acquires virtual space information representing the position of the green screen 11 in a virtual three-dimensional space based on images captured by the multiple cameras 13. Here, the virtual three-dimensional space is a three-dimensional space constructed by a computer or the like. For example, the virtual space information is data that maps the position of the green screen 11 and reproduces the green screen 11 in the virtual three-dimensional space. The format of the virtual space information is not limited, and may be, for example, point cloud data representing each point on the green screen 11 or polygon data that constitutes the screen surface. A method for generating the virtual space information will be described later with reference to Figures 7 and 8, etc.

[0072] In this way, in the line-of-sight guidance controller 32, the attention position acquisition unit 42 acquires the attention position Pin (here, the viewpoint position P2 of the viewer 2) that the subject 1 should look at, and the captured image processing unit 43 acquires the viewpoint position P1 of the subject 1 and virtual space information. In this embodiment, the attention position acquisition unit 42 and the captured image processing unit 43 constitute an acquisition unit that acquires the viewpoint position of the subject photographed using a screen for chromakey compositing provided in real space, the attention position that the subject should look at, and virtual space information that represents the position of the screen in virtual three-dimensional space.

[0073] The line-of-sight guidance processing unit 44 calculates a projection point Pproj of the spot 33 by the irradiation device 31 using the attention position Pin (the viewpoint position P2 of the viewer 2) acquired by the attention position acquisition unit 42 and the viewpoint position P1 of the subject 1 acquired by the captured image processing unit 43. Here, the projection point Pproj is a guidance point to which the line of sight L1 of the subject 1 is actually guided at the shooting side's base. Hereinafter, the position of the projection point Pproj may be referred to as the projection position Pproj. The process of calculating the projection point Pproj uses virtual space information acquired by the captured image processing unit 43 (data reproducing the green screen 11 in a virtual three-dimensional space).

[0074] Fig. 5 is a schematic diagram illustrating the basic operation of the visual guidance system 30. Fig. 5A is a schematic diagram illustrating a virtual three-dimensional space 18, and Fig. 5B is a schematic diagram illustrating a real space 17 corresponding to the base of the camera. Hereinafter, the virtual three-dimensional space 18 will be simply referred to as the virtual space 18.

[0075] The position in real space 17 is represented by three-dimensional coordinates set with respect to the real world. Hereinafter, the axes of the Cartesian coordinates representing the three-dimensional position in real space 17 will be referred to as the X-axis, Y-axis, and Z-axis, and the origin of the XYZ coordinate system will be referred to as O. The X-axis and Y-axis are axes that are perpendicular to each other in, for example, a horizontal plane, and the Z-axis is an axis that represents the vertical direction that is perpendicular to the horizontal plane.

[0076] The position in the virtual space 18 is represented by a virtually set three-dimensional coordinate system. Hereinafter, the axes of the Cartesian coordinate system representing the three-dimensional position in the virtual space 18 will be referred to as the X'-axis, Y'-axis, and Z'-axis, and the origin of the X'Y'Z' coordinate system will be referred to as O'. The X'-axis and Y'-axis are, for example, axes that are perpendicular to each other within a horizontal plane in the virtual space 18, and the Z'-axis is an axis that represents the vertical direction that is perpendicular to the horizontal plane in the virtual space 18.

[0077] Coordinates expressed in the XYZ coordinate system of real space 17 and coordinates expressed in the X'Y'Z' coordinate system of virtual space 18 correspond one-to-one and can be converted into one another. For example, the coordinates of viewpoint position P1 of subject 1 shown in FIG. 5B are expressed in the XYZ coordinate system set in real space 17 of the shooting side's base. The viewpoint position P1' shown in FIG. 5A is obtained by converting this viewpoint position P1 in real space 17 into a position in virtual space 18. The coordinates of viewpoint position P1' are expressed in the X'Y'Z' coordinate system set in virtual space 18.

[0078] On the other hand, the viewpoint position P2 of the viewer 2 is represented by an XYZ coordinate system set in the real space 17 of the viewer's base. The XYZ coordinate system at the viewer's base is a coordinate system set independently from the XYZ coordinate system (the coordinate system shown in FIG. 5B) at the filming base, but the viewpoint position P2 can also be uniquely converted into a position in the virtual space 18. The viewpoint position P2' shown in FIG. 5A is obtained by converting the viewpoint position P2 of the viewer 2 in the real space 17 into a position in the virtual space 18. The coordinates of the viewpoint position P2' are represented by an X'Y'Z' coordinate system set in the virtual space 18, similar to the viewpoint position P1'. In this embodiment, the viewpoint position P2' becomes the focus position Pin' in the virtual space 18.

[0079] A virtual green screen 11 (hereinafter referred to as a virtual screen 19) generated as virtual space information is placed in the virtual space 18. The virtual screen 19 is a reproduction of the green screen 11 in the real space 17 in the virtual space 18, and the position of each point on the virtual screen 19 is expressed in an X'Y'Z' coordinate system.

[0080] In this way, in virtual space 18, the viewpoint position P1 of subject 1 at the filming location and the position of green screen 11 are reproduced by viewpoint position P1' and virtual screen 19. Furthermore, in virtual space 18, the viewpoint position P2 of viewer 2 at the viewing location is reproduced by viewpoint position P2'. By using virtual space 18 in this way, it is possible to virtually arrange subject 1, viewer 2, and green screen 11 in the same space.

[0081] The method for setting the X'Y'Z' coordinate system is not limited. For example, the X'Y'Z' coordinate system of the virtual space 18 may be set to coincide with the XYZ coordinate system of the real space 17 at the shooting location. In this case, the origin O' is set to the same coordinate value as the origin O, and the X', Y', and Z' axes are set in the same directions as the X, Y, and Z axes, respectively. This makes it possible to use the coordinates of the virtual space 18 as the coordinates of the real space 17. Furthermore, for example, a space in which a 3D model 6 of the subject 1 is rendered may be used as the virtual space 18, or the virtual space 18 may be set as a space independent of the space in which the 3D model 6 is rendered. Alternatively, the coordinate system of the virtual space 18 may be set as desired.

[0082] In this embodiment, the line-of-sight guidance processing unit 44 determines a virtual intersection Pc' between a virtual line L' connecting the viewpoint P1' of the subject 1 and the focus position Pin' (the viewpoint P2' of the viewer 2) in the virtual space 18 and the green screen 11 (virtual screen 19) represented by the virtual space information. This is a process of calculating the position coordinates of the virtual intersection Pc' in the virtual space 18.

[0083] 5A, virtual line L' is a straight line passing through P1' and P2'. In this case, virtual line L' is a line along which subject 1 and viewer 2 make eye contact in virtual space 18, and can be said to be a virtual eye contact line. Virtual intersection point Pc' is a point on such virtual line L'. Virtual intersection point Pc' is also a point on virtual screen 19. Therefore, a point in real space 17 corresponding to virtual intersection point Pc' (a point obtained by converting virtual intersection point Pc' into real space 17) is a point on green screen 11, and is a point onto which spot 33 can be projected.

[0084] For example, suppose that subject 1 is looking at a point in real space 17 that corresponds to virtual intersection point Pc'. In this state, subject 1 is looking at viewer 2's viewpoint P2 in content video 5 displayed at the viewer's location. In other words, by having subject 1 look at virtual intersection point Pc' converted into real space 17, it becomes possible to align the line of sight between subject 1 and viewer 2 at the viewer's location.

[0085] Therefore, the point in real space 17 corresponding to virtual intersection point Pc' is set as a guidance point that guides the line of sight L1 of subject 1, i.e., as projection point Pproj onto which spot 33 is projected by irradiation device 31. The position of projection point Pproj is calculated by converting the position coordinates of virtual intersection point Pc' into position coordinates in real space 17. In this way, projection point Pproj can be said to be a point obtained by converting the coordinates of attention position Pin (viewpoint position P2 of viewer 2) that subject 1 should originally look at, in accordance with the shape of green screen 11 around subject 1.

[0086] Furthermore, the line-of-sight guidance processing unit 44 controls the irradiation direction of the irradiation device 31 provided in the real space 17 so as to point to a projection point Pproj (guidance point) on the green screen 11 that corresponds to the virtual intersection point Pc' in the real space 17. As a result, the irradiation device 31 irradiates a laser beam LB toward the projection point Pproj, and a spot 33 is projected onto the projection point Pproj. For example, by moving the line of sight L1 so that the subject 1 looks at the spot 33, it is possible to display the content video 5 at the viewing location so that the line of sight between the subject 1 and the viewer 2 coincides. In this embodiment, the line-of-sight guidance processing unit 44 functions as a determination unit and a control unit.

[0087] As described above, the green screen 11 is configured as a full-surround screen. Therefore, for example, when photographing the subject 1 from the front, the screen exists not only in the background area but also in the area in front of the subject 1. The area in front of the subject 1 is the area that can be seen when the subject 1 looks straight ahead, for example.

[0088] In this embodiment, the spot 33 is projected onto the screen in front of the subject 1. That is, the line-of-sight guidance processing unit 44 controls the irradiation direction of the irradiation device 31 so that the projection point Pproj (guidance point) is pointed to on the area of ​​the green screen 11, which is a full-circle screen, in front of the subject 1 as seen from the subject 1. This allows the subject 1 to easily find the projection point Pproj. As a result, the line of sight between the subject 1 and the viewer 2 can be naturally aligned without causing the subject 1 to make unnatural movements.

[0089] For example, a front-to-back direction corresponding to the real space 17 of the viewer's base is set in advance in the real space 17 of the filming side's base (the filming space 16 surrounded by the green screen 11). In this case, the forward direction in the filming space 16 corresponds to the front direction as seen from the display device 21 at the viewer's base, for example. In this positional relationship, the subject 1 performs as if the viewer 2 is in front of the filming space 16, and faces mainly forward of the filming space 16. Furthermore, the viewpoint position P2' of the viewer 2 in the virtual space 18 is positioned forward as seen from the subject 1. As a result, the projection point Pproj becomes a point on the area of ​​the green screen 11 forward as seen from the subject 1, and the spot 33 is projected onto the area forward as seen from the subject 1.

[0090] Note that viewer 2 can freely move around the viewing location. Therefore, depending on the positional relationship between subject 1 and viewer 2, it is possible that viewer 2's viewpoint position P2' in virtual space 18 is located behind subject 1. In this case, the point corresponding to virtual intersection point Pc' in shooting space 16 is a point on the green screen 11 in an area behind subject 1. In such a case, projection point Pproj may be controlled to guide subject 1's line of sight to the point corresponding to virtual intersection point Pc'. For example, projection point Pproj may be set in an area in front of subject 1, independently of virtual intersection point Pc', and then moved so that projection point Pproj finally coincides with the point corresponding to virtual intersection point Pc'. This makes it possible to naturally guide subject 1's line of sight to coincide with viewer 2's line of sight.

[0091] In the above example, the positional relationship in real space 17 between the filming-side base and the viewer-side base is fixed in advance. However, the present invention is not limited to this. The positional relationship in real space 17 may be adjusted in virtual space 18 so that projection point Pproj corresponding to virtual intersection point Pc' is a point in an area in front of subject 1. In this case, for example, a process of rotating or translating the coordinate system of at least one of the filming-side base or the viewer-side base is performed so that viewpoint position P2' of viewer 2 in virtual space 18 is positioned in front of subject 1. This allows subject 1 to perform a performance with a higher degree of freedom.

[0092] In this way, in the visual guidance system 30, in order to guide the gaze of the subject 1 during full-circle green screen shooting using the green screen 11, the viewpoint position P2 of the viewer 2, which is the focus position Pin, is coordinate-converted to the projection point Pproj. This coordinate conversion is performed in the virtual space 18 using the viewpoint position P2 of the viewer 2, the viewpoint position P1 of the subject 1, and the shape (virtual space information) of the green screen 11. By presenting a spot 33 with the illumination device 31 at the projection point Pproj calculated in this way, it becomes possible to guide the line of sight L1 of the subject 1 to view any three-dimensional coordinates during shooting of a live-action volumetric video or the like.

[0093] The diameter of the spot 33, which is the bright spot of the laser beam LB, is, for example, several centimeters or less, typically 1 cm or less. Therefore, even if the spot 33 appears in the image of the subject 1, the extraction performance of the subject image 3 is hardly reduced. This makes it possible to guide the line of sight of the subject 1 without reducing the extraction performance of the subject image 3.

[0094] [Operation of the Line-of-Sight Guidance System] Figure 6 is a flowchart showing an example of the operation of the line-of-sight guidance system 30. The process shown in Figure 6 is initiated when the line-of-sight guidance system 30 is activated. First, at the shooting location, a three-dimensional space measurement process is executed to measure the shooting space 16 (step 101). The three-dimensional space measurement is executed, for example, before shooting the subject 1. Here, the captured image processing unit 43 generates virtual space information representing the position of the green screen 11 in the virtual space 18 based on images captured by the multiple cameras 13. Note that if the three-dimensional space measurement process has been executed in advance and virtual space information of the green screen 11 has been recorded in the storage unit 40 or the like, the virtual space information is read out as appropriate.

[0095] Next, at the shooting location, a subject viewpoint measurement process is executed to measure the viewpoint position P1 of the subject 1 (step 102). Here, the captured image processing unit 43 calculates the position coordinates of the viewpoint position P1 of the subject 1 in the real space 17 based on the images captured by the multiple cameras 13.

[0096] Next, a focus position acquisition process is executed to acquire the position coordinates of the focus position Pin (step 103). In this embodiment, the viewpoint position P2 of the viewer 2 measured at the viewer's base is acquired as the focus position Pin. Here, the focus position acquisition unit 42 reads the position coordinates of the viewpoint position P2 of the viewer 2 in the real space 17 from the viewer-side controller 23.

[0097] Next, a coordinate conversion process is executed to convert the focus position Pin (the viewpoint position P2 of the viewer 2) into a projection point Pproj (step 104). Here, the line of sight guidance processing unit 44 converts the viewpoint position P2 of the viewer 2 measured at the viewer's location into a projection point Pproj, which is a point in the real space 17 of the shooting location, based on the viewpoint position P1 of the subject 1 and the virtual space information of the green screen 11.

[0098] Next, a process for providing gaze guidance information is executed (step 105), in which the projection device 31 is controlled to project a spot 33 serving as gaze guidance information onto the projection point Pproj. This process presents a mark (spot 33) for directing the gaze L1 toward the subject 1 being photographed. Here, the gaze guidance processing unit 44 calculates an irradiation direction in which the laser beam LB can be irradiated onto the projection point Pproj. The projection device 31 is controlled to face this irradiation direction, and the spot 33 is projected onto the projection point Pproj.

[0099] When the spot 33 is projected, it is determined whether or not to end the photographing of the subject 1 (step 106). For example, if photographing of the subject 1 is to continue (no in step 106), the processing from step 102 onwards is executed again. Also, if the performance of the subject 1 has ended and the photographing is to end (yes in step 106), the operation of the visual guidance system 30 ends. The operation of each step described with reference to FIG. 6 will now be specifically described.

[0100] [Three-dimensional space measurement processing] Here, a specific method of the three-dimensional space measurement processing executed in step 101 of Fig. 6 will be described. As described above, in order to correctly present the spot 33 (gaze guidance information) in accordance with the viewpoint position P1 of the subject 1, it is necessary to acquire three-dimensional position information of the green screen 11. In this embodiment, the three-dimensional space measurement processing is executed using a plurality of cameras 13 installed so as to surround the shooting space 16, which is also used for shooting live-action volumetric video, and an illumination device 31.

[0101] Fig. 7 is a flowchart showing an example of three-dimensional space measurement processing. Fig. 8 is a schematic diagram illustrating an example of three-dimensional space measurement processing. In three-dimensional space measurement processing, the positions of multiple measurement points 37 on the green screen 11 are measured as position coordinates in the real space 17. The positions of the multiple measurement points 37 are represented in the virtual space 18 to form virtual space information. In the following, it is assumed that the internal parameters and external parameters of the multiple cameras 13 have been calibrated in advance, and that the values ​​of each parameter can be referenced as data.

[0102] First, irradiation of the laser beam LB by the irradiation device 31 is set to OFF (step 201), and images are captured by all of the multiple cameras 13 (step 202). The images captured at this time are used as background images that do not capture the spot 33. Next, irradiation of the laser beam LB by the irradiation device 31 is set to ON (step 203), and the irradiation device 31 is controlled to move the spot 33 (step 204).

[0103] 8 schematically illustrates a plurality of measurement points 37 whose positions are measured on the green screen 11. Here, the position of the irradiation device 31 in the real space 17 is denoted as L. The measurement points 37 are points onto which the spots 33 of the irradiation device 31 are projected. For example, the plurality of measurement points 37 are formed by repeatedly slightly driving the irradiation device 31 in the elevation angle direction or the azimuth angle direction. In the three-dimensional space measurement process, the spots 33 are projected onto the plurality of measurement points 37 in sequence, and the position coordinates in the real space 17 are calculated for each measurement point 37.

[0104] When the spot 33 is moved, spot images of the spot 33 are captured by the multiple cameras 13 (step 205). For example, an image captured by a camera 13 whose angle of view includes the projection position of the spot 33 becomes the spot image. When the spot image is captured, the spot position is extracted (step 206). Here, the spot position on the image is calculated from the difference between the spot image and the background image of the camera 13 that captured the spot image.

[0105] Next, it is determined whether the spot position has been acquired by multiple cameras 13 (step 207). The spot position must be acquired by at least two or more cameras 13. Here, it is determined whether the number of cameras 13 that have detected the spot position exceeds a threshold of two or more. If the number of cameras 13 does not exceed the threshold (no in step 207), the process returns to step 204, and the spot 33 is moved to the next measurement point 37. If the number of cameras 13 exceeds the threshold (yes in step 207), the three-dimensional position of the spot 33 is calculated (step 208).

[0106] The three-dimensional position of the spot 33 is calculated by a triangulation method using, for example, the spot positions in the multiple cameras 13 (the planar positions of the spots 33 in the images) and the internal parameters and external parameters of each camera 13. The three-dimensional position of the spot 33 calculated here is stored as the three-dimensional coordinates of one measurement point 37.

[0107] Once the three-dimensional position of the spot 33 has been calculated, it is determined whether sufficient measurement points 37 have been acquired (step 209). For example, a threshold determination is performed to determine whether the measurement points 37 have been acquired at a sufficient density or whether a sufficient number of measurement points 37 have been acquired. If it is determined that sufficient measurement points 37 have not been acquired (no in step 209), the process returns to step 204, where the irradiation device 31 is driven to slightly move the spot 33. Then, the process continues to measure the position of the next measurement point 37. This makes it possible to measure the surface shape of the green screen 11. If it is determined that sufficient measurement points 37 have been acquired (yes in step 209), the three-dimensional space measurement process ends.

[0108] In this manner, in this embodiment, the spots 33 projected onto the green screen 11 by the irradiation device 31 are photographed by the multiple cameras 13, and the three-dimensional coordinates of the spots 33 are calculated based on the images photographed by the multiple cameras 13. This makes it possible to obtain data representing the three-dimensional position of the green screen 11 without using, for example, a dedicated sensor for measuring the shape of the green screen 11. This makes it possible to prepare virtual space information at a lower cost than when a dedicated sensor is used.

[0109] [Subject viewpoint measurement processing] Here, a specific method of the subject viewpoint measurement processing executed in step 102 of Fig. 6 will be described. In order to control the illumination device 31 to correctly present the spot 33 (gaze guidance information) to the subject 1, it is necessary to correctly acquire the viewpoint position P1 of the subject 1. In this embodiment, similar to the case of the three-dimensional space measurement processing, the subject viewpoint measurement processing is executed using multiple cameras 13 and illumination devices 31.

[0110] 9 is a flowchart showing an example of subject viewpoint measurement processing. In the subject viewpoint measurement processing, the viewpoint position P1 of the subject 1 is measured as position coordinates in real space 17. First, images of the subject 1 are captured by the multiple cameras 13 (step 301). Next, the viewpoint position of the subject 1 is estimated in an image in which the subject 1 appears among the images captured by the multiple cameras 13 (step 302). The viewpoint position estimated here is the planar position of the viewpoint P1 of the subject 1 in the image. Methods such as person posture estimation and face detection can be used to estimate the viewpoint position in an image.

[0111] Next, it is determined whether the viewpoint position in the image of the subject 1 has been acquired by the multiple cameras 13 (step 303). The viewpoint position in the image must be acquired by at least two or more cameras 13. Here, it is determined whether the number of cameras 13 that have been able to detect the viewpoint position in the image exceeds a threshold of two or more. If the number of cameras 13 does not exceed the threshold (no in step 303), the process returns to step 301, and photography is performed again using the multiple cameras 13. If the number of cameras 13 exceeds the threshold (yes in step 303), the three-dimensional position of the viewpoint P1 of the subject 1 (viewpoint position P1) is calculated (step 304).

[0112] The viewpoint position P1 of the subject 1 is calculated in the same manner as in step 208 of Fig. 7. That is, the viewpoint position P1 is calculated by triangulation using the viewpoint positions in the images of the multiple cameras 13 and the internal parameters and external parameters of each camera 13. Once the viewpoint position P1 of the subject 1 is calculated, the subject viewpoint measurement process ends.

[0113] [Coordinate Conversion Processing] Fig. 10 is a schematic diagram illustrating an example of coordinate conversion processing. Here, a specific method of coordinate conversion processing executed in step 104 of Fig. 6 will be described. Fig. 10 schematically illustrates the positional relationship between the viewpoint position P1' of the subject 1 in the virtual space 18, the focus position Pin' (the viewpoint position P2' of the viewer 2), and the virtual screen 19. Here, the surface that constitutes the virtual screen 19 is referred to as the constituent surface S'.

[0114] The viewpoint position P1' of subject 1 is obtained by converting the viewpoint position P1 of subject 1 in real space 17 into coordinates in virtual space 18. The focus position Pin' (the viewpoint position P2' of viewer 2) is obtained by converting the focus position Pin (the viewpoint position P2 of viewer 2) in real space 17 into coordinates in virtual space 18. The virtual screen 19 is data (virtual space information) obtained by converting the shape data of the green screen 11 obtained by the three-dimensional space measurement process into coordinates in virtual space 18.

[0115] Here, a virtual intersection Pc' between a virtual line L' connecting the viewpoint position P1' of the subject 1 and the target position Pin' and the virtual screen 19 is calculated. The equation satisfied by the virtual intersection Pc' is written as follows: Pc'=(1-t)P1'+tPin' (1)

[0116] Here, the virtual intersection point Pc' is a point included in the plane S' of the virtual screen 19. Furthermore, t is a positive real number (t≧0). The right side of equation (1) represents, for example, the tip position of a vector (line-of-sight vector) along the virtual line L' starting from the viewpoint position P1' of the subject 1. Furthermore, t is a parameter representing the length of the line-of-sight vector (the distance from the viewpoint position P1').

[0117] The coordinate transformation process is, for example, a process of determining t that satisfies equation (1) and calculating the coordinates of the virtual intersection point Pc'. For example, t may be changed to determine the coordinate (Pc') that satisfies equation (1). Furthermore, for example, if the constituent surface S' of the virtual screen 19 is a surface that is expressed parametrically, t that satisfies equation (1) may be analytically calculated. In addition, the method of calculating the virtual intersection point Pc' is not limited to this, and for example, an intersection calculation method that corresponds to the data format of the virtual space information that represents the constituent surface S' may be used as appropriate.

[0118] The virtual intersection point Pc' is transformed into the real space 17, and a projection point Pproj is calculated. Specifically, the position coordinates of the virtual intersection point Pc' in the virtual space 18 (position coordinates in the X'Y'Z' coordinate system) are transformed into position coordinates in the real space 17 (position coordinates in the XYZ coordinate system). The transformed position coordinates become the position coordinates of the projection point Pproj.

[0119] [Gaze Guidance Information Presentation Process] Fig. 11 is a schematic diagram illustrating an example of a pointing operation using an illumination device. Here, a specific method of the gaze guidance information presentation process executed in step 105 of Fig. 6 will be described. Fig. 11 schematically illustrates the positional relationship between the position L of the illumination device 31 in the real space 17 of the shooting side's base and the projection point Pproj on the green screen 11.

[0120] As described above, the gaze guidance information presentation process is a process of controlling the irradiation device 31 so that the spot 33 is projected onto the projection point Pproj. In this embodiment, the irradiation direction of the irradiation device 31 is calculated based on the position L of the irradiation device 31 and the position of the projection point Pproj (guidance point) in the real space 17. That is, the irradiation direction that enables the spot 33 to be projected from the position L of the irradiation device 31 onto the projection point Pproj is calculated. In addition, the line connecting the position L of the irradiation device 31 and the projection point Pproj becomes the irradiation path 38 of the laser beam LB.

[0121] Specifically, the rotation angle (pan) in the azimuth direction and the rotation angle (tilt) in the elevation direction of the irradiation device 31 are calculated as parameters for controlling the irradiation direction of the irradiation device 31. These rotation angles are rotation angles from a reference irradiation direction Dir, which serves as a reference for the irradiation direction. Therefore, it can be said that the reference irradiation direction Dir is the initial direction when pan = 0 and tilt = 0. It is assumed that the position L of the irradiation device 31 and the orientation of the reference irradiation direction Dir are known in advance. The rotation angle (pan) in the azimuth direction and the rotation angle (tilt) in the elevation direction are calculated according to the following formula: L + t Rpan Rtilt Dir = Pproj (2)

[0122] Here, Rpan is a rotation matrix that rotates any three-dimensional vector in the azimuth direction by an angle pan. Rtilt is a rotation matrix that rotates any three-dimensional vector in the elevation direction by an angle tilt. t is a positive real number (t≧0). Equation (2) is a simultaneous equation that calculates pan, tilt, and t such that the tip of the vector extending from the position L of the irradiation device 31 coincides with Proj.

[0123] In this manner, in this embodiment, the rotation angles (pan, tilt) of the irradiation direction relative to the reference irradiation direction Dir, which serves as a reference for the irradiation direction in the irradiation device 31, are calculated. The calculated pan and tilt are transmitted to the irradiation device 31 as control angles. In the irradiation device 31, the azimuth angle rotation unit 34 is driven so that the angular position coincides with pan. Similarly, the elevation angle rotation unit 35 is driven so that the angular position coincides with tilt. This directs the irradiation direction of the laser beam LB toward the projection point Pproj, making it possible to project the spot 33 on the projection point Pproj.

[0124] 12 is a schematic diagram illustrating another example of a pointing operation by the irradiation device 31. Here, a control example will be described in which the illumination of the irradiation device 31 (laser pointer 36) is controlled to hide the spot 33 on a target on which the spot 33 should not be projected.

[0125] For example, if the laser beam LB were to be directly irradiated onto a photographing device such as the camera 13, a spot 33 of the laser beam LB would appear in the photographed image. Although the intensity of the laser beam LB is sufficiently suppressed, it would be dangerous if the spot 33 were to be irradiated onto people in the photographing space 16, including the subject 1. As such, there are objects in the photographing space 16 that should not be irradiated with the laser beam LB.

[0126] In such a case, it is possible to take measures to prevent irradiation in advance by referring to the irradiation path 38 (irradiation direction) of the laser beam LB described with reference to Fig. 13. Specifically, the line-of-sight guidance processing unit 44 determines whether the irradiation path 38 connecting the position L of the irradiation device 31 and the projection point Pproj (guidance point) in the real space 17 passes through a non-irradiation area 46 where pointing of the projection point Pproj should be avoided. Furthermore, if it is determined that the irradiation path 38 passes through the non-irradiation area 46, pointing of the projection point Pproj is stopped.

[0127] Here, the non-irradiated area 46 where pointing of the projection point Pproj should be avoided is an area where there is an object (non-irradiated object 47) to be prevented from being irradiated with the laser beam LB, such as a photographing device such as the camera 13 or a person in the photographing space 16. In addition, an area that is a certain distance away from the position of the non-irradiated object 47 may be set as the non-irradiated area 46.

[0128] 12, a square object is schematically illustrated as an example of the non-irradiation target 47. For example, if the non-irradiation target 47 is a stationary object, the position of the stationary object in the real space 17 is set to T. Furthermore, an area (the hatched area in the figure) a predetermined distance away from the position T of the non-irradiation target 47 is set as the non-irradiation target 46. Note that the non-irradiation target 46 is typically set to be an area that includes the entire non-irradiation target 47.

[0129] Once the projection point Pproj is calculated, it is determined whether the irradiation path 38 passes through the non-irradiation area 46. For example, the distance from the position T of the non-irradiation object 47 to the irradiation path 38 is calculated using the position of the projection point Pproj and the position L of the irradiation device 31. If this distance is shorter than a predetermined distance that defines the non-irradiation area 46, it is determined that the irradiation path 38 passes through the non-irradiation area 46. In this case, there is a possibility that the laser beam LB will be irradiated onto the non-irradiation object 47, so the laser pointer 36 is turned off and pointing of the projection point Pproj is discontinued.

[0130] In this way, if the distance between the non-irradiated object 47 and the irradiation path 38 is such that the laser beam LB is likely to be irradiated onto the non-irradiated object 47, it is possible to avoid irradiating the non-irradiated object 47 with the laser beam LB by temporarily turning off the laser pointer 36.

[0131] 12, the case where the non-irradiated object 47 is a stationary object in the shooting space 16 has been described, but similar measures can be taken for the subject 1, etc., whose position in the shooting space 16 changes dynamically. The position of the subject 1 can be detected by, for example, triangulation using images captured by multiple cameras 13. There are no other limitations on the method for detecting the position of the subject 1.

[0132] [Calibration of Irradiation Device] Fig. 13 is a schematic diagram illustrating a method for calibrating the irradiation device 31. In the explanation up to this point, a case has been described in which the position L and the reference irradiation direction Dir of the irradiation device 31 have been calibrated in advance and recorded as data. For example, when a new irradiation device 31 is installed or when the irradiation device 31 is moved, the position L and the reference irradiation direction Dir of the irradiation device 31 are calibrated again. Here, a method for calibrating the position L and the reference irradiation direction Dir of the irradiation device 31 will be explained.

[0133] 13 illustrates a method of using the operation of the illumination device 31 and the calibrated physical positions of the multiple cameras 13. This process uses coordinates in the real space 17 of the shooting location. Specifically, the position L of the illumination device 31 and the reference illumination direction Dir are calibrated based on the rotation angle (pan, tilt) of the illumination direction when the illumination device 31 projects a spot 33 onto each of the multiple cameras 13 and the installation positions of the multiple cameras 13.

[0134] In the following, the multiple cameras 13 arranged around the shooting space 16 are referred to as cam ｉ i is an index set for each camera 13. The installation position of each camera 13 is expressed as Pcam ｉ It is written as follows. ｉ is the position of the camera 13 (external parameter) and its position in the coordinate system of the real space 17 is calibrated.

[0135] For example, the rotation angle (pan, tilt) of the irradiation device 31 is controlled to rotate the i-th camera 13 (cam ｉ In this case, the position L of the irradiation device 31, the reference irradiation direction Dir, and the position of the i-th camera 13 (cam ｉ ) Installation location Pcam ｉ The relationship between L and t is expressed by the following formula: ｉ ・Rpan ｉ ・Rtilt ｉ ・Dir=Pcam ｉ ...(3)

[0136] Here, t ｉ is a positive real number (t ｉ≧0). ｉ is the rotation angle pan in the azimuth direction when the spot 33 is projected on the i-th camera 13. ｉ Similarly, Rtilt ｉ is the rotation angle tilt in the elevation angle direction when the spot 33 is projected on the i-th camera 13. ｉ is the rotation matrix expressed as:

[0137] Therefore, in order to calculate the position L of the irradiation device 31 and the reference irradiation direction Dir, the minimization problem expressed by the following equation is solved: ｉ -(L + t ｉ ・Rpan ｉ ・Rtilt ｉ ・Dir)|} ...(4)

[0138] Installation position Pcam of each camera 13 ｉ can be obtained by referring to external parameters. ｉ and Tilt ｉ is recorded each time irradiation is performed on the camera 13. Therefore, by irradiating the laser beam LB on multiple cameras 13 and using the data on the rotation angle at that time, it is possible to solve the minimization problem of equation (4). This makes it possible to estimate the position L of the irradiation device 31 and the reference irradiation direction Dir.

[0139] In this method, the arrangement of the plurality of cameras 13 used to photograph the subject 1 can be used as is, so that the calibration of the illumination device 31 can be realized at low cost. ｉ ) Installation location Pcam ｉ However, if there is a point on the surface of the green screen 11 or in the shooting space 16 whose position coordinates are known, it is also possible to estimate the position L of the illumination device 31 and the reference illumination direction Dir by illuminating that point.

[0140] 14 is a schematic diagram illustrating another method for calibrating the irradiation device. Here, a method for calibrating the position L and the reference irradiation direction Dir of the irradiation device 31 using a position detection marker 39 will be described. In this method, as shown in FIG. 14, the position detection marker 39 is provided on the irradiation device 31.

[0141] The position detection marker 39 is a marker for detecting the position and posture of a target. Here, a planar marker that indicates a position on a plane by a design is used. AR markers such as ArUco markers can be used as such markers. The position detection marker 39 is provided, for example, in a part that is driven together with the laser pointer 36 of the irradiation device 31 (i.e., a part whose position and posture change depending on the irradiation direction).

[0142] When calibrating the position L and the reference irradiation direction Dir of the irradiation device 31, the position L and the reference irradiation direction Dir of the irradiation device 31 are calculated based on images of the position detection markers 39 of the irradiation device 31 captured by the multiple cameras 13. For example, the position detection markers 39 provided on the irradiation device 31 are detected from images captured by the multiple cameras 13, and the position and orientation of the position detection markers 39 are estimated. Based on this estimation result, the position L and the reference irradiation direction Dir of the irradiation device 31 are estimated. In this way, by using the position detection markers 39, it is possible to easily calibrate the position L and the reference irradiation direction Dir of the irradiation device 31 without, for example, operating the irradiation device 31.

[0143] As described above, the control device 41 according to this embodiment determines a virtual intersection Pc' where the virtual line L' connecting the viewpoint position P1' of the subject 1 and the focus position Pin' at which the subject 1 should look in the virtual space 18 intersects with the green screen 11 (virtual screen 19) used for chromakey compositing. Then, the projection point Pproj (guidance point) on the actual green screen 11 corresponding to the virtual intersection Pc' is pointed to by the illumination device 31. By pointing to the projection point Pproj, the direction in which the subject 1 should look can be presented. This makes it possible to guide the gaze of the subject 1 without reducing the performance of extracting the subject image 3.

[0144] A method for extracting the foreground of a captured image is to use a green screen. For example, when capturing images of a subject 1 in a surrounding environment, such as volumetric photography, a green screen is required to capture the entire image of the subject 1.

[0145] For example, if it is desired to guide the line of sight L1 of the subject 1 in a specific direction, a shooting environment such as a full-surround green screen has few features for guiding the line of sight L1 of the subject 1, and the subject 1 may not know where to focus during shooting. On the other hand, if a display or the like is introduced to guide the line of sight L1 of the subject 1, there is a possibility that the accuracy of extracting the subject image 3 will decrease.

[0146] In this embodiment, a spot 33 is projected onto the green screen 11 using an irradiation device 31 (movable laser pointer) that emits a laser beam LB. This makes it possible to present information for guiding the gaze of the subject 1 in a minimum area. This makes it possible to sufficiently maintain the ability to extract the subject image 3.

[0147] Furthermore, in this embodiment, the focus position Pin (the viewpoint position P2 of the viewer 2) to be presented to the subject 1 is coordinate-converted to the projection point Pproj via the virtual space 18. This coordinate conversion uses the viewpoint position P1 of the subject 1 and virtual space information representing the shape of the green screen 11. This makes it possible to calculate with high accuracy the projection point Pproj onto which the spot 33 should be projected in the real space 17, and to appropriately and accurately guide the line of sight L1 of the subject 1.

[0148] In this embodiment, the gaze guidance system 30 is used for live streaming of live-action volumetric video. For example, it is difficult for the subject 1, who is filmed using a green screen 11, to see what the viewer 2 is doing. In contrast, by using the gaze guidance system 30, it becomes possible to easily show the subject 1 where the viewpoint P2 of the viewer 2 is. As a result, the content video 5 viewed by the viewer 2 appears as if the subject 1 is looking at the viewer 2, giving the viewer 2 the feeling that their gazes are aligned with the subject 1's. This makes it possible to achieve interactive communication with a sense of reality.

[0149] Other Embodiments The present technology is not limited to the above-described embodiments, and various other embodiments can be realized.

[0150] In the above embodiment, a movable laser pointer that projects a single spot 33 is used as the projection device 31. The configuration of the projection device 31 is not limited, and for example, a device that can project a pattern other than the single spot 33 as gaze guidance information may be used.

[0151] Fig. 15 is a schematic diagram showing another example of the configuration of an irradiation device. The irradiation device 51 shown in Fig. 15 is a laser beam scanning type projector that scans a laser beam LB using a movable mirror 53. The irradiation device 51 has a light source unit 52 that emits the laser beam LB and a movable mirror 53 that reflects the laser beam LB emitted from the light source unit 52.

[0152] The movable mirror 53 is, for example, a MEMES (Micro Electro Mechanical Systems) mirror that uses a MEMS element. The movable mirror 53 is configured to be tiltable, for example, around two tilt axes that are orthogonal to each other within the mirror plane. The tilt angle of the movable mirror 53 is electrically controlled. This makes it possible to scan the spot 33 of the laser beam LB, thereby enabling any pattern to be drawn.

[0153] When a laser beam scanning type irradiation device 51 is used in this manner, the irradiation device 51 may be controlled to project a pattern 55 including at least one of text and an image onto the projection point Pproj (guidance point). For example, in the example shown in FIG. 15 , a pattern 55 including text information such as the name of the viewer 2 is projected centered on the projection point Pproj. In addition to this, text information such as lines or elapsed time may be used. Furthermore, an image pattern such as a sign or icon indicating instructions for production may be projected. This makes it possible to properly guide the line of sight L1 of the subject 1 being photographed while conveying various information to the subject 1.

[0154] In the above embodiment, a configuration using a single irradiation device has been described, but multiple irradiation devices may be used. By using multiple irradiation devices, for example, it is possible to shorten the irradiation distance of each irradiation device. This makes it possible to suppress irradiation of the laser beam LB on non-irradiated objects, etc.

[0155] When using multiple projection devices, the projection range of each projection device may be set in advance to match the shape of the green screen so that the line-of-sight guidance information, such as spots, does not overlap. If there is a non-illumination target in the projection path of a projection device, it is also possible to project spots, etc., using another projection device.

[0156] In the above embodiment, an example has been described in which a live video of the subject 1 is provided to the viewer 2, and the focus position Pin at which the subject 1 should be viewed is set to the viewpoint position P2 of the viewer 2. The focus position Pin can be set arbitrarily. For example, a three-dimensional position designed in advance in the virtual space 18 can be used as the focus position Pin.

[0157] Fig. 16 is a schematic diagram illustrating another application example of the visual guidance system. Fig. 16 schematically illustrates a viewpoint position P1' and a focus position Pin' of a subject 1 in a virtual space 18. Here, the focus position Pin' is a shooting position Pcam' of a virtual camera 57 set in a virtual three-dimensional space 18 (virtual space 18).

[0158] The virtual camera 57 is a virtual camera that captures an image of a target object (such as a 3D model of the subject 1) in the virtual space 18, and serves as the shooting viewpoint when rendering the target, for example. When the camerawork (movement trajectory 58) of the virtual camera 57 is determined in advance, the shooting position Pcam′ of the virtual camera 57 becomes three-dimensional data that changes in time series along the movement trajectory 58.

[0159] In this case, during shooting of the subject 1, the spot 33 and the like are projected onto the projection point Pproj that is calculated so that the line of sight L1 is directed toward the shooting position Pcam'. As a result, the subject 1 being shot in the real space 17 follows the spot 33 with his or her eyes, and moves while directing his or her line of sight toward the virtual camera 57 that moves along the movement trajectory 58 in the virtual space 18. This allows the subject 1 to move with a feeling similar to that of shooting while moving a camera in the real space 17.

[0160] FIG. 17 is a schematic diagram illustrating another application example of the gaze guidance system. FIG. 17 illustrates gaze guidance when shooting live-action volumetric video targeting multiple subjects 1. In volumetric shooting, when multiple subjects 1 are shot simultaneously, the accuracy of extracting the subject image 3 of each subject 1 may decrease, and the accuracy of the 3D model 6 may decrease. For this reason, multiple subjects 1 may be shot individually. However, when shooting individually, there is a possibility that the sense of unity of the scene may be lost due to reasons such as variations in the direction of the gaze of each subject 1.

[0161] Therefore, even in the individual photographing, the gaze is guided by presenting the same attention position Pin. ａ ' and the focus position Pin ａ Here, the focus position Pin ａ ' is the shooting position Pcam' of the virtual camera 57 that moves along a preset movement trajectory 58.

[0162] 17B shows the viewpoint position P1 of the subject 1b in the virtual space 18. ｂ ' and the focus position Pin ｂFor example, the focus position Pin ｂ ' is the focus position Pin ａ ' is set to the same position as the focus position Pin ｂ 17A. This means that the same attention position Pin is presented to each subject 1 even in individual photography, and it becomes possible to photograph with a sense of unity even when 3D models 6 of multiple people are integrated.

[0163] It should be noted that the focus positions Pin presented to the multiple subjects 1 do not necessarily have to be the same. For example, depending on the shooting scene, different focus positions Pin may be presented for each subject 1, or the same focus position Pin may be presented. This makes it possible to realize various effects in volumetric shooting of multiple subjects 1, etc.

[0164] Fig. 18 is a schematic diagram illustrating another application example of the visual guidance system. Fig. 18A is a schematic diagram showing a shooting space 16 at a shooting site. Fig. 18B is a schematic diagram showing a background 60 for compositing. The background 60 for compositing may be a real image or a virtually constructed image such as 3DCG. Here, the attention position Pin that the subject 1 should look at is the position of an attention target 61 that the subject 1 should look at in the background 60 that is to be composited with the subject image 3 extracted from a captured image of the subject 1.

[0165] For example, in Fig. 18B, the subject image 3 is shown as a gray silhouette when composited with a background 60. For example, the traffic light in the upper left of the background 60 is the target of attention 61. Therefore, the target of attention Pin of the subject 1 is set to the position of the target of attention 61 (the traffic light in the upper left). As shown in Fig. 18A, in the shooting space 16, a projection point Pproj is calculated that guides the line of sight L1 of the subject 1 to the target of attention Pin, and a spot 33 is projected onto the projection point Pproj. Note that in Fig. 18A, the target of attention Pin is set on the far side of the green screen 11 as seen from the subject 1.

[0166] In this way, it is also possible to set the position of an object (target of attention 61) included in the composite background 60 as the target of attention Pin. When the subject image 3 extracted from the image captured by guiding the line of sight L1 of the subject 1 in this way is composited with the background 60, it becomes an image in which the subject is looking at the object that will become the target of attention 61. This makes it possible to realize various effects that match the background 60.

[0167] Although the above description has mainly focused on the case where a subject is photographed using multiple cameras, the number of cameras is not limited, and the present technology can also be applied to the case where a single camera is used for photographing. Furthermore, the position of the camera photographing the subject does not need to be fixed. When a single camera is used, three-dimensional measurement of the green screen can be performed using a dedicated sensor capable of three-dimensional measurement, such as a ranging sensor or a Time of Flight (ToF) camera.

[0168] So far, we have mainly described the case of capturing live-action volumetric video and generating a 3D model of a subject. The method of capturing the subject and the type of video generated are not limited. For example, even in cases where a subject is captured using a green screen, the subject's image is extracted from the image and then composited directly onto the background, by applying this technology, it is possible to properly guide the subject's gaze without compromising the ability to extract the subject's image.

[0169] In the above, the information processing method according to the present technology is executed by a control device that is a processor of a gaze guidance controller provided at a base of the shooting side. However, without being limited to this, the information processing method and program according to the present technology may be executed by linking the gaze guidance controller with another computer that can communicate via a network or the like, thereby constructing an information processing device according to the present technology.

[0170] In other words, the information processing method and program according to the present technology can be executed not only in a computer system composed of a single computer, but also in a computer system in which multiple computers operate in conjunction with each other. In this disclosure, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0171] The execution of the information processing method and program according to the present technology by a computer system includes both cases where, for example, the acquisition of the viewpoint position, focus position, and virtual space information of the subject on the screen, the determination of the virtual intersection, the control of the irradiation direction of the irradiation device, etc. are executed by a single computer, and cases where each process is executed by a different computer. Furthermore, the execution of each process by a specific computer includes having another computer execute part or all of the process and acquiring the results.

[0172] That is, the information processing method and program according to the present technology can also be applied to a cloud computing configuration in which a single function is shared and processed jointly by multiple devices via a network.

[0173] It is also possible to combine at least two of the features of the present technology described above. That is, the various features described in each embodiment may be arbitrarily combined without distinguishing between the embodiments. Furthermore, the various effects described above are merely examples and are not intended to be limiting, and other effects may also be achieved.

[0174] In the present disclosure, the terms "same," "equal," "orthogonal," etc. are concepts that include "substantially the same," "substantially equal," "substantially orthogonal," etc. For example, they also include states that fall within a predetermined range (e.g., a range of ±10%) based on "completely the same," "completely equal," "completely orthogonal," etc.

[0175] Note that the present technology can also be configured as follows. (1) An information processing device comprising: an acquisition unit that acquires a viewpoint position of a subject photographed using a screen for chromakey compositing provided in real space, a focus position at which the subject should look, and virtual space information representing the position of the screen in a virtual three-dimensional space; a determination unit that determines a virtual intersection between a virtual line connecting the viewpoint position and the focus position of the subject in the virtual three-dimensional space and the screen represented by the virtual space information; and a control unit that controls an irradiation direction of an irradiation device provided in the real space so as to point to a guidance point on the screen corresponding to the virtual intersection in the real space. (2) The information processing device described in (1), wherein the focus position is a viewpoint position of a viewer of a display device that displays an image of the subject extracted from an image of the subject photographed with the screen as a background. (3) The information processing device described in (1) or (2), wherein the acquisition unit acquires the viewpoint position of the subject based on images of the subject photographed by multiple cameras provided in the real space so that the screen is the background of the photographing range. (4) The information processing device according to (3), wherein the acquisition unit generates the virtual space information based on images captured by the multiple cameras. (5) The information processing device according to (3) or (4), wherein the acquisition unit captures images of spots projected onto the screen by the irradiation devices with the multiple cameras, and calculates three-dimensional coordinates of the spots based on the images captured by the multiple cameras. (6) The information processing device according to at least one of (3) to (5), wherein the control unit calculates the irradiation direction based on the position of the irradiation device and the position of the guidance point in the real space. (7) The information processing device according to (6), wherein the control unit calculates a rotation angle of the irradiation direction with respect to a reference irradiation direction that serves as a reference for the irradiation direction in the irradiation device.(8) The information processing device according to (7), wherein the control unit calibrates the position of the irradiation device and the reference irradiation direction based on a rotation angle of the irradiation direction when the irradiation device projects a spot onto each of the plurality of cameras and installation positions of the plurality of cameras. (9) The information processing device according to (7), wherein the irradiation device has a position detection marker, and the control unit calculates the position of the irradiation device and the reference irradiation direction based on images of the position detection marker of the irradiation device taken by the plurality of cameras. (10) The information processing device according to at least one of (6) to (9), wherein the control unit determines whether an irradiation path connecting the position of the irradiation device and the induction point in the real space passes through a non-irradiation area where pointing of the induction point should be avoided, and when it is determined that the irradiation path passes through the non-irradiation area, stops pointing of the induction point. (11) The information processing device according to at least one of (1) to (10), wherein the irradiation device is a laser pointer rotatable in an elevation angle direction and an azimuth angle direction. (12) The information processing device according to at least one of (1) to (10), wherein the irradiation device is a laser beam scanning projector that scans a laser beam using a movable mirror, and the control unit controls the projector to project a pattern including at least one of characters or an image onto the guiding point. (13) The information processing device according to at least one of (1) to (12), wherein the screen is a full-circle screen that surrounds the entire periphery of an imaging space in which the subject is imaged, and the control unit controls the irradiation direction of the irradiation device to point to the guiding point on an area of ​​the full-circle screen that is forward as viewed from the subject. (14) The information processing device according to at least one of (1) to (13), wherein the focus position is an imaging position of a virtual camera set in the virtual three-dimensional space.(15) The information processing device according to at least one of (1) to (14), wherein the focus position is a position of a focus target to be looked at by the subject in a background to be combined with an image of the subject extracted from an image of the subject. (16) An information processing method executed by a computer system, comprising: acquiring a viewpoint position of a subject photographed using a screen for chromakey compositing provided in a real space, a focus position to be looked at by the subject, and virtual space information representing the position of the screen in a virtual three-dimensional space; determining a virtual intersection between a virtual line connecting the viewpoint position and the focus position of the subject in the virtual three-dimensional space and the screen represented by the virtual space information; and controlling an irradiation direction of an irradiation device provided in the real space so as to point to a guidance point on the screen corresponding to the virtual intersection in the real space. (17) A computer-readable recording medium having recorded thereon a program for executing the steps of: acquiring the viewpoint position of a subject photographed using a screen for chromakey compositing installed in real space, the focus position at which the subject should look, and virtual space information representing the position of the screen in virtual three-dimensional space; determining a virtual intersection between a virtual line connecting the viewpoint position and the focus position of the subject in the virtual three-dimensional space and the screen represented by the virtual space information; and controlling the irradiation direction of an irradiation device installed in the real space so as to point to a guidance point on the screen corresponding to the virtual intersection in the real space.

[0176] REFERENCE SIGNS LIST 1... Subject 2... Viewer 11... Green screen 13... Camera 17... Real space 18... Virtual space 31... Illumination device 32... Line of sight guidance controller 41... Control device 42... Attention position acquisition unit 43... Captured image processing unit 44... Line of sight guidance processing unit 100... Telepresence system

Claims

1. An information processing device comprising: an acquisition unit that acquires the viewpoint position of a subject photographed using a screen for chromakey compositing installed in real space, a focus position at which the subject should look, and virtual space information representing the position of the screen in a virtual three-dimensional space; a determination unit that determines a virtual line connecting the viewpoint position and focus position of the subject in the virtual three-dimensional space and a virtual intersection with the screen represented by the virtual space information; and a control unit that controls the irradiation direction of an irradiation device installed in the real space so as to point to an induction point on the screen that corresponds to the virtual intersection in the real space.

2. An information processing device according to claim 1, wherein the focus position is a viewpoint position of a viewer of a display device that displays an image of the subject extracted from an image of the subject photographed against the screen.

3. An information processing device according to claim 1, wherein the acquisition unit acquires the viewpoint position of the subject based on images of the subject captured by a plurality of cameras installed in the real space such that the screen is the background of the shooting range.

4. An information processing device according to claim 3, wherein the acquisition unit generates the virtual space information based on images captured by the plurality of cameras.

5. An information processing device according to claim 3, wherein the acquisition unit captures images of spots projected onto the screen by the irradiation device using the multiple cameras, and calculates three-dimensional coordinates of the spots based on the images captured by the multiple cameras.

6. An information processing device according to claim 3, wherein the control unit calculates the irradiation direction based on the position of the irradiation device and the position of the induction point in the real space.

7. An information processing device according to claim 6, wherein the control unit calculates a rotation angle of the irradiation direction relative to a reference irradiation direction that serves as a reference for the irradiation direction in the irradiation device.

8. An information processing device according to claim 7, wherein the control unit calibrates the position of the irradiation device and the reference irradiation direction based on a rotation angle of the irradiation direction when a spot is projected by the irradiation device onto each of the multiple cameras and on installation positions of the multiple cameras.

9. An information processing device according to claim 7, wherein the irradiation device has a position detection marker, and the control unit calculates the position of the irradiation device and the reference irradiation direction based on images of the position detection marker of the irradiation device captured by the multiple cameras.

10. An information processing device according to claim 6, wherein the control unit determines whether an illumination path connecting the position of the illumination device and the induction point in the real space passes through a non-illuminated area where pointing of the induction point should be avoided, and when it is determined that the illumination path passes through the non-illuminated area, stops pointing of the induction point.

11. An information processing device according to claim 1, wherein the irradiation device is a laser pointer that is rotatable in the elevation angle direction and the azimuth angle direction.

12. An information processing device as described in claim 1, wherein the irradiation device is a laser beam scanning type projector that scans a laser beam using a movable mirror, and the control unit controls the projector to project a pattern including at least one of characters or an image onto the guidance point.

13. An information processing device according to claim 1, wherein the screen is a full-circle screen that surrounds the entire circumference of the shooting space in which the subject is shot, and the control unit controls the irradiation direction of the irradiation device so as to point to the induction point on an area of ​​the full-circle screen that is in front of the subject.

14. An information processing device according to claim 1, wherein the focus position is a shooting position of a virtual camera set in the virtual three-dimensional space.

15. An information processing device according to claim 1, wherein the focus position is a position of an attention target to be seen by the subject in a background that is combined with an image of the subject extracted from an image capturing the subject.

16. An information processing method executed by a computer system, which comprises: acquiring a viewpoint position of a subject photographed using a screen for chromakey compositing installed in real space, a focus position at which the subject should look, and virtual space information representing the position of the screen in a virtual three-dimensional space; determining a virtual intersection between a virtual line connecting the viewpoint position and focus position of the subject in the virtual three-dimensional space and the screen represented by the virtual space information; and controlling the irradiation direction of an irradiation device installed in the real space so as to point to an induction point on the screen corresponding to the virtual intersection in the real space.

17. A computer-readable recording medium having recorded thereon a program for executing the steps of: acquiring the viewpoint position of a subject photographed using a screen for chromakey compositing installed in real space, the focus position at which the subject should look, and virtual space information representing the position of the screen in virtual three-dimensional space; determining a virtual intersection between a virtual line connecting the viewpoint position and the focus position of the subject in the virtual three-dimensional space and the screen represented by the virtual space information; and controlling the irradiation direction of an irradiation device installed in the real space so as to point to an induction point on the screen corresponding to the virtual intersection in the real space.

Images