Information processing device, display range determination method, and program

Abstract

This information processing device comprises a display range determination unit that, when a display image is switched from a pre-switch viewpoint image to a post-switch viewpoint image among a plurality of viewpoint images capable of displaying a display target from a plurality of viewpoints, said pre-switch viewpoint image corresponding to a pre-switch viewpoint and said post-switch viewpoint image corresponding to a post-switch viewpoint, determines the display range for the post-switch viewpoint image on the basis of viewpoint location information for the post-switch viewpoint, specific target information that is information relating to a specific target within the display target, and gaze direction information that results from estimating the direction of a user's gaze with regard to the display image.

Description

[Technical field]

[0001] The present technology relates to an information processing device that determines a display range, which is an image range to be displayed to a user from within an image displaying a display target, and a display range determination method and program. [Background technology]

[0002] For example, in a display device such as an HMD (Head Mounted Display), the display range of an image is controlled according to the user's line of sight direction estimated from the direction of the user's head, etc. For example, when displaying a wide-angle image such as a panoramic image, the display range in the wide-angle image is changed to follow the change in the user's line of sight direction.

[0003] Some display devices have a function of switching image viewpoints, that is, they switch between and display to the user images seen from a plurality of viewpoints.

[0004] As a related prior art, the following Patent Document 1 can be mentioned, which discloses a technology for switching a virtual viewpoint in a virtual viewpoint image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-92491 A Summary of the Invention [Problem to be solved by the invention]

[0006] Here, let us consider a case where a display device such as an HMD that controls the display range of an image according to the user's line of sight has a viewpoint switching function. When combining control of the display range according to the line of sight and viewpoint switching, if the control of the image display range after the viewpoint switching is not performed appropriately, the user may feel uncomfortable due to the viewpoint switching.

[0007] The present technology has been made in consideration of the above circumstances, and aims to alleviate the discomfort felt by the user when switching viewpoints. [Means for solving the problem]

[0008] An information processing device related to the present technology includes a display range determination unit that, when switching a display image from a source viewpoint image corresponding to a source viewpoint to a destination viewpoint image corresponding to a destination viewpoint, determines a display range for the destination viewpoint image based on viewpoint position information of the destination viewpoint, specific target information which is information related to a specific target in the display target, and gaze direction information which estimates the direction of a user's gaze relative to the display image, for a multiple viewpoint image capable of displaying a display target from multiple viewpoints. Here, the viewpoint of the display object is, for example, the viewpoint of a camera capturing an image of the object if the display object is an object in real space. Alternatively, if the display object is an object in virtual space, the viewpoint of the display object defined in the virtual space is. The specific object information is information related to a specific object in the display object, and for example, if the display object is a live music performance in real space, information on the position, posture, and direction of a specific object such as an artist on stage, and information on the position of a specific space in the display object such as the center of the stage can be exemplified. According to the above configuration, when the display range of an image is controlled according to the user's line of sight, it is possible to determine the display range of the switched-to viewpoint image based on the relationship between the position of the switched-to viewpoint and the position of a specific object among the displayed objects, etc.

[0009] In the information processing device according to the present technology described above, the specific target information may be information including a position of the specific target. This makes it possible to determine the display range of the switching destination viewpoint image based on the relationship between the position of the switching destination viewpoint and the position of the specific target.

[0010] In the information processing device according to the present technology described above, the position of the specific target can be detected by image analysis. This makes it possible to eliminate the need to attach a sensor for detecting the position of a specific object to the specific object when detecting the position of the specific object.

[0011] In the information processing device according to the present technology described above, the position of the specific target can be detected based on a detection signal of a sensor attached to the specific target. This makes it possible to eliminate the need to perform image analysis processing for detecting the position of a specific target.

[0012] In the information processing device according to the present technology described above, the display range determination unit may be configured to use information on a line of sight direction from the switching destination viewpoint in determining the display range. This makes it possible to accommodate cases where the observation direction from the viewpoint is variable, such as by changing the orientation of a camera that captures an image of a display object.

[0013] In the information processing device according to the present technology described above, the display range determination unit can be configured to determine the display range so that the specific target is displayed in a predetermined direction of the user's line of sight. As a result, after switching the viewpoint, if the user directs his / her line of sight in a predetermined direction, he / she can see the specific target in the predetermined direction.

[0014] In the information processing device according to the present technology described above, the display range determination unit can be configured to determine the display range so that the specific target is displayed in a front direction of the user's line of sight. As a result, after switching the viewpoint, if the user turns his / her line of sight forward, he / she can see the specific target in front of him / her.

[0015] In the information processing device according to the present technology described above, the specific object may be a specific space within the display object. This makes it possible to determine the display range of the switching destination viewpoint image based on a space serving as a guide within the display target, such as the center of the stage, as a reference.

[0016] In the information processing device related to the above-mentioned present technology, the display range determination unit can be configured to determine the display range so that the specific target is displayed in the line of sight of the user when switching the viewpoint. This allows the user to view the specific target directly in front of him / her after switching the viewpoint while maintaining the line of sight direction at the time of switching the viewpoint.

[0017] In the information processing device according to the present technology described above, the specific target may be an object that is determined to be looked at by the user in the display image of the switch source viewpoint. This makes it possible to determine the display range of the switched-after viewpoint image based on the object that the user was looking at in the displayed image of the switched-after viewpoint.

[0018] In the information processing device related to the above-mentioned present technology, the display range determination unit can be configured to determine whether or not to use the viewpoint position information, the specific target information, and the gaze direction information to determine the display range, based on the type of target that the user was looking at in the display image from the source viewpoint. This makes it possible to make an appropriate decision as to whether or not to determine the display range using the above three pieces of information based on the type of object the user is looking at, for example by determining not to determine the display range using the viewpoint position information, specific object information, and gaze direction information when the user is looking at an object outside the stage, such as the audience seating area.

[0019] In the information processing device related to the above-mentioned present technology, the display range determination unit can be configured to determine the display range using the viewpoint position information, the specific target information, and the line of sight direction information when it is determined that the user is looking at a specific object in the display image of the original viewpoint. This makes it possible to avoid determining the display range using the three pieces of information when the user is looking at an object other than a specific object, such as an audience seating area.

[0020] In addition, a display range determination method related to the present technology is a display range determination method in which, when an information processing device switches a display image from a source viewpoint image corresponding to a source viewpoint to a destination viewpoint image corresponding to a destination viewpoint for a multiple viewpoint image capable of displaying a display object from multiple viewpoints, the display range of the destination viewpoint image is determined based on viewpoint position information of the destination viewpoint, specific target information which is information related to a specific target in the display object, and gaze direction information which estimates the direction of a user's gaze relative to the display image.

[0021] With such a display range determination method, the same effect as that of the information processing device according to the present technology described above can be obtained.

[0022] Furthermore, the program related to the present technology is a program readable by a computer device, and when switching a display image from a source viewpoint image corresponding to a source viewpoint to a destination viewpoint image corresponding to a destination viewpoint for a multiple viewpoint image in which a display target can be displayed from multiple viewpoints, the program causes the computer device to execute a process of determining a display range for the destination viewpoint image based on viewpoint position information of the destination viewpoint, specific target information which is information related to a specific target in the display target, and gaze direction information which estimates the direction of a user's gaze relative to the display image.

[0023] Such a program realizes the information processing device according to the present technology described above. [Brief description of the drawings]

[0024] [Figure 1] 1 is a diagram illustrating an example of the configuration of a display system including an information processing device according to an embodiment of the present technology. [Diagram 2] FIG. 2 is a diagram showing an example of camera arrangement in the embodiment. [Diagram 3] 11A and 11B are diagrams illustrating control of a display range according to a line of sight direction. [Figure 4] FIG. 13 is a diagram for considering a method for determining a display range after switching a viewpoint. [Diagram 5] FIG. 11 is an explanatory diagram of a display range determination method as Pattern 1 in the embodiment. [Figure 6] FIG. 11 is an explanatory diagram of a display range determination method as Pattern 2 in the embodiment. [Figure 7] FIG. 11 is an explanatory diagram of a display range determination method as Pattern 3 in the embodiment. [Figure 8] 11A to 11C are diagrams for considering the viewpoint tracking control in the cases of patterns 1 and 3 in the embodiment. [Figure 9] 11A to 11C are diagrams for considering the viewpoint tracking control in the case of pattern 2 in the embodiment. [Figure 10] 11 is a flowchart illustrating an example of a specific process procedure to be executed to realize a display range determination method according to an embodiment. [Figure 11] 13 is a flowchart showing a process as a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Hereinafter, with reference to the accompanying drawings, embodiments of the present technology will be described in the following order. <1. Display system as an embodiment> <2. Display range determination method as an embodiment> <3. Processing Procedure> <4. Modifications> <5. Program> <6. Summary of the embodiment> <7. This Technology>

[0026] <1. Display system as an embodiment> FIG. 1 shows an example of the configuration of a display system including a playback device 1 that is an embodiment of an information processing device according to the present technology. As shown in the figure, the display system includes a playback device 1, a plurality of cameras 2, a communication device 3, and a server device 4.

[0027] In the display system of this example, images captured by a camera 2 of a live music performance held in real space are distributed as display content to a playback device 1. In particular, in this example, it is assumed that images as display content are distributed live. The playback device 1 is, for example, an HMD (Head Mounted Display), and is used while being worn on the user's head. A captured image serving as display content is captured from a camera 2 into a communication device 3, and then uploaded to a server device 4, and distributed from the server device 4 to a playback device 1.

[0028] Here, in the display system of this example, the live music performance that each camera 2 captures can be said to be an object displayed to the user as display content. In this specification, an object displayed to the user as display content in this way is called a "display object."

[0029] Each camera 2 is an imaging device that captures images using an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. Each camera 2 is placed at a different position in the live venue and captures images from different viewpoints.

[0030] FIG. 2 shows an example of the arrangement of the cameras 2. In this example, the display subject is a live music concert, and as shown in the figure, the cameras 2 are positioned to capture images of the stage from different positions. In this example, the cameras 2 are positioned to observe the stage from the front, from the left side, and from the right side. In the figure, the field of view Fv of each camera 2 is shown in schematic form along with the position of each camera 2. In this example, the field of view Fv of each camera 2 is, for example, 180 degrees, which is a relatively wide range. In other words, the camera captured image can be said to be a wide-angle image. Although the case where the number of cameras 2 is three is illustrated here, the number of cameras 2 may be at least two or more.

[0031] The communication device 3 includes an object position detection unit 31 , a control unit 32 , and a communication unit 33 . The object position detection unit 31 detects the position of a specific object in the display target. Specifically, in this example, the object position detection unit 31 detects the position of a person who is an artist performing live music. The position of the specific object as an artist can be detected by various sensors. For example, the position of the specific object can be detected by image analysis of an image captured by one or more cameras (image sensors) installed in the venue that capture the specific object. The position of the specific object can also be detected by using a depth sensor. Alternatively, the position of the specific object can be detected based on a detection signal from a physical sensor, such as an acceleration sensor, a gyro sensor, or a GNSS (Global Navigation Satellite System) sensor, attached to the specific object as an artist. The position of the specific object can also be detected based on both the image analysis and the detection signal from a physical sensor. Information on the position of a specific object detected by object position detection unit 31 will be referred to as "object position information" hereinafter.

[0032] The communication unit 33 performs wired or wireless data communication with an external device. The communication unit 33 in this example performs data communication with the server device 4 via the network NT.

[0033] Here, various examples of the configuration of the network NT are assumed. For example, the Internet, an intranet, an extranet, a LAN (Local Area Network), a CATV (Community Antenna TeleVision) communication network, a virtual private network, a telephone line network, a mobile communication network, a satellite communication network, etc. are assumed. In addition, various examples of the transmission media constituting all or part of the network NT are assumed. For example, wired media such as IEEE (Institute of Electrical and Electronics Engineers) 1394, USB (Universal Serial Bus), power line carrier, and telephone line, as well as wireless media such as infrared such as IrDA (Infrared Data Association), Bluetooth (registered trademark), 802.11 wireless, a mobile phone network, a satellite line, and a terrestrial digital network can be used.

[0034] The control unit 32 is configured with a microcomputer having a memory device such as a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and performs overall control of the communication device 3. In particular, the control unit 32 performs control for uploading images captured by each camera 2 (camera captured images) and object position information detected by the object position detection unit 31 to the server device 4 via the communication unit 33. The control unit 32 also controls the uploading of camera position information, which is information indicating the placement position of each camera 2, to the server device 4 together with the camera-captured images and object position information.

[0035] The playback device 1 includes a communication unit 11, a control unit 12, a display unit 13, a gaze direction estimation unit 14, and an operation unit 15. The communication unit 11 performs wired or wireless data communication with external devices, and in this example in particular performs data communication with the server device 4 via the network NT.

[0036] The display unit 13 is configured to have a display capable of displaying images, such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display, and displays various types of information to the user based on instructions from the control unit 12.

[0037] The gaze direction estimating unit 14 estimates the gaze direction of the user. Specifically, gaze direction estimating unit 14 in this example has a physical sensor, such as an acceleration sensor or a gyro sensor, that detects the movement of the user's head, and estimates the gaze direction from the amount of movement of the user's head. Here, when the head of a user wearing the playback device 1 as an HMD is rotated, for example, to the right from a direction defined as the front direction (direction of 0 degrees (deg)), the direction of the user's line of sight can be regarded as having changed to the right. Therefore, the direction of the user's line of sight can be estimated from the amount of movement of the head. In this specification, the "gaze direction" of a user does not strictly refer to the direction of the eyeball (pupil direction), but broadly includes directions indirectly estimated from the direction of the user's head, etc., as described above.

[0038] The operation unit 15 includes input devices such as various hardware keys, an operation dial, and a touch panel, detects the user's input operation, and transmits it to the control unit 12. The control unit 12 determines the operation of the playback device 1 in response to the user's input operation, and controls each unit to perform the required operation.

[0039] The control unit 12 is configured with a microcomputer having a CPU and memory devices such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and performs overall control of the communication device 3. In particular, the control unit 12 performs processing related to the display of camera-captured images distributed from the server device 4. Specifically, as a process related to the display of a camera-captured image, the control unit 12 performs a process of controlling the display range, which is the image range of the entire range of the camera-captured image that is displayed to the user via the display unit 13, in accordance with the gaze direction estimated by the gaze direction estimation unit 14.

[0040] FIG. 3 is an explanatory diagram of such control of the display range according to the line of sight. The control of the display range according to the line of sight direction is based on the forward direction defined for the user's line of sight and the forward direction in the field of view Fv of the camera 2. Here, as shown in Fig. 3A, the forward direction in the line of sight direction and the forward direction in the field of view Fv are both set to 0 degrees. 3A, when the line of sight is in the 0 degree direction, a predetermined range centered on the 0 degree direction in the field of view Fv is cut out from the image captured by the camera as the image range to be displayed on the display unit 13. This range cut out as the target image range for display is referred to as the "cut-out range Rc." The front direction (the center direction of the angle of view) in the cut-out range Rc is denoted as the "cut-out direction Dc" as shown in the figure. The cut-out direction Dc can be said to be the direction pointing to the center of the image in the cut-out range Rc, that is, the display image.

[0041] In controlling the display range according to the line of sight direction, when the line of sight direction is 0 degrees as described above, the display range is set to a predetermined range centered on the 0 degree direction in the field of view Fv. This is because it is difficult to imagine a user looking at a displayed image for a long period of time with their line of sight deviated from the front direction, and it can be assumed that in most cases the user faces the front direction. As shown in Fig. 3B, when the user's line of sight changes, the cut-out direction Dc is changed to follow the change in the line of sight. Fig. 3B shows an example in which the line of sight changes from the 0 degree direction to the right by an angle a, and in this case, the cut-out direction Dc is similarly changed from the 0 degree direction to the right by an angle a. This control of changing the cut-out direction Dc to follow changes in the line of sight, in other words, the control of changing the cut-out range Rc (display range) to follow changes in the line of sight, is referred to as "gaze tracking control" of the display range. Here, among the gaze tracking control of the display range, the control in which the cut-out direction Dc when the gaze direction is 0 degrees, as in the example of Figure 3, is set to the 0 degree direction in the field of view Fv, will be referred to as ``normal gaze tracking control.''

[0042] 1, the control unit 12 performs the gaze tracking control of the display range as well as the viewpoint switching control of the displayed image as a control related to the display of the camera captured image. The viewpoint switching control can be said as a control to switch the image to be displayed on the display unit 13 among the camera captured images by each camera 2. In the viewpoint switching control, the control unit 12 may receive each camera captured image from the server device 4 and select an image to be displayed on the display unit 13 from among them, or may selectively receive only the camera captured image by the camera 2 as the switched-to viewpoint from the server device 4 and display it on the display unit 13.

[0043] Furthermore, in response to switching of the viewpoint, the control unit 12 may perform control that is different from normal viewpoint tracking control as viewpoint tracking control for the camera-captured image of the switched-to viewpoint, but this will be described later.

[0044] <2. Display range determination method as an embodiment> First, with reference to FIG. 4, a method for determining the display range after switching the viewpoint will be considered. Here, it is assumed that normal gaze tracking control is performed as the gaze tracking control of the display range. For example, suppose that an image captured by the front camera 2 in Fig. 4 is displayed on the display unit 13. At this time, suppose that the user shifts the line of sight from the front (0 degrees) to the right as shown in the figure in order to capture a specific object (an artist in this example) located to the right of the stage in front of the field of view (i.e., the center of the displayed image). The amount of deviation of the line of sight from the front in this case is assumed to be x degrees. In the following description, the position of a specific object on the stage will be represented as position Po, as shown in the figure.

[0045] From the above state, for example, it is assumed that the viewpoint is switched to the left camera 2. Since the user's line of sight is shifted by x degrees to the right from the front direction, the displayed image after the viewpoint is switched is an image cut out with the cut-out direction Dc being the direction shifted by x degrees to the right from the front direction in the field of view Fv of the left camera 2.

[0046] On the other hand, if the viewpoint is switched to the right-side camera 2, the user's line of sight before the viewpoint is switched is similarly shifted x degrees to the right from the front direction, so the displayed image after the viewpoint is switched will be an image cut out in the cut-out direction Dc, which is shifted x degrees to the right from the front direction in the field of view Fv of the right-side camera 2.

[0047] As can be seen by referring to the cut-out directions Dc for the left camera 2 and the right camera 2, under normal gaze tracking control, after switching the viewpoint, an object different from the object seen before switching the viewpoint may be seen, and there is a risk that the object captured in the displayed image before and after switching the viewpoint may be significantly different. This may cause the user to feel uncomfortable when switching the viewpoint.

[0048] Therefore, in this embodiment, as a method for determining the display range when switching the viewpoint, rather than determining the display range based only on information about the user's gaze direction as in normal gaze tracking control, we propose the following determination method that uses object position information of a specific object and camera position information.

[0049] FIG. 5 is an explanatory diagram of a display range determination method as pattern 1. In FIG. Pattern 1 determines the display range of the camera-captured image from the switched-to viewpoint so that a specific object is displayed in the user's line of sight when the user's line of sight is forward. In other words, after switching the viewpoint, if the user's line of sight is directed forward, the specific object can be seen in front of the field of view (i.e., in the center of the displayed image). Such a pattern 1 can be expressed as determining the display range of the switched-to viewpoint image so that a specific object is displayed in the front direction of the user's line of sight. The "front direction of the user's line of sight" referred to here can be rephrased as the front direction defined with respect to the user's line of sight. In this example, the display range is determined so that the specific object is displayed in the front direction of the user's line of sight, but the position at which the specific object is displayed does not necessarily have to be exactly the front direction of the user's line of sight, and may be any position that can be considered as the front direction. Alternatively, the position at which the specific object is displayed is not limited to the front direction of the user's line of sight, and may be a predetermined direction, such as a direction shifted by a predetermined angle from the front direction. As a result, after the viewpoint is switched, the user can see the object that was viewed before the viewpoint is switched in front by directing the line of sight in a predetermined direction. In other words, the object that was viewed before the viewpoint is switched can be seen in front by directing the line of sight in a predetermined direction. Therefore, it is possible to alleviate the discomfort felt by the user due to the viewpoint switching.

[0050] Here, in this example, the specific object present at position Po is the object the user was looking at in the display image from the original viewpoint. In pattern 1 of this example (the same goes for pattern 2 described below), the display range of the camera-captured image from the new viewpoint is determined based on the specific object the user was looking at in the display image from the original viewpoint.

[0051] FIG. 6 is an explanatory diagram of a display range determination method as pattern 2. In FIG. Pattern 2 determines the display range of the camera-captured image of the switched-to viewpoint so that a specific object is displayed ahead of the user's line of sight when the user's line of sight coincides with the line of sight direction at the time of switching the viewpoint. In other words, after switching the viewpoint, the user can see a specific object in front of his or her field of vision while maintaining the line of sight direction at the time of switching the viewpoint. Such pattern 2 can be expressed as determining the display range of the switched-to viewpoint image so that a specific object is displayed in the line of sight direction of the user at the time of switching the viewpoint.

[0052] FIG. 7 is an explanatory diagram of a display range determination method as pattern 3. In FIG. Pattern 3 is a method similar to pattern 1, and determines the display range of the camera-captured image from the switched-to viewpoint so that a specific space in the display target is displayed ahead of the user's line of sight when the user's line of sight is in the forward direction. Here, in this example, the specific space is the center of the stage. In the figure, the position of the center of the stage is indicated as position Pc. Here, it is conceivable that the control unit 12 acquires the information on the position Pc from the communication device 3 via the server device 4, for example.

[0053] As in the above-mentioned patterns 1 to 3, in the display range determination method of the embodiment, the display range is determined based on the position of a specific object or a specific space in the display target. Therefore, the display range determination of the embodiment uses object position information of the specific object. Furthermore, when determining the display range of the camera 2 of the switching destination viewpoint based on the position of a specific object or specific space, it is necessary to know in which direction the specific object or specific space exists within the field of view Fv of the camera 2 of the switching destination viewpoint. For this reason, the display range determination in the embodiment uses camera position information about the camera 2 of the switching destination viewpoint.

[0054] Next, the viewpoint tracking control when the display range is determined according to the above-mentioned pattern 1 to pattern 3 will be considered. FIG. 8 is a diagram for considering the viewpoint tracking control in the cases of patterns 1 and 3. As described above, when determining the display range based on the position of a specific object or a specific space, it is necessary to know in which direction the specific object or specific space exists within the field of view Fv of the camera 2 at the switched-to viewpoint. In Fig. 8, the angle of the direction in which the specific object (position Po) or specific space (position Pc) exists within the field of view Fv of the camera 2 at the switched-to viewpoint is shown as "angle α". This angle α can be obtained based on the camera position information about the camera 2 of the switching destination viewpoint and the object position information of a specific object or the position information of a specific space.

[0055] In patterns 1 and 3, a specific object or a specific space may be displayed in the center of the display image when the gaze direction is in the 0 degree direction. In other words, when the gaze direction is in the 0 degree direction, the direction of α degrees may be set as the cut-out direction Dc. Therefore, in gaze tracking control in the cases of patterns 1 and 3, the cut-out direction Dc may be determined to be the direction specified by the angle obtained by adding an offset of angle α to the gaze direction angle estimated by gaze direction estimation unit 14.

[0056] FIG. 9 is a diagram for considering the viewpoint tracking control in the case of pattern 2. In FIG. In pattern 2, the display range is determined based on the position of a specific object, and therefore angle α, which is the angle of the direction in which a specific object (position Po) exists within the field of view Fv of camera 2 at the switched-to viewpoint, is used as shown in Fig. 9A. Furthermore, in pattern 2, the display range is determined so that a specific object is displayed ahead of the user's line of sight when the user's line of sight coincides with the line of sight at the time of switching the viewpoint, and therefore angle x, which indicates the line of sight at the time of switching the viewpoint as shown in Fig. 9B, is also used for viewpoint tracking control. In pattern 2, when the user's gaze direction is in the direction of angle x, the direction of α degrees may be set as the cut-out direction Dc. Therefore, in the gaze tracking control in the case of pattern 2, the cut-out direction Dc may be determined to be a direction specified by an angle obtained by adding an offset represented by "α-x" to the gaze direction angle estimated by gaze direction estimation unit 14.

[0057] <3. Processing Procedure> Next, a specific process for implementing the display range determination method according to the embodiment described above will be described with reference to the flowchart of FIG. In this example, the process shown in FIG. 10 is executed by the control unit 12 based on a program stored in a predetermined storage device such as a ROM.

[0058] First, in step S101, the control unit 12 waits for a viewpoint switching instruction. For example, the control unit 12 waits for a viewpoint switching instruction given by the user to the playback device 1 through an operation on the operation unit 15. The viewpoint switching instruction also includes information about a switching destination viewpoint. In addition, viewpoint switching is not limited to being performed based on user operation, but may also be performed in response to the establishment of certain conditions other than operation, such as being performed based on switching instruction information added to a camera-captured image, or at the time of a specific scene switch based on the results of scene analysis of a camera-captured image.

[0059] When a viewpoint switching instruction has been issued, the control unit 12 proceeds to step S102 and calculates an angle α of a direction in which a specific target exists within the field of view Fv of the switched-to viewpoint. When the display range determination method of patterns 1 and 2 is adopted, in step S102, an angle α of a direction in which a specific object exists within the field of view Fv of camera 2 at the switched-to viewpoint is calculated. When the display range determination method of pattern 3 is adopted, in step S102, an angle α of a direction in which a specific space exists within the field of view Fv of camera 2 at the switched-to viewpoint is calculated. In calculating this angle α, the control unit 12 uses the camera position information of the switched-to viewpoint and the object position information of the specific object or the position information of the specific space.

[0060] In step S103 following step S102, control unit 12 calculates an offset for the viewpoint direction based on angle α. When the display range determination method of patterns 1 and 3 is adopted, in step S103, angle α is set as an offset to be added to the angle of the viewing direction estimated by viewing direction estimation unit 14. On the other hand, when the display range determination method of pattern 2 is adopted, in step S103, an offset expressed as "α-x" is calculated based on angle x and angle α of the viewing direction at the time of switching the viewpoint.

[0061] In step S104 following step S103, the control unit 12 performs output processing of the cut-out image. That is, the direction specified by the angle obtained by adding the offset obtained in step S102 to the angle of the gaze direction estimated by the gaze direction estimation unit 14 is determined as the cut-out direction Dc, and an image of a predetermined range centered on the cut-out direction Dc is cut out from the camera image of the switching destination viewpoint. Then, the cut-out image is output to the display unit 13.

[0062] After executing the process of step S104, the control unit 12 ends the series of processes shown in FIG.

[0063] In addition, when patterns 1 and 2 are adopted, if the position of a specific object moves, the angle α is successively recalculated based on the object position information of the specific object, and the offset value to be added to the angle of the line of sight direction is successively updated.

[0064] <4. Modifications> The embodiment is not limited to the specific example described above, and various modified configurations can be adopted. For example, it may be possible to determine whether or not to perform display range determination as in the embodiments exemplified as patterns 1, 2, and 3, based on the type of object that the user was looking at in the display image from the switch source viewpoint.

[0065] FIG. 11 is a flowchart showing a modified example of processing for determining whether or not to determine the display range according to the embodiment. Note that the same processes as those explained in FIG. 10 are denoted by the same reference numerals and the explanation thereof will be omitted.

[0066] In this case, when it is determined in step S101 that a viewpoint switching instruction has been issued, the control unit 12 proceeds to step S201 to determine whether or not to perform gaze offset control. Specifically, it is a determination as to whether or not to perform display range determination using three pieces of information, namely, the camera position information of the switched-to viewpoint, the position information of the specific target, and the gaze direction information estimated by the gaze direction estimation unit 14, as exemplified in patterns 1, 2, and 3. As the determination process of this step S201, the control unit 12 performs a determination based on the type of target that the user was looking at in the display image of the switched-to viewpoint. Specifically, in this example, it is determined whether or not the target that the user was looking at in the display image of the switched-to viewpoint is a specific object (an artist in this example). Here, whether or not the user has seen a specific object in the display image of the switching source viewpoint can be determined by performing image analysis of the display image. Alternatively, whether or not a specific object is located within the field of view Fv of the camera 2 of the switching source viewpoint may be determined based on the camera position information of the switching source viewpoint and the object position information of the specific object. In this case, it is possible to determine whether or not the specific object has been seen by imposing conditions such as not only that the specific object is located within the switching source viewpoint image or within the switching source field of view Fv, but also that the image size of the specific object is a predetermined size or more, or that the specific object is within a certain range within the field of view Fv (for example, the central range of the field of view Fv).

[0067] In step S201, if the target the user has been looking at through the display image of the original viewpoint is a specific object, the control unit 12 obtains a determination result that the line-of-sight offset control is to be performed. In this case, the control unit 12 advances the process to step S102. This causes the process for determining the display range as the embodiment exemplified as patterns 1, 2, and 3 to be performed.

[0068] On the other hand, in step S201, if the target that the user was looking at through the display image of the original viewpoint is not a specific object, the control unit 12 obtains a determination result that the gaze offset control is not to be performed. In this case, the control unit 12 advances the process to step S202 and selects normal gaze tracking control.

[0069] The above-described process can prevent the user from feeling uncomfortable when switching the viewpoint due to the display range being determined indiscriminately using the three pieces of information.

[0070] The method of determining whether or not to perform the display range determination according to the embodiment based on the type of object the user was looking at through the display image from the switching source viewpoint is not limited to the method of determining whether or not the user was looking at a specific object as described above. For example, it is also possible to adopt a method of determining not to perform the display range determination according to the embodiment when the user was looking at an object outside the stage, such as the audience seating area.

[0071] Here, in the above description, an HMD has been given as an example of the playback device 1, but the device form of the playback device 1 is not limited to an HMD. For example, device forms such as a smartphone, a tablet terminal, or a large display device such as a TV (Television) receiver are also possible. For example, in the case of a portable device form such as a smartphone or a tablet terminal, the user's line of sight direction can be estimated as the orientation of the device. The orientation of the device can be detected by a physical sensor such as an acceleration sensor or a gyro sensor. Furthermore, in the case of a stationary device such as a large display device such as a TV (Television) receiver, the user's line of sight can be estimated by eye tracking.

[0072] In the above explanation, it has been assumed that the orientation of the camera 2 is fixed, but if changes in the orientation of the camera are permitted, the display range of the switched-to viewpoint image is determined using not only the position information of the camera 2 as the switched-to viewpoint, but also the orientation information of the camera 2 (pan, tilt, roll directions). The orientation information of the camera 2 can be detected, for example, by an acceleration sensor or a gyro sensor provided on the camera 2. Alternatively, the orientation information of the camera 2 can be detected by calibration using a board on which a predetermined figure is drawn. Here, the orientation information of the camera 2 can be said to be the direction of the line of sight from the viewpoint.

[0073] In the above explanation, the viewpoint switching and the display range determination have been mainly described in the pan direction (left and right direction), but the viewpoint switching and the display range determination can also be performed in the tilt direction in a similar manner. In the case of the tilt direction, it is possible to use not only the position information of a specific object, but also the posture and orientation information of the specific object. For example, when it is desired to capture the front of the face of an artist, which is a specific object, the posture and orientation information of the specific object is used to determine the display range.

[0074] In addition, in the explanation so far, an example has been given of detecting the position of a specific object using a sensor, but if the object to be displayed is, for example, a classical concert where the artist is seated in a predetermined fixed position, information on that predetermined fixed position can also be used.

[0075] It is also possible to switch the method of determining the display range depending on the type of the playback device 1. For example, it is possible to change the display range after switching the viewpoint depending on whether the playback device 1 is an HMD or another display device (tablet terminal or TV receiver). Specifically, when a specific object is located near the edge of the image from the switched-to viewpoint, in the case of an HMD, it is easy to see the specific object if it is located in the center of the displayed image, but in the case of a tablet terminal or the like, it is undesirable for the edge to become a non-image portion and be filled in with a predetermined color such as black, so it is possible to determine the display range so that the edge of the display frame coincides with the edge of the image from the switched-to viewpoint.

[0076] In addition, in the explanation so far, examples have been given in which the specific object or specific space is an object or space in real space, but the display range determination method of the embodiment can also be suitably applied to cases in which the display target is an object or space in a virtual space such as a VR (Virtual Reality) space or an AR (Augmented Reality) space.

[0077] <5. Program> The information processing device (playback device 1) according to the embodiment has been described above, and the program according to the embodiment is a program that causes a computer device such as a CPU to execute the processing of the playback device 1.

[0078] The program of the embodiment is a program readable by a computer device, and when switching a display image from a source viewpoint image corresponding to a source viewpoint to a destination viewpoint image corresponding to a destination viewpoint for a multiple viewpoint image in which a display object can be displayed from multiple viewpoints, the program causes the computer device to execute a process of determining a display range for the destination viewpoint image based on viewpoint position information of the destination viewpoint, specific target information which is information relating to a specific target in the display object, and gaze direction information which estimates the direction of the user's gaze relative to the display image. In other words, this program corresponds to a program that causes a computer device to execute the processes described with reference to FIG. 10, FIG. 11, and the like.

[0079] Such a program can be stored in advance in a storage medium readable by a computer device, such as a ROM, a hard disk drive (HDD), a solid state drive (SSD), etc. Alternatively, the program can be temporarily or permanently stored (stored) in a removable storage medium such as a semiconductor memory, a memory card, an optical disk, a magneto-optical disk, a magnetic disk, etc. Furthermore, such a removable storage medium can be provided as a so-called package software. In addition, such a program can be installed from a removable storage medium onto a personal computer or the like, or can be downloaded from a download site to a required information processing device such as a smartphone via a network such as a LAN or the Internet.

[0080] <6. Summary of the embodiment> As described above, the information processing device (playback device 1) as an embodiment includes a display range determination unit (control unit 12) that, when switching a display image from a source viewpoint image corresponding to a source viewpoint to a destination viewpoint image corresponding to a destination viewpoint for a multiple viewpoint image in which a display object can be displayed from multiple viewpoints, determines a display range for the destination viewpoint image based on viewpoint position information of the destination viewpoint, specific target information which is information relating to a specific target in the display object, and gaze direction information which estimates the direction of the user's gaze relative to the display image. Here, the viewpoint of the display object is, for example, the viewpoint of a camera capturing an image of the object if the display object is an object in real space. Alternatively, if the display object is an object in virtual space, the viewpoint of the display object defined in the virtual space is. The specific object information is information related to a specific object in the display object, and for example, if the display object is a live music performance in real space, information on the position, posture, and direction of a specific object such as an artist on stage, and information on the position of a specific space in the display object such as the center of the stage can be exemplified. According to the above configuration, when the display range of an image is controlled according to the user's line of sight, it is possible to determine the display range of the switched-to viewpoint image based on the relationship between the position of the switched-to viewpoint and the position of a specific object among the displayed objects, etc. Therefore, it is possible to prevent the object captured in the displayed image from being significantly different before and after the viewpoint is switched, and to alleviate the sense of discomfort felt by the user when the viewpoint is switched.

[0081] In addition, in the information processing device as the embodiment, the specific target information is information including the position of the specific target. This makes it possible to determine the display range of the switching destination viewpoint image based on the relationship between the position of the switching destination viewpoint and the position of the specific target. Therefore, it is possible to prevent the object captured in the displayed image from being significantly different before and after the viewpoint is switched, and to alleviate the sense of discomfort felt by the user when the viewpoint is switched.

[0082] In addition, in the information processing device according to the embodiment, the position of the specific target is detected by image analysis. This makes it possible to eliminate the need to attach a sensor for detecting the position of a specific object to the specific object when detecting the position of the specific object. Therefore, for example, when the specific target is a specific object such as an artist on a stage, the effort of attaching a sensor to the specific object can be eliminated.

[0083] In addition, in the information processing device as the embodiment, the position of a specific target is detected based on a detection signal from a sensor attached to the specific target. This makes it possible to eliminate the need to perform image analysis processing for detecting the position of a specific target. Therefore, the processing load associated with detecting the position of a specific target can be reduced.

[0084] Furthermore, in the information processing device as the embodiment, the display range determination unit uses information on the line of sight direction from the switched-to viewpoint in determining the display range. This makes it possible to accommodate cases where the direction of the line of sight from the viewpoint is variable, such as by varying the orientation of a camera that captures an image of a display object. This allows the angle at which the display object is viewed from the viewpoint to be changed, improving the degree of freedom in the image content displayed to the user.

[0085] Furthermore, in the information processing device as the embodiment, the display range determination unit determines the display range so that the specific target is displayed in a predetermined direction of the user's line of sight (see patterns 1 and 3). As a result, after switching the viewpoint, if the user directs his / her line of sight in a predetermined direction, he / she can see a specific target in the predetermined direction. Therefore, the sense of discomfort felt by the user due to viewpoint switching can be alleviated.

[0086] Furthermore, in the information processing device as the embodiment, the display range determination unit determines the display range so that the specific target is displayed in front of the user's line of sight (see patterns 1 and 3). As a result, after switching the viewpoint, if the user turns his / her line of sight forward, he / she can see the specific target in front of him / her. Therefore, the sense of discomfort felt by the user due to viewpoint switching can be alleviated.

[0087] Furthermore, in the information processing device as the embodiment, the specific object is a specific space in the display object (see pattern 3). This makes it possible to determine the display range of the switching destination viewpoint image based on a space serving as a guide within the display target, such as the center of the stage, as a reference. Therefore, it is possible to prevent the user from becoming unclear as to which position in the display object is visible after switching the viewpoint, and to alleviate the sense of discomfort felt by the user when switching the viewpoint.

[0088] Furthermore, in the information processing device as the embodiment, the display range determination unit determines the display range so that the specific target is displayed in the line of sight of the user when switching the viewpoint (see pattern 2). This allows the user to view the specific target directly in front of him / her after switching the viewpoint while maintaining the line of sight direction at the time of switching the viewpoint. Therefore, it is possible to prevent the user from becoming unclear as to which position in the display object is visible after switching the viewpoint, and to alleviate the sense of discomfort felt by the user when switching the viewpoint.

[0089] Furthermore, in the information processing device according to the embodiment, the specific target is an object that is determined to be looked at by the user in the display image from the switch source viewpoint. This makes it possible to determine the display range of the switched-after viewpoint image based on the object that the user was looking at in the displayed image of the switched-after viewpoint. Therefore, it is possible to ensure that the object that the user has been looking at continues to be captured in the displayed image even after the viewpoint has been switched, and it is possible to alleviate the sense of discomfort felt by the user due to the switching of the viewpoint.

[0090] Furthermore, in an information processing device as an embodiment, the display range determination unit determines whether or not to determine the display range using viewpoint position information, specific target information, and gaze direction information, based on the type of target that the user was looking at in the display image from the original viewpoint. This makes it possible to make an appropriate decision as to whether or not to determine the display range using the above three pieces of information based on the type of object the user is looking at, for example by determining not to determine the display range using the viewpoint position information, specific object information, and gaze direction information when the user is looking at an object outside the stage, such as the audience seating area. Therefore, it is possible to prevent the user from feeling uncomfortable when switching the viewpoint due to the display range being determined indiscriminately using the three pieces of information.

[0091] Furthermore, in the information processing device of the embodiment, when it is determined that the user is looking at a specific object in the display image from the original viewpoint, the display range determination unit determines the display range using the viewpoint position information, the specific object information, and the gaze direction information (see FIG. 11). This makes it possible to avoid determining the display range using the three pieces of information when the user is looking at an object other than a specific object, such as an audience seating area. Therefore, it is possible to prevent the user from feeling uncomfortable when switching the viewpoint due to the display range being determined indiscriminately using the three pieces of information.

[0092] Furthermore, a display range determination method as an embodiment is a display range determination method in which, when an information processing device switches a display image from a source viewpoint image corresponding to a source viewpoint to a destination viewpoint image corresponding to a destination viewpoint for a multiple viewpoint image in which a display object can be displayed from multiple viewpoints, the information processing device determines a display range for the destination viewpoint image based on viewpoint position information of the destination viewpoint, specific target information which is information relating to a specific target in the display object, and gaze direction information which estimates the direction of the user's gaze relative to the display image. With this display range determination method, it is possible to obtain the same functions and effects as those of the information processing device according to the above embodiment.

[0093] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0094] <7. This Technology> The present technology can also be configured as follows. (1) With regard to a multiple viewpoint image capable of displaying a display object from multiple viewpoints, when switching a display image from a source viewpoint image corresponding to a source viewpoint to a destination viewpoint image corresponding to a destination viewpoint, the display range determination unit determines a display range of the destination viewpoint image based on viewpoint position information of the destination viewpoint, specific target information which is information related to a specific target in the display object, and line of sight direction information which estimates a direction of a user's line of sight with respect to the display image. Information processing device. (2) The specific target information is information including a position of the specific target. The information processing device according to (1). (3) The position of the specific object is detected by image analysis. The information processing device according to (2). (4) The position of the specific object is detected based on a detection signal from a sensor attached to the specific object. The information processing device according to (2) or (3). (5) The display range determination unit uses information on a line of sight direction from the switching destination viewpoint in determining the display range. The information processing device according to any one of (1) to (4). (6) The display range determination unit determines the display range such that the specific target is displayed in a predetermined direction of the user's line of sight. The information processing device according to any one of (1) to (5). (7) The display range determination unit determines the display range so that the specific target is displayed in a front direction of a line of sight of the user. The information processing device according to (5). (8) The specific object is a specific space in the display object. The information processing device according to (6) or (7). (9) The display range determination unit determines the display range so that the specific object is displayed in a line of sight of the user when a viewpoint is switched. The information processing device according to any one of (1) to (5). (10) The specific object is an object that is determined to be looked at by the user in the display image of the switch source viewpoint. The information processing device according to any one of (1) to (7) and (9). (11) The display range determination unit determines whether or not to determine the display range using the viewpoint position information, the specific target information, and the line of sight direction information, based on a type of target that the user was looking at in the display image from the switching source viewpoint. The information processing device according to any one of (1) to (10). (12) The display range determination unit determines the display range using the viewpoint position information, the specific object information, and the line of sight direction information when it is determined that the user is looking at a specific object in the display image of the switch source viewpoint. The information processing device according to (11). (13) An information processing device, With respect to a multiple viewpoint image capable of displaying a display object from multiple viewpoints, when switching a display image from a source viewpoint image corresponding to a source viewpoint to a destination viewpoint image corresponding to a destination viewpoint, a display range of the destination viewpoint image is determined based on viewpoint position information of the destination viewpoint, specific target information which is information related to a specific target in the display object, and line of sight direction information which estimates a direction of a user's line of sight with respect to the display image. How to determine the display range. (14) A computer readable program, With respect to a multiple viewpoint image capable of displaying a display target from multiple viewpoints, when switching a display image from a source viewpoint image corresponding to a source viewpoint to a destination viewpoint image corresponding to a destination viewpoint, the computer device is caused to execute a process of determining a display range of the destination viewpoint image based on viewpoint position information of the destination viewpoint, specific target information which is information related to a specific target in the display target, and line of sight direction information which estimates a direction of a user's line of sight with respect to the display image. program. [Explanation of symbols]

[0095] 1 Playback device 2 Camera 3. Communications equipment 4. Server equipment NT Network 11 Communications Department 12 Control section 13 Display section 14 Gaze direction estimation unit 15 Control section 31 Object position detection unit 32 Control section 33 Communications Department Fv field of view Rc cutting range Dc cutting direction

Claims

1. The display device includes a display range determination unit that, when switching a display image from a source viewpoint image corresponding to a source viewpoint to a destination viewpoint image corresponding to a destination viewpoint, determines a display range for the destination viewpoint image based on viewpoint position information of the destination viewpoint, specific target information that is information related to a specific target in the display object, and line of sight direction information that estimates a direction of a user's line of sight with respect to the display image. Information processing device.

2. The specific target information is information including a position of the specific target. The information processing device according to claim 1 .

3. The position of the specific object is detected by image analysis. The information processing device according to claim 2 .

4. The position of the specific object is detected based on a detection signal from a sensor attached to the specific object. The information processing device according to claim 2 .

5. The display range determination unit uses information on a line of sight direction from the switching destination viewpoint in determining the display range. The information processing device according to claim 1 .

6. The display range determination unit determines the display range such that the specific target is displayed in a predetermined direction of the user's line of sight. The information processing device according to claim 1 .

7. The display range determination unit determines the display range so that the specific target is displayed in a front direction of a line of sight of the user. The information processing device according to claim 5 .

8. The specific object is a specific space in the display object. The information processing device according to claim 6.

9. The display range determination unit determines the display range so that the specific object is displayed in a line of sight of the user when a viewpoint is switched. The information processing device according to claim 1 .

10. The specific object is an object that is determined to be looked at by the user in the display image of the switch source viewpoint. The information processing device according to claim 1 .

11. The display range determination unit determines whether to determine the display range using the viewpoint position information, the specific target information, and the line of sight direction information, based on a type of target that the user was looking at in the display image of the switching source viewpoint. The information processing device according to claim 1 .

12. The display range determination unit determines the display range using the viewpoint position information, the specific object information, and the line of sight direction information when it is determined that the user is looking at a specific object in the display image of the switch source viewpoint. The information processing device according to claim 11.

13. An information processing device, With respect to a multiple viewpoint image capable of displaying a display object from multiple viewpoints, when switching a display image from a source viewpoint image corresponding to a source viewpoint to a destination viewpoint image corresponding to a destination viewpoint, a display range of the destination viewpoint image is determined based on viewpoint position information of the destination viewpoint, specific target information which is information related to a specific target in the display object, and line of sight direction information which estimates a direction of a user's line of sight with respect to the display image. How to determine the display range.

14. A computer readable program, With respect to a multiple viewpoint image capable of displaying a display target from multiple viewpoints, when switching a display image from a source viewpoint image corresponding to a source viewpoint to a destination viewpoint image corresponding to a destination viewpoint, the computer device is caused to execute a process of determining a display range of the destination viewpoint image based on viewpoint position information of the destination viewpoint, specific target information which is information related to a specific target in the display target, and line of sight direction information which estimates a direction of a user's line of sight with respect to the display image. program.

Images