Display control apparatus, display control method, and program

Abstract

To provide a technique which allows a user to easily grasp a position of a fixation point of a virtual viewpoint on a virtual space.SOLUTION: A display control apparatus 130 acquires information indicative of a position of a fixation point of a first virtual view point corresponding to a virtual viewpoint image generated based on a plurality of captured image captured by a plurality of imaging devices 110, and displays an image 600 including a fixation point object 602 indicative of the fixation point and a projected fixation point object 603 indicative of a position on a projected plane of the fixation point.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to a display control device, a control method for a display control device, and a program. [Background technology]

[0002] In recent years, a technology that generates a virtual viewpoint image as if it were taken from a virtual camera viewpoint by using a plurality of images obtained by installing a plurality of physical cameras at different positions and taking synchronous images has been attracting attention. Patent Document 1 discloses a technology that generates a virtual viewpoint image by using images taken of a subject by arranging a plurality of cameras so as to surround the subject. Furthermore, Patent Document 1 discloses that the position of a virtual camera and the position of a gaze point indicating where the virtual camera is gazing are specified by the user. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-215828 A Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, for example, when the virtual camera is set to the viewpoint of the subject, there is a problem that it is difficult for the user to intuitively grasp the position of the gaze point in the virtual space.

[0005] In view of the above problems, an object of the present disclosure is to provide a technique that enables a user to easily grasp the position of the gaze point of a virtual viewpoint in a virtual space. [Means for solving the problem]

[0006] One aspect of a display control device of the present disclosure includes an acquisition means for acquiring information indicating the position of a gaze point of a virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by imaging with a plurality of imaging devices, and a display control means for displaying an image including a gaze point object indicating the gaze point and a projected gaze point object indicating the position of the gaze point on a projection surface. Effect of the Invention

[0007] According to the present disclosure, a user can easily grasp the position of the gaze point of a virtual viewpoint in a virtual space. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an image processing system. [Diagram 2] FIG. 2 is a diagram illustrating an example of installation of an imaging device. [Diagram 3] FIG. 2 is a diagram illustrating a hardware configuration of a display control device. [Figure 4] FIG. 2 is a diagram illustrating a functional configuration of a display control device according to the first embodiment. [Diagram 5] 4 is a diagram showing an operation flow of the display control device in the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of a screen display in the first embodiment. [Figure 7] FIG. 11 is a diagram illustrating a functional configuration of a display control device according to a second embodiment. [Figure 8] FIG. 11 is a diagram showing an operation flow of the display control device in the second embodiment. [Figure 9] FIG. 11 is a diagram showing an example of a screen display in the second embodiment. [Figure 10] FIG. 11 is a diagram illustrating a functional configuration of a display control device according to a third embodiment. [Figure 11] 13A to 13C are diagrams illustrating position correction of a second virtual viewpoint in the third embodiment. [Figure 12] FIG. 13 is a diagram showing an example of a screen display in the third embodiment. [Figure 13] 13A to 13C are diagrams illustrating position correction of a second virtual viewpoint in the third embodiment. [Figure 14] FIG. 13 is a diagram showing an example of a screen display in the third embodiment. [Figure 15] 13A to 13C are diagrams illustrating position correction of a second virtual viewpoint in the third embodiment. [Figure 16] FIG. 11 is a diagram showing an operation flow of the display control device in the third embodiment. [Figure 17] FIG. 13 is a diagram illustrating a functional configuration of a display control device according to a fourth embodiment. [Figure 18] FIG. 13 is a diagram showing an operation flow of the display control device in the fourth embodiment. [Figure 19] FIG. 13 is a diagram showing an example of a screen display in the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the components described in the following embodiments are examples of the embodiments, and the present disclosure is not limited to them. Also, not all of the combinations of features described in the embodiments are necessarily essential to the solution.

[0010] A virtual viewpoint image is an image generated by freely manipulating the position and orientation of a virtual camera by a user and / or a dedicated operator, and is also called a free viewpoint image or an arbitrary viewpoint image. In addition, although the present disclosure will mainly describe a case where a virtual viewpoint is specified by a user operation, the virtual viewpoint may be specified automatically based on the result of image analysis, etc. In addition, unless otherwise specified, the term "image" will be described as including the concepts of both a moving image and a still image.

[0011] The virtual camera is a virtual camera different from the multiple imaging devices actually installed around the imaging area, and is a concept for conveniently explaining the virtual viewpoint related to the generation of the virtual viewpoint image. That is, the virtual viewpoint image can be considered to be an image captured from a virtual viewpoint set in a virtual space associated with the imaging area. The position and orientation of the viewpoint in the virtual imaging can be expressed as the position and orientation of the virtual camera. In other words, the virtual viewpoint image can be said to be an image simulating an image captured by a camera when it is assumed that a camera exists at the position of the virtual viewpoint set in the space. In addition, in this embodiment, the content of the transition of the virtual viewpoint over time is expressed as a virtual camera path. However, it is not necessary to use the concept of a virtual camera to realize the configuration of this embodiment. That is, it is sufficient that at least information representing a specific position and information representing a direction in the space are set, and a virtual viewpoint image is generated according to the set information.

[0012] The imaging device may have a physical camera. The imaging device may have a function of performing various image processing in addition to the physical camera. For example, the imaging device may have a processing unit that performs foreground / background separation processing. The imaging device may have a control unit that performs transmission control to transmit an image of a part of the captured image. The imaging device may have multiple physical cameras.

[0013] <Embodiment 1> 1 is a diagram showing an image processing system 100 according to this embodiment. The image processing system 100 includes a plurality of imaging devices 110, an image generating device 120, a display control device 130, and a display 140. The imaging devices 110, the image generating device 120, and the display control device 130 are connected via a communication cable such as a LAN (Local Area Network) cable. In this embodiment, the communication cable is a LAN cable, but the communication cable is not limited to this embodiment. Also, the image generating device 120 and the display 140 are connected via a video signal transmission cable.

[0014] The imaging device 110 is, for example, a digital camera capable of capturing images (still images and videos). Fig. 2 is a diagram showing an example of installation of the imaging devices 110. Each imaging device 110 is installed so as to surround a specific area in a stadium or the like, and captures images (video) of subjects within the area. The captured image is transmitted from the imaging device 110 to the image generating device 120. Only an image corresponding to a partial area of ​​the captured image (for example, the area of ​​the subject) may be transmitted.

[0015] The image generating device 120 is, for example, a server device, and has a database function and an image processing function. The image generating device 120 stores images captured in advance of a scene in which no subject is present, such as before the start of photographing a subject in a stadium, as background images. The image generating device 120 also stores captured images obtained by the imaging device 110. When the image generating device 120 receives virtual viewpoint information and playback time information (e.g., a time code) through a user operation of the display control device 130, it generates a virtual viewpoint image based on the stored captured images. Here, the virtual viewpoint information is information indicating the three-dimensional position and angle of a virtual viewpoint (virtual viewpoint) in a virtual space, a gaze point position, etc. The virtual viewpoint information includes at least a relative position with respect to a predetermined origin position such as the center of the stadium where the image was taken, that is, position information of front / back, left / right, and up / down with respect to the origin position, and a direction from the predetermined position, that is, directional information of an angle with the front / back, left / right, and up / down axes as axes. The virtual viewpoint information also includes gaze point position information indicating which three-dimensional position is being watched from the virtual viewpoint position, and distance information from the gaze point position to the virtual viewpoint position.

[0016] The playback time information is time information at the time of shooting of a captured image, which is composed of hours, minutes, seconds, number of frames per second, etc., and by specifying the playback time, the scene at the recorded time is generated as a virtual viewpoint. For example, the number of frames per second is 60 frames. Based on a time server (not shown), the multiple image capturing devices 110 perform synchronized shooting, and the time information at the time of shooting indicates the timing of shooting at the multiple image capturing devices 110.

[0017] In addition, in a scene in which a subject exists, the image processing device 120 may separate the foreground of a specific object that is the subject as a specific object image by image processing. Note that the specific object may be not only a person but also an object such as a ball, whose image pattern is previously determined.

[0018] The virtual viewpoint image corresponding to the virtual viewpoint information is generated from a background image and a specific object image managed in a database. For example, model-based rendering (MBR) is used as a method for generating the virtual viewpoint image. MBR is a method for generating a virtual viewpoint image using a three-dimensional shape generated based on a plurality of captured images of a subject photographed from a plurality of directions. Specifically, it is a technology for generating an image of the scene seen from a virtual viewpoint by using a three-dimensional shape (model) of a target scene obtained by a three-dimensional shape restoration method such as a volume intersection method or Multi-View-Stereo (MVS). Note that the method for generating the virtual viewpoint image may use a rendering method other than MBR. The generated virtual viewpoint image is transmitted to the display 140 via a video signal transmission cable.

[0019] The display control device 130 is, for example, a PC (Personal Computer) or a tablet. The viewpoint controller 131 is a device for setting parameters such as the position and attitude of the virtual camera. For example, the viewpoint controller 131 is a mouse, a keyboard, a joystick, a six-axis controller, a touch panel, or a game controller. The time controller 132 is a device for setting a playback time, and is, for example, an operation device equipped with a turntable. The viewpoint controller 131 and the time controller 132 are operated by a user. The display control device 130 receives information on user operations from the viewpoint controller 131 and the time controller 132. Then, the display control device 130 converts the information into virtual viewpoint information indicating the position and attitude of the virtual camera and playback time information according to the amount of operation, etc., and transmits the information to the image generating device 120.

[0020] It should be noted that the output from the display control device 130 using the operation device is not limited to continuous movement, and can also move to a pre-set virtual viewpoint such as a front position, a rear position, or a position looking down from above of the subject in the virtual space. Also, by setting a playback time in advance, it is possible to instantly move the virtual viewpoint to that time. The display control device 130 also displays objects in a three-dimensional space based on user operations on the screen via an application displayed on the display unit 305 by executing a control program described later.

[0021] 3 is a diagram showing a hardware configuration of the display control device 130. The display control device 130 includes a CPU 301, a ROM 302, a RAM 303, a HDD 304, a display unit 305, an input unit 306, and a communication unit 307. The CPU 301 controls the entire display control device 130 using control programs and data stored in the ROM 302 and the RAM 303. The display control device 130 may include one or more dedicated hardware pieces different from the CPU 301, and the dedicated hardware may execute at least a part of the processing by the CPU 301. Examples of such dedicated hardware pieces include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor).

[0022] ROM 302 stores programs that do not require modification. RAM 303 temporarily stores programs and data supplied from HDD 304, and data supplied from the outside via communication unit 307. RAM 303 is also used as a temporary storage area such as the main memory and work area of ​​CPU 301. HDD 304 stores various data, various programs, etc.

[0023] The display unit 305 is composed of, for example, a liquid crystal display or LEDs, and displays various information. The input unit 306 can be connected to a keyboard, mouse, six-axis controller, etc., and accepts various operations by the user. The communication unit 307 performs communication processing with an external device via a network. An example of the network is Ethernet (registered trademark). As another example, the communication unit 307 may wirelessly communicate with an external device. A system bus 308 connects each unit of the display control device 130 to transmit information.

[0024] The functions and processes of the display control device 130 described later are realized by the CPU 301 reading out a program stored in the ROM 302 or the HDD 304 and executing the program. The hardware configuration of the image generation device 120 is similar to that of the display control device 130.

[0025] 4 is a diagram showing the functional configuration of the display control device 130. The controller operation acquisition unit 133 periodically acquires operation information on the virtual viewpoint acquired via the viewpoint controller 131 and the time controller 132.

[0026] The controller operation acquisition unit 133 converts operation information for the virtual viewpoint into virtual viewpoint movement amount information and playback time movement amount information, and outputs them to the first virtual viewpoint information determination unit 134. The virtual viewpoint movement amount is the movement amount relative to the current virtual viewpoint position or the line of sight direction from the virtual viewpoint. The playback time movement amount is the movement amount relative to the current playback time. These pieces of information are determined by a conversion coefficient set for the operation amount of the user operation input to the viewpoint controller 131 or the time controller 132.

[0027] The first virtual viewpoint information determination unit 134 determines virtual viewpoint information corresponding to the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint specified by the user operation based on the input virtual viewpoint movement amount information. Then, the first virtual viewpoint information determination unit 134 outputs the virtual viewpoint information to the virtual viewpoint object generation unit 136 as the first virtual viewpoint information. Similarly, the first virtual viewpoint information determination unit 134 outputs the first virtual viewpoint information to the gaze point object generation unit 137, the projected gaze point object generation unit 138, the second virtual viewpoint information determination unit 142, and the virtual viewpoint information transmission unit 144. Note that the virtual space coordinate system is the same as the coordinate system of each imaging device 110. For example, the center of the stadium or the like may be set as the center of the coordinate system, or may be appropriately set by the user. The virtual viewpoint information is expressed as a three-dimensional position in this coordinate system.

[0028] The time code determination unit 135 determines playback time information specified by a user operation based on the input playback time movement amount, and outputs the playback time information to the virtual viewpoint information transmission unit 144. Note that the playback time is based on the date and time when shooting started by each image capture device 110. Also, the playback time may be the shooting time itself, or may be expressed as the time elapsed from the shooting start time with the shooting start time set to 0.

[0029] The virtual viewpoint object generation unit 136 acquires the first virtual viewpoint information from the first virtual viewpoint information determination unit 134. In addition, the virtual viewpoint object generation unit 136 acquires virtual viewpoint object information from the object information storage unit 139 described later. The virtual viewpoint object generation unit 136 generates a virtual viewpoint object that enables the user to recognize the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint by the user operation in the virtual space based on the first virtual viewpoint information and the virtual viewpoint object information. The virtual viewpoint object includes data such as a shape for expressing the virtual viewpoint object and information on its position in space. In addition, the virtual viewpoint object generation unit 136 outputs the virtual viewpoint object to the second virtual viewpoint information determination unit 142.

[0030] The gaze point object generating unit 137 acquires first virtual viewpoint information from the first virtual viewpoint information determining unit 134. Also, the gaze point object generating unit 137 acquires gaze point object information from the object information holding unit 130. Based on gaze point position information included in the first virtual viewpoint information and gaze point object information, the gaze point object generating unit 137 generates a gaze point object that allows the user to recognize the position of the gaze point of the virtual viewpoint by the user operation in the virtual space. The gaze point object includes data such as a shape for expressing the gaze point object and information on its position in space. Also, the gaze point object generating unit 137 outputs the gaze point object to the second virtual viewpoint information determining unit 142.

[0031] The projection gaze point object generating unit 138 acquires the first virtual viewpoint information from the first virtual viewpoint information determining unit 134. The projection gaze point object generating unit 138 also acquires the projection gaze point object information from the object information holding unit 139. The projection gaze point object generating unit 138 also acquires the projection plane position information from the plane position setting unit 140 described later. Based on the acquired information, the projection gaze point object generating unit 138 generates a projection gaze point object that allows the user to recognize the position of the projection gaze point obtained by projecting the position of the gaze point in the virtual space onto the projection plane position. The projection gaze point object includes data such as a shape for expressing the projection gaze point object and information on its position in space. The projection gaze point object generating unit 138 also outputs the projection gaze point object to the second virtual viewpoint information determining unit 142.

[0032] The object information storage unit 139 stores object information such as virtual viewpoint object information, gaze point object information, and projection gaze point object information in advance, and outputs the corresponding object information. That is, the object information storage unit 139 outputs the virtual viewpoint object information to the virtual viewpoint object generation unit 136. Also, the object information storage unit 139 outputs gaze point object information to the gaze point object generation unit 137. Also, the object information storage unit 139 outputs the projection gaze point object information to the projection gaze point object generation unit 138. Here, the object information is a three-dimensional model, and is data having three-dimensional coordinates created by dedicated software or the like, and is data showing a three-dimensional shape by connecting a plurality of three-dimensional coordinates (vertices). Also, by storing color information and texture images in association with a surface formed by connecting a plurality of vertices, it becomes an object recognizable by the user. For example, the object information may be mesh data composed of a plurality of polygons.

[0033] The surface position setting unit 140 determines the surface to be projected based on surface information set from the surface information storage unit 141 described later. As a specific example, the position of a floor surface (height is set to 0) of a stadium or the like, which has a constant height, is set as the projection surface position. The surface position setting unit 140 outputs the set projection surface position information to the projection gaze point object generation unit 138. The projection surface position is not limited to this, and may be specified by the user as appropriate. For example, the projection surface may be a wall or a ceiling that constitutes a structure.

[0034] The surface information storage unit 141 stores a plane in a three-dimensional space in advance, and outputs the corresponding surface information. That is, the surface information storage unit 141 outputs the surface information to the surface position setting unit 140. Here, the surface information is a three-dimensional model, similar to the object information, and is data having three-dimensional coordinates created by dedicated software or the like, and is data that indicates a plane or the like by connecting a plurality of three-dimensional coordinates (vertices). For example, the surface information may be mesh data composed of a plurality of polygons. In this embodiment, a certain floor surface (height is set to 0) such as a stadium will be described. A floor surface that can be recognized by a user will be obtained by storing color information and texture images associated with a surface formed by connecting a plurality of vertices. Note that the surface information is not limited to a flat floor surface, and may be a three-dimensional surface formed by combining a plurality of vertices.

[0035] The second virtual viewpoint information determination unit 142 acquires a virtual viewpoint object from the virtual viewpoint object generation unit 136, acquires a gaze point object from the gaze point object generation unit 137, and acquires a projection gaze point object from the projection gaze point object generation unit 138. In addition, the second virtual viewpoint information determination unit 142 acquires the first virtual viewpoint information from the first virtual viewpoint information determination unit 134. The second virtual viewpoint information determination unit 142 determines second virtual viewpoint information in which the virtual viewpoint object, the gaze point object, and the projection gaze point object can be displayed based on the first virtual viewpoint information. The second virtual viewpoint information is output to the display control unit 143. Specifically, the second virtual viewpoint information is information indicating the position of a virtual viewpoint other than the virtual viewpoint operated by the user and the line of sight direction from the virtual viewpoint. For example, the position of the virtual viewpoint indicated by the second virtual viewpoint information may be behind the first virtual viewpoint (the virtual viewpoint operated by the user) indicated by the first virtual viewpoint information, and may be a position in which the projection gaze point object can be displayed. In addition, the line of sight direction from the second virtual viewpoint indicated by the second virtual viewpoint information may be a direction in which the projection gaze point object can be displayed. Furthermore, the position of the second virtual viewpoint and the line of sight direction from the virtual viewpoint may be a position and orientation that allows a virtual viewpoint object to be displayed. "Rear" does not mean that the position of the second virtual viewpoint is connected in a straight line, but means that the position of the second virtual viewpoint is on the opposite side of the gaze point with respect to the position of the first virtual viewpoint. Furthermore, the field of view of the first virtual viewpoint specified by the first virtual viewpoint information does not include the position of the second virtual viewpoint. On the other hand, the field of view of the second virtual viewpoint includes the position of the first virtual viewpoint, the position of the gaze point, and the position of the projected gaze point.

[0036] In addition, being displayable means that it is possible to display in a virtual viewpoint image generated on the display device based on the second virtual viewpoint information. In addition, in this virtual viewpoint image, the virtual viewpoint object, the gaze point object, and the projected gaze point object may not be displayed at the same time. For example, the gaze point object and the projected gaze point object may be displayed, and the virtual viewpoint object may not be displayed. Alternatively, for example, one of the objects may be hidden by a display instruction from the user. In addition, these objects may be displayed semi-transparently. In this case, the objects are prevented from being occluded by each other, making it easier for the user to recognize the objects. In addition, when a subject is present on the virtual viewpoint image, the subject is prevented from being occluded, making it easier for the user to recognize the subject.

[0037] The display control unit 143 acquires the second virtual viewpoint information from the second virtual viewpoint information determination unit 142. The display control unit 143 displays the virtual viewpoint object, the gaze point object, and the projected gaze point object seen from the second virtual viewpoint on a two-dimensional image. That is, by displaying on the display unit 305 of the display control device 130, the position of the virtual camera (first virtual viewpoint) based on the virtual viewpoint operation of the user, the gaze point, and the shadow of the gaze point projected on the floor surface are displayed. This display is performed in a virtual viewpoint image generated based on the second virtual viewpoint information. This virtual viewpoint image may be generated based on the imaging device 110, or may be generated based on a background model such as a stadium stored in advance in the image generating device 120. This virtual viewpoint image may be generated by the display control unit 143.

[0038] The virtual viewpoint information transmitting unit 144 transmits the first virtual viewpoint information input from the first virtual viewpoint information determining unit 134 and the playback time information input from the time code determining unit 135 to the image generating device 120. Note that the virtual viewpoint image generated by the image generating device 120 based on the virtual viewpoint information and the playback time information is output to and displayed on the display 140. The virtual viewpoint image generated by the image generating device 120 is a virtual viewpoint image seen from the first virtual viewpoint, and is different from the virtual viewpoint image seen from the second virtual viewpoint.

[0039] Next, a description will be given of the operation of the display control device 130. Fig. 5 is a flowchart showing the operation of the display control device 130 according to this embodiment. The CPU 301 reads out and executes a program stored in the ROM 302 or the HDD 304, thereby carrying out the following processing.

[0040] In step S501, first, the controller operation acquisition unit 133 acquires movement amount information of a lever of a six-axis controller such as a joystick from the viewpoint controller 131. The controller operation acquisition unit 133 also acquires movement amount information of, for example, a rotating disk turned by the user from the time controller 132. Next, the controller operation acquisition unit 133 converts the movement amount information acquired from each of the viewpoint controller 131 and the time controller 132 into virtual viewpoint movement amount information and playback time movement amount information, and outputs them to the first virtual viewpoint information determination unit 134.

[0041] In step S502, the first virtual viewpoint information determination unit 134 determines the first virtual viewpoint information based on the input virtual viewpoint movement amount information. Then, the first virtual viewpoint information determination unit 134 outputs the first virtual viewpoint information to the virtual viewpoint object generation unit 136, the gaze point object generation unit 137, and the projected gaze point object generation unit 138. The first virtual viewpoint is a virtual viewpoint that is an object of operation by the user, and corresponds to a virtual viewpoint image displayed on the display 140.

[0042] In step S503, the virtual viewpoint object generation unit 136 generates a virtual viewpoint object. Specifically, the virtual viewpoint object generation unit 136 acquires virtual viewpoint information from the first virtual viewpoint information determination unit 134. The virtual viewpoint object generation unit 136 also acquires virtual viewpoint object information from the object information storage unit 139. Based on the virtual viewpoint information and the virtual viewpoint object information, the virtual viewpoint object generation unit 136 generates a virtual viewpoint object that allows the user to recognize the position of the first virtual viewpoint in the virtual space and the line of sight from the first virtual viewpoint. For example, the virtual viewpoint object may have a camera shape or another shape.

[0043] In step S504, the gaze point object generating unit 137 generates a gaze point object. Specifically, the gaze point object generating unit 137 acquires virtual viewpoint information from the first virtual viewpoint information determining unit 134. In addition, the gaze point object generating unit 137 acquires gaze point object information from the object information holding unit 130. The gaze point object generating unit 137 generates a gaze point object that allows the user to recognize the position of the gaze point of the first virtual viewpoint in the virtual space, based on gaze point position information included in the virtual viewpoint information and gaze point object information. For example, the gaze point object may have a spherical shape or a rectangular prism shape.

[0044] In step S505, the projection gaze point object generating unit 138 generates a projection gaze point object. Specifically, the projection gaze point object generating unit 138 acquires virtual viewpoint information from the first virtual viewpoint information determining unit 134. Also, the projection gaze point object generating unit 138 acquires projection gaze point object information from the object information holding unit 139. Also, the projection gaze point object generating unit 138 acquires projection plane position information from the plane position setting unit 140. Based on the acquired information, the projection gaze point object generating unit 138 generates a projection gaze point object that allows the user to recognize the position of the gaze point of the first virtual viewpoint in the virtual space projected onto the plane position. For example, the projection gaze point object may have a circular shape or a rectangular shape. Also, the projection gaze point object indicates the position of the gaze point on the projection plane. The projection gaze point object may be generated by projecting the gaze point object onto the projection plane. In this case, the projection gaze point object generating unit 138 may acquire the gaze point object from the gaze point object generating unit 137.

[0045] In step S506, the second virtual viewpoint information determination unit 142 determines the second virtual viewpoint information. Specifically, the second virtual viewpoint information determination unit 142 acquires the first virtual viewpoint information from the first virtual viewpoint information determination unit 134. In addition, the second virtual viewpoint information determination unit 142 acquires the virtual viewpoint object, the gaze point object, and the projection gaze point object. The second virtual viewpoint information determination unit 142 determines the second virtual viewpoint information based on the acquired information. The second virtual viewpoint information indicates information of a second virtual viewpoint from which the virtual viewpoint object, the gaze point object, and the projection gaze point object can be virtually photographed. More specifically, the second virtual viewpoint information is determined so that the position of the second virtual viewpoint is a position behind the first virtual viewpoint, and the line of sight from the second virtual viewpoint is a direction in which the projection gaze point object can be virtually photographed. The second virtual viewpoint information determination unit 142 outputs the second virtual viewpoint information to the display control unit 143.

[0046] In step S507, the display control unit 143 acquires the second virtual viewpoint information from the second virtual viewpoint information determination unit 142. The display control unit 143 generates and outputs a two-dimensional image (virtual viewpoint image) including a virtual viewpoint object, a gaze point object, and a projected gaze point object viewed from the second virtual viewpoint. That is, the display control unit 143 displays a two-dimensional image including the virtual viewpoint object, the gaze point object, and the projected gaze point object on the display unit 305 of the display control device 130. This displays the positions of the first virtual viewpoint, its gaze point, and the projected gaze point in the virtual space based on the virtual viewpoint operation by the user. This allows the user to easily grasp the position of the gaze point in the virtual space.

[0047] 6 is an example of a screen displayed on the display unit 305 of the display control device 130. On a screen 600, a virtual viewpoint object 601, a gaze point object 602, and a projected gaze point object 603 are displayed.

[0048] The virtual viewpoint object 601 has the shape of a camera. The position of this camera indicates the position of the first virtual viewpoint, and the orientation of the camera indicates the line of sight from the first virtual viewpoint. The gaze point object 602 has a spherical shape. The position of this gaze point object 602 indicates the three-dimensional position of the gaze point of the first virtual viewpoint in the virtual space. It is difficult for the user to intuitively grasp at what height in the virtual space this gaze point object 602 is located just from the gaze point object 602. For this reason, a projected gaze point object 603 is displayed. The projected gaze point object 603 is an object obtained by projecting the gaze point object 602 onto a projection surface (for example, a floor surface), and has an elliptical shape.

[0049] For example, the projected gaze point object 603 corresponds to the shadow of the gaze point object 602 when a virtual point light source is placed on a line connecting the projection surface and the gaze point object 602 in a direction perpendicular to the projection surface. The shape of the projected gaze point object 603 may be a shape that matches the projection surface. For example, if the projection surface has irregularities, the shape of the projected gaze point object 603 may be displayed in a shape that matches the irregularities. In this case, the user can easily grasp the position of the gaze point.

[0050] The projected gaze point object 603 does not have to faithfully reproduce the shape of the gaze point object 602 projected onto the projection surface. The shape of the projected gaze point object 603 may be emphasized, for example, so that the diameter of the projected gaze point object 603 is larger than the diameter of the gaze point object 602. In this case, the user can more easily grasp the position of the gaze point. Other emphasis methods, such as blinking the projected gaze point object 603, may also be used. Note that the gaze point object 602 may also be blinking.

[0051] As described above, according to this embodiment, a projected gaze point object based on the position of the gaze point is displayed, so that a shadow or the like is recognized on the floor surface, making it easier for the user to intuitively grasp the gaze point position.

[0052] In this embodiment, a shadow is displayed on the projection surface as the projection gaze point object 603, but the present invention is not limited to this. A projection gaze point object such as an arrow or bar shape that combines the gaze point position and the projection gaze point position may be generated and displayed. This makes it easier to recognize the positional relationship even if the gaze point and the projection gaze point are separated from each other.

[0053] <Embodiment 2> Fig. 7 is a diagram showing the functional configuration of a display control device 700 according to embodiment 2. The display control device 700 has the following functional units in addition to the configuration of the display control device 130 according to embodiment 1 described with reference to Fig. 4. That is, the display control device 700 further has a projection gaze point object color determination unit 701 in addition to the display control device 130. The other functional units are the same as those of the display control device 130, and therefore descriptions thereof will be omitted.

[0054] The projection gaze point object color determination unit 701 acquires surface information from the surface information storage unit 141. In addition, the projection gaze point object color determination unit 701 acquires the position information of the projection gaze point and the projection gaze point object information from the projection gaze point of object generation unit 138. The projection gaze point object color determination unit 701 specifies the projection surface color information, which is the color information of the surface position corresponding to the position of the projection gaze point, based on the position information of the projection gaze point and the surface information. In addition, the projection gaze point object color determination unit 701 compares the color information included in the projection gaze point object with the specified projection surface color information. If the compared color information of the projection gaze point object is within a predetermined range, the projection gaze point object color determination unit 701 changes the color information of the projection gaze point object and outputs the changed color information to the projection gaze point object generation unit 138.

[0055] 8 is a flowchart showing the operation of the display control device 700 according to this embodiment. Note that steps S501 to S504 and steps S505 to S507 are the same as those in FIG. 5, and therefore the explanation thereof will be omitted.

[0056] In step S801, the projection gaze point object color determination unit 701 determines the color of the projection gaze point object. Specifically, the projection gaze point object color determination unit 701 specifies the projection surface color information, which is the color information of the surface position corresponding to the position of the projection gaze point, based on the position information of the projection gaze point and the surface information. Then, the projection gaze point object color determination unit 701 compares the color information included in the projection gaze point object with the specified projection surface color information. If the color information of the projection gaze point object is within a predetermined range for the projection surface color information, the projection gaze point object color determination unit 701 changes the color information of the projection gaze point object and outputs the changed color information to the projection gaze point object generation unit 138. On the other hand, if the color information of the projection gaze point object exceeds the predetermined range for the projection surface color information, the projection gaze point object color determination unit 701 does not change the color information of the projection gaze point object. In this case, the projection gaze point object color determination unit 701 notifies the projection gaze point object generation unit 138 that the color information will not be changed. Or, even if there is no change, the projection gaze point object generating unit 138 may output the determined color information to the projection gaze point object generating unit 138 .

[0057] Specifically, the brightness of the color information of the floor surface for the position of the projection gaze point is set to 1 on a scale of 1 to 10, and the brightness based on the color information of the projection gaze point object information is also set to 1. In this case, the brightness of the projection gaze point position on the floor surface and the brightness of the projection gaze point object become the same, making it difficult for the user to recognize the projection gaze point position, so the brightness of the projection gaze point object is increased. That is, the color information of the projection gaze point object is changed to, for example, 5 out of 1 to 10 levels of brightness, and the changed color information is output to the projection gaze point object generating unit 138. Depending on the brightness of the floor surface, the brightness of the projection gaze point object may be lower than that of the floor surface.

[0058] 9 is an example of a screen displayed on the display unit 305 of the display control device 700 according to this embodiment. A virtual viewpoint object 601, a gaze point object 602, a low-brightness floor surface 901 in which the floor surface brightness is 1 out of 10 levels, and a projection gaze point object 902 are displayed on the screen 600. When the projection gaze point moves onto the low-brightness floor surface 901, the brightness of the projection gaze point object 902 is increased. For example, when the brightness based on the color information of the projection gaze point object 902 is 1 out of 10 levels, the projection gaze point object is displayed with color information changed to color information of 5. This enables the user to recognize the projection gaze point position.

[0059] In the case of the brightness that changes according to the position on the floor surface, the brightness of the projection gaze point object 902 also changes according to the position, but this is not limited to the above example. For example, the projection gaze point object color determination unit 701 may determine the color information of the projection gaze point object 902 with a brightness different from the brightness value of the entire region by acquiring the brightness value that changes according to the position on the floor surface for the entire region. In this case, since it is not necessary to change the color information of the projection gaze point object 902 according to the position of the gaze point, the processing load of the display control device 700 can be reduced.

[0060] In addition, although an example has been described in which the brightness changes depending on the position on the floor surface, if the brightness changes over time even at the same position on the floor surface, the projection gaze point object color determination unit 701 may change the color information of the projection gaze point object 902 over time.

[0061] In this embodiment, the change in brightness based on the color information of the projection gaze point object has been described as an example, but the present invention is not limited to this and the saturation or hue based on the color information may be changed.

[0062] <Embodiment 3> Fig. 10 is a diagram showing a functional configuration of a display control device 1000 according to this embodiment. The display control device 1000 has the following functional units in addition to the display control device 13 according to the first embodiment described with reference to Fig. 4. That is, the display control device 1000 also has a second virtual viewpoint information correction unit 1001.

[0063] The second virtual viewpoint information correction unit 1001 acquires a virtual viewpoint object from the virtual viewpoint object generation unit 136, acquires a gaze point object from the gaze point object generation unit 137, and acquires a projection gaze point object from the projection gaze point object generation unit 138. In addition, the second virtual viewpoint information correction unit 1001 acquires second virtual viewpoint information from the second virtual viewpoint information determination unit 142. The second virtual viewpoint information correction unit 1001 corrects the second virtual viewpoint information based on the virtual viewpoint object, the gaze point object, and the projection gaze point object. Specifically, at least one of the gaze point object and the projected gaze point object may be occluded by the virtual viewpoint object at the position of the second virtual viewpoint determined by the second virtual viewpoint information determination unit 142. In that case, the second virtual viewpoint information correction unit 1001 corrects the position of the second virtual viewpoint so that the object can be displayed simultaneously. The second virtual viewpoint information correction unit 1001 outputs the second virtual viewpoint information of the second virtual viewpoint whose position has been corrected to the display control unit 143. Note that the second virtual viewpoint information correction unit 1001 may correct the line of sight from the second virtual viewpoint instead of the position of the second virtual viewpoint, or may correct both the position of the second virtual viewpoint and the line of sight from the second virtual viewpoint.

[0064] Furthermore, the second virtual viewpoint information correction unit 1001 may correct the second virtual viewpoint information when the gaze point object is occluded by the virtual viewpoint object so that the gaze point object is not occluded by the virtual viewpoint object. Furthermore, the second virtual viewpoint information correction unit 1001 may correct the second virtual viewpoint information when the projected gaze point object is occluded by the virtual viewpoint object so that the projected gaze point object is not occluded by the virtual viewpoint object. Furthermore, the second virtual viewpoint information correction unit 1001 may correct the second virtual viewpoint information when the projected gaze point object is occluded by the virtual viewpoint object so that the gaze point object and the gaze point object are not occluded by the virtual viewpoint object.

[0065] The following describes a method for determining the position of the second virtual viewpoint by the second virtual viewpoint information correction unit 1001. For example, as shown in Fig. 11, the second virtual viewpoint information determination unit 142 determines the position of the second virtual viewpoint 1701 to be located a predetermined distance 1801 behind the virtual viewpoint object 601 (in the positive direction of the Y axis) and at the same height as the Z axis position of the virtual viewpoint object 601. This allows imaging at the second virtual viewpoint 1701 so that the virtual viewpoint object 601 and the gaze point object 602 do not overlap. However, in the case of this determination method, when the Z axis positions of the virtual viewpoint object 601 and the gaze point object 602 are at the same height, the gaze point object 602 is occluded by the virtual viewpoint object 601 when viewed from the second virtual viewpoint position 1701. Therefore, the second virtual viewpoint information correction unit 1001 corrects the position of the second virtual viewpoint 1701. That is, when the virtual viewpoint object 601 and the gaze point object 602 exist in the virtual space, the position of the second virtual viewpoint 1701 is determined by these positions. Specifically, first, the second virtual viewpoint information correction unit 1001 determines whether the second virtual viewpoint 1701 exists on a straight line 1702 connecting the gaze point object 602 and the virtual viewpoint object 601. When the second virtual viewpoint 1701 exists on a straight line 1702, the display when the second virtual viewpoint 1701 captures an image is such that the fixation point object 602 is hidden by the virtual viewpoint object 601, as shown in FIG. 12. To avoid this, the second virtual viewpoint information correction unit 1001 raises the second virtual viewpoint 1701 by a predetermined distance in the Z-axis direction. In the example of FIG. 13, the position of the second virtual viewpoint 1701 is corrected by moving the second virtual viewpoint 1701 by a distance 1703 in the Z-axis direction. Note that in this embodiment, the value of the distance 1703 is set in advance, but this is not limited thereto. For example, the distance 1703 may be dynamically determined according to the moving speed and moving direction of the camera object 601. This allows both the virtual viewpoint object 601 and the fixation point object 602 to be visible from the position of the second virtual viewpoint 1701. That is, in the image when the image is captured from the second virtual viewpoint 1701, the virtual viewpoint object 601 and the fixation point object 602 are displayed without being blocked, as shown in FIG. 14. Note that the position of the second virtual viewpoint 1701 may be moved not only in the Z-axis direction but also in the X-axis direction. Also, the position of the second virtual viewpoint 1701 may be moved in the positive direction or in the negative direction of the Z-axis. Also, the position of the second virtual viewpoint 1701 may be moved in the positive direction or in the negative direction of the X-axis.

[0066] Although an example has been shown in which the second virtual viewpoint 1701 is corrected so that the fixation point object 602 is not occluded by the virtual viewpoint object 601, the present embodiment is not limited to this. For example, the second virtual viewpoint information correcting unit 1001 may correct the second virtual viewpoint 1701 so that the projection fixation point object 603 is not occluded by the virtual viewpoint object 601. This embodiment can also be applied to a case in which the fixation point object 602 and the projection fixation point object 603 are occluded by the virtual viewpoint object 601. FIG. 15 shows an example in which the position of the second virtual viewpoint 1701 is corrected so that the fixation point object 602 and the projection fixation point object 603 are not occluded by the virtual viewpoint object 601.

[0067] The second virtual viewpoint information correction unit 1001 first judges whether the second virtual viewpoint 1701 exists on a straight line 1903 connecting the fixation point object 602 and the virtual viewpoint object 601. If the second virtual viewpoint 1701 exists on the straight line 1903, the second virtual viewpoint 1701 is raised in the Z-axis direction by a predetermined distance 1901 to perform a first correction (first correction). Next, the second virtual viewpoint 1701 after the correction is judged whether it exists on a straight line 1904 connecting the projected fixation point object 603 and the virtual viewpoint object 601. If the second virtual viewpoint 1701 exists on the straight line 1904, the second virtual viewpoint 1701 after the first correction is further raised in the Z-axis direction by a predetermined distance 1902 to perform a second correction (second correction). This prevents the fixation point object 602 and the projected fixation point object 603 from being displayed as being occluded by the virtual viewpoint object 601. The first correction and the second correction may be performed simultaneously.

[0068] Fig. 16 is a flowchart showing the operation of the display control device 1000 according to this embodiment. Note that steps S501 to S506 are the same as those in Fig. 5, and therefore the explanation will be omitted. The CPU 301 reads out and executes a program stored in the ROM 302 or the HDD 304, thereby performing the following processing.

[0069] In step S1101, the second virtual viewpoint information correction unit 1001 judges whether the gaze point object and the virtual viewpoint object are occluded (whether they overlap) when viewed from the position of the second virtual viewpoint indicated by the input second virtual viewpoint information. If the gaze point object is occluded by the virtual viewpoint object (Yes in S1101), the process proceeds to step S1102, and otherwise the process proceeds to step S1103. Note that this judgment is as described above, and therefore description thereof will be omitted. In step S1102, the position is corrected so that the virtual viewpoint object 601 and the gaze point object 602 move to the second virtual viewpoint at a position where they can be displayed simultaneously. The specific method of correction is also as described above.

[0070] In step S1103, the second virtual viewpoint information correction unit 1001 judges whether the fixation point object and the projected fixation point object are occluded (overlapped) when viewed from the position of the second virtual viewpoint indicated by the input second virtual viewpoint information. If the projected fixation point object is occluded by the virtual viewpoint object (Yes in S1103), the process proceeds to step S1104, and otherwise the process proceeds to step S507. Note that this judgment is as described above, and therefore the explanation is omitted. In step S1104, the second virtual viewpoint information is corrected so that the virtual viewpoint object 601, the fixation point object 602, and the projected fixation point object 603 move to the second virtual viewpoint at a position where they can be displayed simultaneously. Note that the specific method of is also as described above. The second virtual viewpoint information based on the corrected second virtual viewpoint is output to the display control unit 143.

[0071] In step S507, the virtual viewpoint object, the gaze point object, and the projected gaze point object as viewed from the input second virtual viewpoint position are displayed on the two-dimensional image. That is, by displaying them on the display unit of the display control device 130, the virtual camera position based on the user's virtual viewpoint operation is displayed.

[0072] As described above, according to this embodiment, by placing the second virtual viewpoint at the same height as the virtual viewpoint object and behind it, the operation of moving the virtual camera can be displayed intuitively and easily to the operator. Furthermore, when the gaze point position overlaps with the virtual viewpoint object, the gaze point position can be always easily grasped by correcting the second virtual viewpoint position.

[0073] In this embodiment, when the virtual viewpoint object, the gaze point object, and the projected gaze point object are displayed, it is not necessary that the entire shape of each object is displayed, and only a portion of the shape of each object may be displayed.

[0074] In addition, in the present embodiment, in FIG. 15, an example is described in which the second virtual viewpoint is moved when both the virtual viewpoint object and the gaze point object and the projected gaze point object are occluded by the virtual viewpoint object, but this is not limited thereto. For example, this embodiment is also applicable to a case in which only one of the gaze point object and the projected gaze point object is occluded by the virtual viewpoint object. For example, in FIG. 16, the flow is shown in which the second virtual viewpoint is corrected (second correction, S1104) when the projected gaze point object overlaps, but the processes of S1103 and S1104 may be omitted. Similarly, the processes of S1101 and S1102 (first correction) may be omitted. In this way, the first correction or the second correction may not be performed.

[0075] In addition, the movement to the second virtual viewpoint is not limited to the above. For example, in Fig. 15, when overlap with the projected gaze point object is detected after the object is raised by the distance 1901, the object may be lowered by the distance 1901 downward from the position before the raising as a second correction. Also, the object may move on a curve or a sphere whose radius is the sum of the distance between the gaze point object and the virtual viewpoint object and the length of the straight line 1903, with the gaze point object as the center. Similarly, the object may move in the horizontal direction, taking into account the size of the virtual viewpoint 601.

[0076] In addition, the display control device 1000 of the present embodiment has the second virtual viewpoint information correction unit 1001, but the present embodiment is not limited to this. For example, the position of the second virtual viewpoint 142 may be determined so that the second virtual viewpoint 142 does not overlap with the virtual viewpoint object and at least one of the gaze point object and the projected gaze point object. In this case, the second virtual viewpoint information correction unit 1001 does not need to be a functional unit of the display control device 1000 of the present embodiment.

[0077] <Embodiment 4> This embodiment is an embodiment in which the display mode of a first virtual viewpoint object is changed. Fig. 17 is a diagram showing the functional configuration of a display control device 1400 according to this embodiment. Note that the display control device 1400 has the following functional units in addition to the configuration of the display control device 130 according to the first embodiment described with reference to Fig. 4.

[0078] The display mode control unit 1401 acquires second virtual viewpoint information from the second virtual viewpoint information determination unit 142. The display mode control unit 1401 also acquires a virtual viewpoint object, a gaze point object, and a projected gaze point object via the second virtual viewpoint information determination unit 142. The display mode control unit 1401 also determines whether the virtual viewpoint object occludes either the gaze point object or the projected gaze point object when viewed from a virtual viewpoint position based on the second virtual viewpoint information. When at least one object is occluded by the virtual viewpoint object, the display mode control unit 1401 determines to change the display mode so as to make the virtual viewpoint object semi-transparent. Then, the display mode control unit 1401 makes the virtual viewpoint object semi-transparent and outputs an image tentatively taken from the second virtual viewpoint position to the display control unit 143. By making the virtual viewpoint object semi-transparent, the gaze point object, the projected gaze point object, and the projected gaze point object can be recognized by the user.

[0079] Fig. 18 is a flowchart showing the operation of the display control device 1400 according to the fourth embodiment. Note that steps S501 to S506 are the same as those in Fig. 5, and therefore the explanation will be omitted. The CPU 301 reads out and executes a program stored in the ROM 302 or the HDD 304, thereby performing the following processing.

[0080] In step S1501, the display mode control unit 1401 determines whether the gaze point object, the projected gaze point object, and the virtual viewpoint object are occluded from the position of the second virtual viewpoint indicated by the input second virtual viewpoint information. If the gaze point object and the projected gaze point object or the projected gaze point object are occluded by the virtual viewpoint object (Yes in S1501), the process proceeds to step S1502. If the gaze point object and the projected gaze point object are not occluded by the virtual viewpoint object and can be displayed (No in S1501), the display mode control unit 1401 proceeds to step S507.

[0081] In step S1502, the display mode control unit 1401 changes the display mode to make the virtual viewpoint object semi-transparent. The display mode control unit 1401 also outputs second virtual viewpoint information based on the second virtual viewpoint to the display control unit 143.

[0082] In step S507, an image including a semi-transparent virtual viewpoint object, a gaze point object, and a projected gaze point object as viewed from the input second virtual viewpoint position is displayed. That is, by displaying the virtual viewpoint object semi-transparently as an image to be displayed on the display unit of the display control device 130, the user can recognize the virtual viewpoint object, the gaze point object, and the projected gaze point object.

[0083] 19 is an example of a screen display of the display control device 1400. An image viewed from a position based on the second virtual viewpoint information is displayed on the display viewpoint screen 600 of the display control device 1000. A semi-transparent virtual viewpoint object 1601 allows the user to recognize a gaze point object 602 and a projected gaze point object 603.

[0084] As described above, according to the fourth embodiment, when the fixation point position overlaps with the virtual viewpoint object, the fixation point position can be easily grasped by making the virtual viewpoint object semi-transparent. Note that, in the present embodiment, the fixation point object and the projected fixation point object are occluded by the virtual viewpoint object, but the present embodiment is not limited to this, and when any of the object shapes is occluded, the occluding object may be made semi-transparent. Also, in the present embodiment, the display form is described as semi-transparent, but for example, the size of the virtual viewpoint object may be reduced so that the fixation point object and the projected fixation point object can be visually recognized.

[0085] <Other embodiments> The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0086] <Other> The disclosure of the above-described embodiments includes the following configurations, methods, and programs.

[0087] (Configuration 1) an acquisition means for acquiring information indicating a position of a gaze point of a virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by imaging with a plurality of imaging devices; a display control means for displaying an image including a gaze point object indicating the gaze point and a projected gaze point object indicating the position of the gaze point on a projection surface; A display control device having the following.

[0088] (Configuration 2) 2. The display control device according to configuration 1, wherein the projected gaze point object is generated based on a projection of the gaze point onto the projection surface.

[0089] (Configuration 3) 3. The display control device according to configuration 1 or 2, wherein the image is a virtual viewpoint image corresponding to another virtual viewpoint different from the virtual viewpoint.

[0090] (Configuration 4) 4. The display control device according to any one of configurations 1 to 3, wherein the image includes a virtual viewpoint object indicating the virtual viewpoint.

[0091] (Configuration 5) A display control device as described in configuration 4, comprising a determination means for determining another virtual viewpoint corresponding to an image including the gaze point object, the projected gaze point object, and the virtual viewpoint object, the other virtual viewpoint being different from the virtual viewpoint.

[0092] (Configuration 6) 6. The display control device according to configuration 4 or 5, wherein in the image, the virtual viewpoint object is located at a position that does not overlap with the gaze point object.

[0093] (Configuration 7) 7. The display control device according to any one of configurations 4 to 6, wherein in the image, the virtual viewpoint object is located at a position that does not overlap with the projected gaze point object.

[0094] (Configuration 8) 8. The display control device according to any one of configurations 4 to 7, wherein the virtual viewpoint object is located at a position that does not overlap with the gaze point object and the projected gaze point object.

[0095] (Configuration 9) 9. The display control device according to any one of configurations 4 to 8, wherein the virtual object is displayed semi-transparently.

[0096] (Configuration 10) 10. The display control device according to any one of configurations 1 to 9, wherein the projected gaze point object is spaced apart from the gaze point object.

[0097] (method) an acquisition step of acquiring information indicating a position of a gaze point of a virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by imaging with a plurality of imaging devices; a display control step of displaying an image including a gaze point object indicating the gaze point and a projected gaze point object indicating a position of the gaze point on a projection plane; A display control method comprising:

[0098] (program) A program for causing a computer to execute the display control method described above. [Explanation of symbols]

[0099] 110 Imaging device 130, 700, 1000, 1400 Display control device

Claims

1. An acquisition means for acquiring information indicating the position of a first virtual camera corresponding to a virtual viewpoint image generated based on multiple captured images obtained by imaging from multiple imaging devices, and information indicating the position of the gaze point of the first virtual camera. The system includes a determination means for determining the position of a second virtual camera at a position away from a straight line passing through the position of the first virtual camera and the position of the point of focus, based on the position of the first virtual camera and the position of the point of focus. The distance from the position of the second virtual camera to the position of the first virtual camera is greater than the distance from the position of the second virtual camera to the position of the first virtual camera. An information processing device characterized by the following.

2. The information processing apparatus according to claim 1, characterized in that the position of the second virtual camera is higher than the position of the first virtual camera.

3. The information processing apparatus according to claim 2, characterized in that the determination means determines the position of the second virtual camera to be higher than the position of the first virtual camera when the position of the first virtual camera and the position of the point of focus are at the same height.

4. The information processing apparatus according to claim 1, characterized in that the position of the second virtual camera is at a predetermined distance from the first virtual camera.

5. The information processing apparatus according to claim 1, characterized in that the position of the second virtual camera is at the same height as the position of the first virtual camera.

6. The information processing apparatus according to claim 1, characterized in that the position of the second virtual camera is a position away from the position obtained by projecting the point of gaze onto a predetermined surface and a straight line passing through the first virtual camera.

7. The information processing apparatus according to claim 1, characterized in that the acquisition means acquires information indicating the position of the first virtual camera and information indicating the position of the point of gaze based on user operation.

8. An acquisition step of acquiring information indicating the position of a first virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by imaging by a plurality of imaging devices, and information indicating the position of the gaze point of the first virtual camera, The process includes a determination step of determining the position of a second virtual camera at a position away from the straight line passing through the position of the first virtual camera and the position of the point of gaze, based on the position of the first virtual camera and the position of the point of gaze. The distance from the position of the second virtual camera to the position of the first virtual camera is greater than the distance from the position of the second virtual camera to the position of the first virtual camera. An information processing method characterized by the following.

9. A program for causing a computer to execute the information processing method described in claim 8.

Images