Display control device, display control method, and program

Abstract

To provide a technique that allows a user to easily recognize a position of a gazing point of a virtual viewpoint in a virtual space.SOLUTION: A display control device 130 determines an object size indicating a gazing point when distance from a position of a first virtual viewpoint to a position of the gazing point is second distance, which is longer than a first distance, to be larger than a size of an object indicating the gazing point when distance from the position of the first virtual viewpoint to the position of the gazing point is a first distance, and performs control of displaying a second virtual viewpoint image corresponding to the second virtual viewpoint that is different from the first virtual viewpoint and where a distance from the position of the first virtual viewpoint to the position of the second virtual viewpoint is a predetermined distance, the second virtual viewpoint image including an object indicating the gazing point of the determined size and an object indicating the first virtual viewpoint.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a display control device, a display control method, and a program.

Background Art

[0002] In recent years, a technique of installing a plurality of physical cameras at different positions for synchronous shooting and generating a virtual viewpoint image as if taken from a virtual camera viewpoint using a plurality of images obtained by shooting has attracted attention. Patent Document 1 discloses a technique of generating a virtual viewpoint image using images obtained by arranging a plurality of cameras so as to surround a subject and shooting the subject. Further, Patent Document 1 discloses that a user can specify the position of a virtual camera and the position of a fixation point indicating where to fixate when viewed from the virtual camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, there is a problem that it is difficult for a user to intuitively grasp the position of a fixation point in a virtual space.

[0005] Therefore, in view of the above problems, an object of the present disclosure is to provide a technique that facilitates a user to grasp the position of a fixation point of a virtual viewpoint in a virtual space.

Means for Solving the Problems

[0006] One aspect of the display control device of the present disclosure includes information indicating the position of a first virtual viewpoint corresponding to a first virtual viewpoint image generated based on a plurality of captured images obtained by capturing images with a plurality of imaging devices, acquisition means for acquiring information indicating the position of a fixation point corresponding to the first virtual viewpoint, and when the distance from the position of the first virtual viewpoint to the position of the fixation point is a first distance, determining means for determining to increase the size of the object indicating the fixation point compared to the size of the object indicating the fixation point when the distance from the position of the first virtual viewpoint to the position of the fixation point is a second distance longer than the first distance, and a second virtual viewpoint different from the first virtual viewpoint, and a second virtual viewpoint image corresponding to the second virtual viewpoint, wherein the distance from the position of the first virtual viewpoint to the position of the second virtual viewpoint is a predetermined distance, and display control means for performing control to display a second virtual viewpoint image including an object indicating the fixation point of the determined size and an object indicating the first virtual viewpoint.

Advantages of the Invention

[0007] According to the present disclosure, a user can easily grasp the position of the fixation point of a virtual viewpoint in a virtual space.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the components described in the following embodiments are examples of the embodiments, and the present disclosure is not limited thereto.

[0010] The virtual viewpoint image is an image generated by a user and / or a dedicated operator freely operating the position and orientation of a virtual camera, and is also called a free viewpoint image or an arbitrary viewpoint image. In the present disclosure, the case where the virtual viewpoint is specified by a user operation will be mainly described, but the virtual viewpoint may be automatically specified based on the result of image analysis or the like. Also, unless otherwise specified, the term "image" will be described as including both the concepts of moving images and still images.

[0011] A virtual camera is a virtual camera that is different from a plurality of imaging devices actually installed around an imaging area, and is a concept for conveniently explaining a virtual viewpoint related to the generation of a virtual viewpoint image. That is, the virtual viewpoint image can be regarded as an image captured from a virtual viewpoint set in a virtual space associated with the imaging area. And the position and orientation of the viewpoint in the virtual imaging can be represented as the position and orientation of the virtual camera. In other words, it can be said that the virtual viewpoint image is an image that simulates the captured image obtained by the camera when it is assumed that a camera exists at the position of the virtual viewpoint set in the space. In this embodiment, the content of the change of the virtual viewpoint over time is denoted as a virtual camera path. However, it is not essential to use the concept of a virtual camera to implement the configuration of this embodiment. That is, at least information representing a specific position in the space and information representing the orientation are set, and a virtual viewpoint image may be generated according to the set information.

[0012] The imaging device only needs to have a physical camera. In addition to the physical camera, the imaging device may also have functions for performing various image processes. For example, the imaging device may have a processing unit for performing foreground-background separation processing. Also, the imaging device may have a control unit for performing transmission control to transmit an image of a part of the captured image. Further, the imaging device may have a plurality of physical cameras.

[0013] <Embodiment 1> FIG. 1 is a diagram showing an image processing system 100 according to this embodiment. The image processing system 100 includes a plurality of imaging devices 110, an image generation device 120, a display control device 130, and a display 140. Each of the imaging devices 110, the image generation device 120, and the display control device 130 are connected via a communication cable such as a LAN (Local Area Network) cable. In this embodiment, it is assumed that the communication cable is a LAN cable, but the communication cable is not limited to the embodiment. Also, the image generation device 120 and the display 140 are connected via a video signal transmission cable.

[0014] The imaging device 110 is, for example, a digital camera capable of photographing images (still images and moving images). FIG. 2 is a diagram showing an installation example of the imaging device 110. Each imaging device 110 is installed so as to surround a specific area in a stadium or the like, and photographs an image (video) of a subject within the area. The photographed image is transmitted from the imaging device 110 to the image generation device 120. Only the corresponding image of a partial area (for example, the area of the subject) of the photographed image may be transmitted.

[0015] The image generation device 120 is, for example, a server device, and has a database function and an image processing function. The image generation device 120 stores, as a background image, an image photographed in a scene where there is no subject in advance, such as before the start of photographing a subject in a stadium. The image generation device 120 also stores the captured images obtained by the imaging device 110. Further, when the image generation device 120 receives virtual viewpoint information and reproduction time information (for example, time code) by a user operation of the display control device 130, it generates a virtual viewpoint image based on the stored captured images. Here, the virtual viewpoint information is information indicating a three-dimensional position of a virtual viewpoint (virtual viewpoint) in a virtual space, a line-of-sight direction from the virtual viewpoint, an angle of view, a fixation point position, and the like. The virtual viewpoint information shall include at least relative positions with respect to a predetermined origin position such as the center in the photographed stadium, that is, front-back, left-right, and up-down position information with respect to the origin position, and directions of orientation from the predetermined position, that is, direction information of angles with respect to the front-back, left-right, and up-down axes. The virtual viewpoint information shall also include fixation point position information indicating which three-dimensional position is being focused on from the virtual viewpoint position, and distance information from the fixation point position to the virtual viewpoint position.

[0016] The reproduction time information is time information at the time of photographing of a captured image, which consists of hours, minutes, seconds, the number of frames per second, etc. By specifying the reproduction time, a scene at the recorded time is generated as a virtual viewpoint. Note that, for example, the number of frames per second is 60 frames or the like. Based on a time server (not shown), a plurality of imaging devices 110 perform synchronous photographing, and the time information at the time of photographing indicates the photographing timing in the plurality of imaging devices 110.

[0017] In addition, in a scene where a subject exists, the image processing apparatus 120 may perform separation processing by image processing on a foreground such as a specific object that becomes the subject as a specific object image. Note that the specific object may be an object in which an image pattern of not only a person but also an implement such as a ball is predetermined.

[0018] Assume that the virtual viewpoint image corresponding to the virtual viewpoint information is generated from the background image and the specific object image managed in the database. As a method for generating the virtual viewpoint image, for example, model-based rendering (MBR) is used. MBR is a method for generating a virtual viewpoint image using a three-dimensional shape generated based on a plurality of captured images obtained by photographing a subject from a plurality of directions. Specifically, it is a technique for generating, as an image, the appearance of a scene from a virtual viewpoint using the three-dimensional shape (model) of a target scene obtained by a three-dimensional shape restoration method such as a volume intersection method or multi-view stereo (MVS). Note that a rendering method other than MBR may be used as the method for generating the virtual viewpoint image. The generated virtual viewpoint image is transmitted to the display 140 via a video signal transmission cable.

[0019] The display control device 130 is, for example, a PC (Personal Computer) or a tablet. The viewpoint controller 131 is a device for setting parameters such as the position and orientation of a virtual camera. For example, the viewpoint controller 131 is a mouse, a keyboard, a joystick, a 6-axis controller, a touch panel, or a game controller. The time controller 132 is a device for setting the playback time, and is, for example, an operation device equipped with a turntable. The user operates the viewpoint controller 131 and the time controller 132. The display control device 130 receives information on user operations from the viewpoint controller 131 and the time controller 132. Then, the display control device 130 converts the information into virtual viewpoint information indicating the position and orientation of the virtual camera and playback time information according to the operation amount and the like, and transmits the information to the image generation device 120.

[0020] Note that, by the output from the display control device 130 using the operation device, not only continuous movement but also movement to a predetermined virtual viewpoint set in advance, such as the front position, back position, or position looking down from above of the subject in the virtual space, is possible. Also, by setting the playback time in advance, it is possible for the virtual viewpoint to move instantaneously to that time. Further, the display control device 130 displays an object in the three-dimensional space based on a user operation via an application displayed on the display unit 305 by executing a control program described later.

[0021] FIG. 3 is a diagram showing the hardware configuration of the display control device 130. The display control device 130 includes a CPU 301, a ROM 302, a RAM 303, an HDD 304, a display unit 305, an input unit 306, and a communication unit 307. The CPU 301 controls the entire display control device 130 using control programs and data stored in the ROM 302 and the RAM 303. Note that the display control device 130 may have one or more dedicated hardware different from the CPU 301, and at least a part of the processing by the CPU 301 may be executed by the dedicated hardware. Examples of such dedicated hardware include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor).

[0022] The ROM 302 stores programs that do not require modification. The RAM 303 temporarily stores programs and data supplied from the HDD 304, and data supplied from the outside via the communication unit 307. Also, the RAM 303 is used as a temporary storage area such as the main memory and work area of the CPU 301. The HDD 304 stores various data and various programs.

[0023] The display unit 305 is composed of, for example, a liquid crystal display, an LED, etc., and displays various types of information. The input unit 306 can connect a keyboard, a mouse, a six-axis controller, etc., and accepts various operations by the user. The communication unit 307 performs communication processing with an external device via a network. Note that, as the network, Ethernet (registered trademark) can be mentioned. Also, as another example, the communication unit 307 may communicate with an external device wirelessly. The system bus 308 connects each part of the display control device 130 and transmits information.

[0024] Note that the functions and processes of the display control device 130 described later are realized by the CPU 301 reading out a program stored in the ROM 302 or the HDD 304 and executing this program. Also, the hardware configuration of the image generation device 120 is the same as the hardware configuration of the display control device 130.

[0025] FIG. 4 is a diagram showing the functional configuration of the display control device 130. The controller operation acquisition unit 133 periodically acquires operation information for the virtual viewpoint acquired via the viewpoint controller 131 and the time controller 132.

[0026] The controller operation acquisition unit 133 converts the operation information for the virtual viewpoint into virtual viewpoint movement amount information and playback time movement amount information, and outputs it to the first virtual viewpoint information determination unit 134. The virtual viewpoint movement amount is the amount of movement with respect to the position of the current virtual viewpoint and the line-of-sight direction from the virtual viewpoint. Also, the playback time movement amount is the amount of movement with respect to the current playback time. These pieces of information are determined by a set conversion coefficient with respect to the operation amount of the user operation input to the viewpoint controller 131 and the time controller 132.

[0027] The first virtual viewpoint information determination unit 134 determines virtual viewpoint information corresponding to the position of the virtual viewpoint specified by the user operation and the line-of-sight direction from the virtual viewpoint based on the input virtual viewpoint movement amount information. Then, the first virtual viewpoint information determination unit 134 outputs the virtual viewpoint information as the first virtual viewpoint information to the virtual viewpoint object generation unit 136. Note that the first virtual viewpoint information shall include fixation point position information indicating the position of the fixation point corresponding to the first virtual viewpoint. Similarly, the first virtual viewpoint information determination unit 134 outputs the first virtual viewpoint information to the fixation point object generation unit 137, the second virtual viewpoint information determination unit 139, and the virtual viewpoint information transmission unit 142. Note that the virtual space coordinate system is the same as the coordinate system of each imaging device 110. For example, the center of a stadium or the like may be the center of the coordinate system, or it may be appropriately set by the user. The virtual viewpoint information is expressed as a position in the three-dimensional direction in this coordinate system. Note that the fixation point in the present disclosure is a point located on the optical axis of the virtual camera and having three-dimensional coordinates. Also, the distance between the fixation point and the virtual camera is determined based on the user operation.

[0028] The time code determination unit 135 determines the reproduction time information specified by the user operation based on the input reproduction time movement amount, and outputs the reproduction time information to the virtual viewpoint information transmission unit 142. Note that the reproduction time is based on the date and time when each imaging device 110 started shooting. Also, the reproduction time may be the shooting time itself, or may be expressed as the time elapsed from the shooting start time with the shooting start time set to 0.

[0029] The virtual viewpoint object generation unit 136 acquires the first virtual viewpoint information from the first virtual viewpoint information determination unit 134. Further, the virtual viewpoint object generation unit 136 acquires virtual viewpoint object information from the object information holding unit 138, which will be described later. The virtual viewpoint object generation unit 136 generates a virtual viewpoint object that enables the user to recognize the position of the virtual viewpoint and the line-of-sight direction from the virtual viewpoint by a user operation in the virtual space based on the first virtual viewpoint information and the virtual viewpoint object information. The virtual viewpoint object includes data such as a shape for expressing the virtual viewpoint object and information on its position in the space. Further, the virtual viewpoint object generation unit 136 outputs the virtual viewpoint object to the second virtual viewpoint information determination unit 139.

[0030] The fixation point object generation unit 137 acquires the first virtual viewpoint information from the first virtual viewpoint information determination unit 134. Further, the fixation point object generation unit 137 acquires fixation point object information from the object information holding unit 138. The fixation point object generation unit 137 generates a fixation point object that enables the user to recognize the position of the fixation point of the virtual viewpoint by a user operation in the virtual space based on the fixation point position information included in the first virtual viewpoint information and the fixation point object information. The fixation point object includes data such as a shape for expressing the fixation point object and information on its position in the space. Further, the fixation point object generation unit 137 outputs the fixation point object to the second virtual viewpoint information determination unit 139.

[0031] The object information holding unit 138 pre-holds object information such as virtual viewpoint object information and fixation point object information, and outputs the corresponding object information. That is, the object information holding unit 138 outputs the virtual viewpoint object information to the virtual viewpoint object generation unit 136. Also, the object information holding unit 138 outputs the fixation point object information to the fixation point object generation unit 137. Here, the object information is a three-dimensional model, which is data having three-dimensional coordinates created by dedicated software or the like, and is data indicating a three-dimensional shape by connecting a plurality of three-dimensional coordinates (vertices). Also, by associating and holding color information and texture images with the surfaces formed by connecting a plurality of vertices, it becomes an object recognizable by the user. For example, the object information may be mesh data composed of a plurality of polygons.

[0032] The second virtual viewpoint information determination unit 139 acquires a virtual viewpoint object from the virtual viewpoint object generation unit 136 and acquires a fixation point object from the fixation point object generation unit 137. Also, the second virtual viewpoint information determination unit 139 acquires first virtual viewpoint information from the first virtual viewpoint information determination unit 134. Based on the first virtual viewpoint information, the second virtual viewpoint information determination unit 139 determines second virtual viewpoint information at which the virtual viewpoint object and the fixation point object can be displayed, and outputs the second virtual viewpoint information to the second virtual viewpoint image generation unit 140. Specifically, the second virtual viewpoint information is information indicating the position of a virtual viewpoint different from the virtual viewpoint operated by the user and the line-of-sight direction from the virtual viewpoint. For example, the position of the virtual viewpoint indicated by the second virtual viewpoint information may be a position behind the first virtual viewpoint (the virtual viewpoint operated by the user) indicated by the first virtual viewpoint information. Also, the line-of-sight direction from the second virtual viewpoint indicated by the second virtual viewpoint information may be a direction in which the fixation point object can be displayed. Also, the position of the second virtual viewpoint and the line-of-sight direction from the virtual viewpoint may be a position and an orientation in which the virtual viewpoint object can be displayed. By "behind" is not meant being connected on a straight line, but rather that the position of the second virtual viewpoint is on the side opposite to the fixation point side with respect to the position of the first virtual viewpoint. Also, the position of the second virtual viewpoint is not included in the field of view of the first virtual viewpoint specified by the first virtual viewpoint information. On the other hand, the field of view of the second virtual viewpoint includes the position of the first virtual viewpoint and the position of the fixation point.

[0033] Note that "displayable" means that it can be displayed in the virtual viewpoint image generated based on the second virtual viewpoint information on the display device. In this virtual viewpoint image, the virtual viewpoint object and the fixation point object do not necessarily need to be displayed simultaneously. For example, the fixation point object may be displayed and the virtual viewpoint object may not be displayed. Alternatively, for example, either object may be made non-displayed according to the user's display instruction. Also, these objects may be displayed semi-transparently. In this case, it is suppressed that these objects shield each other, and it becomes easier for the user to recognize the objects. Also, when there is a subject on the virtual viewpoint image, it is suppressed that the subject is shielded, and it becomes easier for the user to recognize the subject.

[0034] The second virtual viewpoint image generation unit 140 acquires the second virtual viewpoint information from the second virtual viewpoint information determination unit 139, and generates a second virtual viewpoint image including a virtual viewpoint object and a fixation point object based on the second virtual viewpoint information. This second virtual viewpoint image may be generated based on the imaging device 110, or may be generated based on a background model such as a stadium previously stored in the image generation device 120. This second virtual viewpoint image may be generated by the display control unit 141. The second virtual viewpoint image generation unit 140 transmits the generated second virtual viewpoint image to the display control unit 141.

[0035] The display control unit 141 performs control to display the second virtual viewpoint image acquired from the second virtual viewpoint image generation unit 140. That is, by displaying it on the display unit 305 of the display control device 130, the position of the virtual camera (first virtual viewpoint) based on the user's virtual viewpoint operation and the display of its fixation point are performed. This display is performed in the second virtual viewpoint image generated based on the second virtual viewpoint information.

[0036] The virtual viewpoint information transmission unit 142 transmits the first virtual viewpoint information input from the first virtual viewpoint information determination unit 134 and the playback time information input from the time code determination unit 135 to the image generation device 120. Note that the virtual viewpoint image generated by the image generation device 120 based on the virtual viewpoint information and the playback time information is output to and displayed on the display 140. The virtual viewpoint image generated by the image generation device 120 is the first virtual viewpoint image viewed from the first virtual viewpoint, and is different from the second virtual viewpoint image viewed from the second virtual viewpoint.

[0037] Here, an example in which the second virtual viewpoint image is displayed on the display unit 305 in FIG. 5 is shown. A virtual viewpoint object 501 and a fixation point object 502 are displayed on the screen of the display unit 305 of the display control device 130. In the present embodiment, the larger the distance between the position of the virtual viewpoint and the position of the fixation point in the virtual space, the larger the size of the fixation point object. For example, FIG. 5(a) is an example in which the distance between the position of the virtual viewpoint and the position of the fixation point is large, but the size of the fixation point object is not changed. The fixation point object far from the second virtual viewpoint appears small, and the visibility is reduced. In FIG. 5(b), the size of the fixation point object is increased by the process described later, and the visibility is improved by the fixation point object being prominently displayed in the second virtual viewpoint image.

[0038] FIG. 6 is a flowchart showing the operation of the display control device 130 according to the present embodiment. The following processing is performed by the CPU 301 reading and executing a program stored in the ROM 302 or the HDD 304.

[0039] In step S601, the controller operation acquisition unit 133 acquires the movement amount information of the lever of a 6-axis controller such as a joystick from the viewpoint controller 131. Also, the controller operation acquisition unit 133 acquires the movement amount information of, for example, a turntable rotated by the user from the time controller 132. Next, the controller operation acquisition unit 133 converts the movement amount information acquired from each of the viewpoint controller 131 and the time controller 132 into virtual viewpoint movement amount information and playback time movement amount information, and outputs the information to the first virtual viewpoint information determination unit 134.

[0040] In step S602, the first virtual viewpoint information determination unit 134 determines the first virtual viewpoint information based on the input virtual viewpoint movement amount information. Then, the first virtual viewpoint information determination unit 134 outputs the information to the virtual viewpoint object generation unit 136, the fixation point object generation unit 137, and the projected fixation point object generation unit 138. The first virtual viewpoint is the virtual viewpoint that is the operation target of the user and corresponds to the virtual viewpoint image displayed on the display 140.

[0041] In step S603, the virtual viewpoint object generation unit 136 generates a virtual viewpoint object. Specifically, the virtual viewpoint object generation unit 136 acquires the virtual viewpoint information from the first virtual viewpoint information determination unit 134. Also, the virtual viewpoint object generation unit 136 acquires the virtual viewpoint object information from the object information holding unit 139. The virtual viewpoint object generation unit 136 generates a virtual viewpoint object that enables the user to recognize the position of the first virtual viewpoint in the virtual space and the line-of-sight direction from the first virtual viewpoint based on the virtual viewpoint information and the virtual viewpoint object information. For example, the virtual viewpoint object may have the shape of a camera or may have other shapes.

[0042] In step S604, the fixation point object generation unit 137 determines the size of the fixation point object. Specifically, the fixation point object generation unit 137 acquires virtual viewpoint information from the first virtual viewpoint information determination unit 134. From the position of the first virtual viewpoint and the position of the fixation point included in the virtual viewpoint information, the distance between the position of the first virtual viewpoint and the position of the fixation point is calculated. Next, based on the graph shown in FIG. 7, which represents the relationship between the distance between the position of the first virtual viewpoint and the position of the fixation point and the size of the fixation point object, the size of the fixation point object is determined. When the distance between the position of the first virtual viewpoint and the position of the fixation point is less than the threshold D_th, the size coefficient of the fixation point object is set to 1.0. The size coefficient is a coefficient indicating how much the size is enlarged with respect to the initial value of the size of the fixation point object. That is, the acquired size of the fixation point object is determined to be the same size. Here, the threshold D_th may be set separately by the user or determined by the shooting target. For example, when shooting a baseball to generate a virtual viewpoint image, referring to the distance between the home base and the second base, for example, the threshold D_th is set to 40 m. Next, when the distance between the position of the first virtual viewpoint and the position of the fixation point is equal to or greater than the threshold D_th, the size coefficient of the fixation point object is linearly determined, and a value obtained by multiplying the size coefficient by a predetermined value is added to the initial value of the size of the fixation point object to determine the size. However, the determination of the size coefficient is not limited to this, and it may be non-linear such that it becomes larger as it gets farther away. Also, the size may be determined by multiplying the initial value of the size of the fixation point object by the size coefficient. The predetermined value to be multiplied by the size coefficient is assumed to be set separately by the user. As a result, when the first virtual viewpoint is more than the threshold D_th away from the fixation point, the fixation point object is linearly (proportionally) enlarged. Note that the design of the function for determining the size coefficient is not limited to this example, and a non-linear function or the like may be used as long as there is a positive correlation between the distance and the size coefficient. Also, by designing the size coefficient for a distance of less than the threshold D_th to be less than 1.0, the fixation point may be scaled down so that the fixation point does not look too large when approaching the threshold D_th. However, from the viewpoint of the continuity of the operation, it is desirable that the size of the fixation point also changes continuously, so it is desirable that the function for determining the size coefficient is also continuous.

[0043] In step S605, the fixation point object generation unit 137 generates a fixation point object. Specifically, the fixation point object generation unit 137 acquires fixation point object information from the object information holding unit 130. Based on the fixation point position information included in the virtual viewpoint information and the fixation point object information, the fixation point object generation unit 137 generates a fixation point object that enables the position of the fixation point of the first virtual viewpoint in the virtual space to be recognizable by the user. The size of the generated fixation point object is the size determined in step S604. Note that the fixation point object may have a spherical shape or a quadrangular prism shape.

[0044] In step S606, the second virtual viewpoint information determination unit 142 determines the second virtual viewpoint information. Specifically, the second virtual viewpoint information determination unit 142 acquires the first virtual viewpoint information from the first virtual viewpoint information determination unit 134. Also, the second virtual viewpoint information determination unit 142 acquires the virtual viewpoint object and the fixation point object. Based on the acquired information, the second virtual viewpoint information determination unit 142 determines the second virtual viewpoint information. The second virtual viewpoint information indicates information of a second virtual viewpoint from which the virtual viewpoint object and the fixation point object can be virtually photographed. More specifically, the position of the second virtual viewpoint is a position behind the position of the first virtual viewpoint at a predetermined distance, and the second virtual viewpoint information is determined such that the line-of-sight direction from the second virtual viewpoint is a direction in which the fixation point object can be virtually photographed. The second virtual viewpoint information determination unit 142 outputs the second virtual viewpoint information to the display control unit 143.

[0045] In step S607, the second virtual viewpoint image generation unit 140 generates a second virtual viewpoint image. Specifically, the second virtual viewpoint image generation unit 140 acquires the second virtual viewpoint information from the second virtual viewpoint information determination unit 139, and based on the second virtual viewpoint information, generates a second virtual viewpoint image including the virtual viewpoint object and the fixation point object. The second virtual viewpoint image generation unit 140 outputs the generated second virtual viewpoint image to the display control unit 141.

[0046] In step S608, the display control unit 143 acquires a second virtual viewpoint image from the second virtual viewpoint image generation unit 140. Next, the display control unit 143 displays the second virtual viewpoint image including the virtual viewpoint object and the fixation point object on the display unit 305 of the display control device 130. Thereby, the positions of the first virtual viewpoint, its fixation point, and the projected fixation point in the virtual space based on the user's virtual viewpoint operation are displayed. This enables the user to easily grasp the position of the fixation point in the virtual space.

[0047] As described above, in the present embodiment, when the distance between the position of the first virtual viewpoint and the position of the fixation point is greater than the threshold, by increasing the size of the fixation point object, the visibility of the fixation point in the second virtual viewpoint image used as a reference when the user operates the first virtual viewpoint is improved.

[0048] Note that in the present embodiment, the second virtual viewpoint is set at a position a predetermined distance behind the virtual viewpoint, but it is not limited to this. For example, the second virtual viewpoint may be set near the fixation point. In that case, the second virtual viewpoint is determined on the opposite side of the first virtual viewpoint with respect to the axis of the fixation point so that the virtual viewpoint object enters the viewing angle. In this case, the size of the virtual viewpoint object is changed according to the distance between the virtual viewpoint object and the fixation point object. Also, the second virtual viewpoint may be determined at a position away from both the fixation point and the virtual viewpoint. In that case, the sizes of both the fixation point object and the virtual viewpoint object are changed.

[0049] Also, in the present embodiment, the visibility of the fixation point or the virtual viewpoint is improved by changing the size of each object, but other parameters related to the object display may be changed. For example, the color of the object included in the parameters may be changed entirely or partially such as the contour part. In addition, the visibility may be improved by periodically changing the size and color of the object.

[0050] In addition, the definition of object information is not limited to a three-dimensional model, and it may be held, input, and resized as an object that is always drawn facing the second virtual viewpoint such as a two-dimensional image.

[0051] Also, in this embodiment, the display device 140 and the display unit 305 have been described as separate blocks, but it is not limited to this. For example, the second virtual viewpoint image may be displayed on the display device 140. It is also possible to configure it with multiple windows, including a window for displaying the first virtual viewpoint image and a window for displaying the second virtual viewpoint image, and display them simultaneously.

[0052] Note that, although the virtual viewpoint information has been described using the three-dimensional position and the line-of-sight direction from the virtual viewpoint, it is not limited to this, and information representing the posture including rotation or the viewing angle may also be used.

[0053] <Modification Example of Embodiment 1> In Embodiment 1, the size of the fixation point object was changed according to the distance between the first virtual viewpoint and the fixation point, but it may be changed according to their respective three-dimensional positions. Hereinafter, the description of the same configuration as that in Embodiment 1 will be omitted.

[0054] In step S505, in each of the fixation point object generation unit 137 and the virtual viewpoint object generation unit 136, when the three-dimensional position of each object is included in a predetermined partial space, the size of the object is determined so as to have a predetermined size. For example, when photographing a baseball, when the three-dimensional position of the first virtual viewpoint belongs to the inside of a spherical partial space with a radius of 20 m from the home base, the size coefficient of the process for determining the size of the fixation point object is set to 1.0. When it belongs to the outside of the partial space and the inside with a radius of 40 m from the home base, the size coefficient is set to 2.0. When it belongs to the outside of the partial space and the outside with a radius of 40 m from the home base, the size coefficient is set to 3.0. As described above, at least one predetermined partial space in the virtual space is set, and parameters such as the size coefficient of the fixation point object are set in each partial space, so that the size of the fixation point object can be determined according to the position of the first virtual viewpoint.

[0055] As a result of the above processing, the visibility of the fixation point can be improved, and at the same time, it becomes possible to intuitively grasp which space the virtual viewpoint belongs to.

[0056] <Embodiment 2> In Embodiment 1, the size of the fixation point object was determined according to the distance between the first virtual viewpoint and the fixation point. However, in Embodiment 2, the size of the fixation point object is determined according to the distance between the second virtual viewpoint and the fixation point. Hereinafter, the description of the same configuration as that in Embodiment 1 will be omitted.

[0057] FIG. 8 is a diagram showing the functional configuration of the display control device 130 according to Embodiment 2.

[0058] The second virtual viewpoint information determination unit 801 acquires the first virtual viewpoint information from the first virtual viewpoint information determination unit 134. Next, based on the first virtual viewpoint information, the second virtual viewpoint information indicating the second virtual viewpoint that includes the first virtual viewpoint and the fixation point within the viewing angle is determined. Since the position of the second virtual viewpoint and the line-of-sight direction from the second virtual viewpoint are the same as those in Embodiment 1, the description thereof is omitted. The second virtual viewpoint information determination unit 801 outputs the determined second virtual viewpoint information to the fixation point object generation unit 802.

[0059] The fixation point object generation unit 802 acquires the fixation point object information from the object information holding unit 138. Also, the fixation point object generation unit 802 acquires the first virtual viewpoint information from the first virtual viewpoint information determination unit 134. Further, the fixation point object generation unit 802 acquires the second virtual viewpoint information from the second virtual viewpoint information determination unit 801. The fixation point object generation unit 802 determines the size of the fixation point object based on the fixation point position information included in the acquired first virtual viewpoint information and the second virtual viewpoint information. Specifically, the size of the fixation point object is determined based on the distance between the second virtual viewpoint and the fixation point. Next, a fixation point object with the determined size is generated and output to the second virtual viewpoint image generation unit 804.

[0060] The virtual viewpoint object generation unit 803 acquires the first virtual viewpoint information from the first virtual viewpoint information determination unit 134. Also, the virtual viewpoint object generation unit 136 acquires the virtual viewpoint object information from the object information holding unit 138, which will be described later. The virtual viewpoint object generation unit 136 generates a virtual viewpoint object that enables the user to recognize the position of the virtual viewpoint and the line-of-sight direction from the virtual viewpoint by a user operation in the virtual space based on the first virtual viewpoint information and the virtual viewpoint object information. The virtual viewpoint object generation unit 803 outputs the generated virtual viewpoint object to the second virtual viewpoint image generation unit 804.

[0061] The second virtual viewpoint image generation unit 804 acquires second virtual viewpoint information from the second virtual viewpoint information determination unit 801. Further, the second virtual viewpoint image generation unit 804 acquires a fixation point object from the fixation point object generation unit 802. Further, the second virtual viewpoint image generation unit 804 acquires a virtual viewpoint object from the virtual viewpoint object generation unit 803. The second virtual viewpoint image generation unit 804 generates a second virtual viewpoint image including the fixation point object and the virtual viewpoint object based on the second virtual viewpoint information. The second virtual viewpoint image generation unit 804 transmits the generated second virtual viewpoint image to the display control unit 141.

[0062] FIG. 9 is a flowchart showing the operation of the display control means 130 according to Embodiment 2. By the CPU 301 reading and executing a program stored in the ROM 302 or the HDD 304, the following processing is performed. Note that description of steps that perform the same processing as the flowchart shown in FIG. 6 is omitted.

[0063] In step S901, the second virtual viewpoint information determination unit 801 acquires first virtual viewpoint information from the first virtual viewpoint information determination unit 134. Next, based on the first virtual viewpoint information, second virtual viewpoint information indicating a second virtual viewpoint that includes the first virtual viewpoint and the fixation point within the viewing angle is determined. Since the position of the second virtual viewpoint and the viewing direction from the second virtual viewpoint are the same as those in Embodiment 1, the description thereof is omitted. The second virtual viewpoint information determination unit 801 outputs the determined second virtual viewpoint information to the fixation point object generation unit 802.

[0064] In step S902, the fixation point object generation unit 802 generates a fixation point object. Specifically, the fixation point object generation unit 802 determines the size of the fixation point object based on the fixation point position information included in the acquired first virtual viewpoint information and the second virtual viewpoint information. Specifically, the size of the fixation point object is determined based on the distance between the second virtual viewpoint and the fixation point.

[0065] Through the above processing, the size of the fixation point object is determined based on the distance between the second virtual viewpoint and the fixation point, and the visibility of the fixation point in the second virtual viewpoint image can be improved.

[0066] <Embodiment 3> In Embodiment 1, the size of the spherical fixation point object was determined according to the distance between the first virtual viewpoint and the fixation point. In Embodiment 3, an example will be described in which the type of the fixation point object is replaced with an object of actual scale. In this embodiment, when the distance between the first virtual viewpoint and the fixation point approaches a threshold value, it becomes possible to easily grasp an appropriate distance in virtual viewpoint image generation. Hereinafter, descriptions of the same configurations as those in Embodiment 1 will be omitted.

[0067] The object shape holding unit 138 holds, for example, in addition to the sphere described in Embodiment 1 as an example of the type of the fixation point object, a three-dimensional model of a baseball player created in advance. At this time, the shape of the three-dimensional model of the same type may be deformed over time, and a plurality of three-dimensional models of the same type may be held. Note that the type of the fixation point object is not limited to the above. For example, the color of the fixation point object may be changed according to the distance between the first virtual viewpoint and the fixation point. Alternatively, when the distance between the first virtual viewpoint and the fixation point becomes equal to or less than the threshold value, the contour of the fixation point object on the virtual viewpoint image may be emphasized.

[0068] FIG. 10 is a diagram showing the functional configuration of the display control device 130 according to this embodiment.

[0069] The first virtual viewpoint information determination unit 1001 determines virtual viewpoint information corresponding to the position of the virtual viewpoint specified by the user operation and the line-of-sight direction from the virtual viewpoint based on the input virtual viewpoint movement amount information. The generated virtual viewpoint information is the same as that in Embodiment 1. Then, the first virtual viewpoint information determination unit 134 outputs the generated virtual viewpoint information as the first virtual viewpoint information to the object information holding unit 1002.

[0070] The object information holding unit 1002 acquires the first virtual viewpoint information from the first virtual viewpoint information determining unit 1001. Based on the acquired first virtual viewpoint information, the object information holding unit 1002 determines, among a plurality of fixation point objects held in advance, the fixation point object to be displayed in the second virtual viewpoint image. Specifically, when the distance between the first virtual viewpoint and the fixation point is smaller than the threshold value D_th2, a three-dimensional model showing the shape of a baseball player held is determined as the fixation point object. When the distance between the first virtual viewpoint and the fixation point is greater than or equal to the threshold value D_th2, a three-dimensional model of a sphere is determined as the fixation point object in the same manner as in the first embodiment. The object information holding unit 1002 outputs the determined object to the fixation point object generation unit 137.

[0071] By generating a three-dimensional model of a baseball player to be created in advance based on a plurality of captured images, when the distance between the first virtual viewpoint and the fixation point is less than D_th2, a three-dimensional model close to the size of an actual baseball player can be used as the fixation point object. Alternatively, the three-dimensional model to be generated in advance may be set as a rectangular parallelepiped with a height of 1 m. The three-dimensional model to be generated in advance only needs to be different from the three-dimensional model determined as the fixation point object when the distance between the first virtual viewpoint and the fixation point is greater than or equal to D_th2.

[0072] By the above processing, the fixation point object displayed when the distance between the first virtual viewpoint and the fixation point is less than D_th2 and the fixation point object displayed when the distance between the first virtual viewpoint and the fixation point is greater than or equal to D_th2 are different three-dimensional models. As a result, it is possible to more intuitively grasp the distance between the first virtual viewpoint and the fixation point.

[0073] In addition, in this embodiment, an example of changing the type of the gaze point object according to the distance between the first virtual viewpoint and the gaze point has been described, but the present invention is not limited to the above. For example, when the distance between the first virtual viewpoint and the gaze point becomes equal to or greater than a threshold value, text indicating the approximate position of the gaze point may be displayed on the second virtual viewpoint image. Specifically, assume that baseball is the shooting target, the threshold value is 20 m, the distance between the first virtual viewpoint and the gaze point is 25 m, and the position of the gaze point is within 3 m around the three-dimensional position of the home base. At this time, text such as "around the home base" may be displayed around the gaze point object on the second virtual viewpoint image. Note that it is assumed that the conditions for displaying text such as within 3 m around the three-dimensional position of the home base are set in advance for each piece of text. By doing so, the user can easily grasp the approximate position of the gaze point.

[0074] <Embodiment 4> In Embodiment 1, a method of improving the visibility of the gaze point on the second virtual viewpoint image by increasing the size of the gaze point object when the distance between the first virtual viewpoint and the gaze point is large has been described. In Embodiment 4, a method will be described in which it becomes possible to more accurately grasp the position of the gaze point by additionally generating and displaying a guide object for additionally presenting the background object and their respective mutual positional relationships. Hereinafter, descriptions of the same configurations as those in Embodiment 1 will be omitted.

[0075] FIG. 11 shows the functional configuration of the display control device 130 according to the present embodiment.

[0076] In addition to the function of the first virtual viewpoint information determination unit 134 described with reference to FIG. 4, the first virtual viewpoint information determination unit 1101 outputs the generated first virtual viewpoint information to the guide object generation unit 1105.

[0077] In addition to the function of the virtual viewpoint object generation unit 136 described with reference to FIG. 4, the virtual viewpoint object generation unit 1102 outputs the generated virtual viewpoint object to the guide object generation unit 1105.

[0078] In addition to the function of the fixation point object generation unit 137 described with reference to FIG. 4, the fixation point object generation unit 1103 outputs the generated fixation point object to the guide object generation unit 1105.

[0079] In addition to the function of the object information holding unit 138 described with reference to FIG. 4, the object information holding unit 1104 holds a background object which is, for example, a three-dimensional model of a baseball stadium. The object information holding unit 1104 holds a plurality of types of background objects, and outputs the background object selected by a user operation to the guide object generation unit 1105. Note that the present invention is not limited to this, and among the plurality of types of background objects, the background object to be output may be determined based on a photographed image.

[0080] The guide object generation unit 1105 inputs the fixation point information, the virtual viewpoint information, and the background object, and generates a guide object. The generated guide object is output to the second virtual viewpoint image generation unit 1106. The guide object is, for example, a three-dimensional model of a line or a plane as described later.

[0081] The second virtual viewpoint image generation unit 1106 generates a second virtual viewpoint image including the acquired guide object. The second virtual viewpoint image generation unit 1106 outputs the generated second virtual viewpoint image to the display control unit 141.

[0082] FIG. 12 shows an example of display of a guide object according to the present embodiment.

[0083] In FIG. 12(a), no guide object is displayed, and a three-dimensional model of a baseball stadium is displayed as the background object 1203, and a virtual viewpoint object 1201 and a fixation point object 1202 are displayed in the same manner as in the first embodiment.

[0084] In FIG. 12(b), as guide objects, a linear object 1206 extending from the first virtual viewpoint toward the fixation point and a plane 1205 slicing in the vertical direction of the three-dimensional virtual space including the linear object 1206 from the position and orientation of the first virtual viewpoint are displayed. Here, the plane 1205 is displayed while limiting the angular range in the vertical direction according to the viewing angle of the first virtual viewpoint. Further, the plane 1205 extends to the point where it intersects with the background object 1203, and the dashed line includes the contact point with the background object 1203. These guide objects make it easier to intuitively grasp the orientation of the first virtual viewpoint, the positional relationship with the background object 1203, and the line-of-sight direction.

[0085] In FIG. 12(c), as guide objects, a linear object 1204 and a linear object 1207 extending vertically downward from the fixation point and the first virtual viewpoint toward the background object 1203 are displayed. Also, a plane 1208 slicing in the horizontal direction of the three-dimensional virtual space from the first virtual viewpoint toward the background object 1203 is displayed. Here, similar to the plane 1205, the plane 1208 is displayed while limiting the angular range according to the viewing angle of the first virtual viewpoint. Further, since the plane 1208 extends horizontally toward the background object 1203, it is in contact with the 2F portion of the stadium seats of the background object 1203. These guide objects make it easier to intuitively grasp the heights of the fixation point and the first virtual viewpoint and the positions within the range of the infield.

[0086] As described above, by generating and displaying guide objects for assisting in presenting the mutual positional relationships of the respective objects, it becomes possible to more intuitively grasp the viewing angle from the first virtual viewpoint and the arrangements of the fixation point and the first virtual viewpoint.

[0087] Note that according to the background texture or the like, the color of each object may be set to a color with good visibility, for example, the complementary color of the average value of the color of the background around the object.

[0088] <Embodiment 5> (Adjusting the size of the fixation point object on the virtual viewpoint image) In Embodiment 1, the size of the fixation point object of the sphere was determined according to the distance between the first virtual viewpoint and the fixation point. In Embodiment 5, an example will be described in which, regardless of the distance between the second virtual viewpoint and the fixation point, the sizes of the virtual viewpoint object and the fixation point object on the second virtual viewpoint image are controlled to be the same size.

[0089] FIG. 13 shows the functional configuration of the display control device 130 according to the present embodiment.

[0090] In addition to the function of the virtual viewpoint object generation unit 136 described with reference to FIG. 4, the virtual viewpoint object generation unit 1301 outputs the generated virtual viewpoint object to the display form control unit 1304.

[0091] In addition to the function of the fixation point object generation unit 137 described with reference to FIG. 4, the fixation point object generation unit 1302 outputs the generated fixation point object to the display form control unit 1304. In Embodiment 1, the fixation point object generation unit 137 determined the size of the fixation point object, but in the present embodiment, the fixation point object generation unit 137 does not determine the size of the fixation point object.

[0092] In addition to the function of the second virtual viewpoint information determination unit 139 described with reference to FIG. 4, the second virtual viewpoint information determination unit 1303 outputs the generated second virtual viewpoint information to the display form control unit 1304.

[0093] The display form control unit 1304 acquires the virtual viewpoint object from the virtual viewpoint object generation unit 1301. The display form control unit 1304 acquires the fixation point object from the fixation point object generation unit 1302. The display form control unit 1304 acquires the second virtual viewpoint information from the second virtual viewpoint information determination unit 1303. The display form control unit 1305 controls the sizes of the virtual viewpoint object and the fixation point object on the second virtual viewpoint image to be the same size regardless of the distance between the second virtual viewpoint and the fixation point based on the acquired information. The sizes can be set by the user respectively.

[0094] By the above processing, since the sizes of the virtual viewpoint object and the fixation point object on the second virtual viewpoint image are controlled to be the same size, the visibility of the virtual viewpoint object and the fixation point object can be improved.

[0095] Although an example has been described in which the sizes of the virtual viewpoint object and the fixation point object on the second virtual viewpoint image are controlled to be the same size regardless of the distance between the second virtual viewpoint and the fixation point, the present invention is not limited thereto. For example, the sizes of the virtual viewpoint object and the fixation point object on the second virtual viewpoint image may be controlled to be the same size regardless of the distance between the first virtual viewpoint and the fixation point. Alternatively, the size of either the virtual viewpoint object or the fixation point object on the second virtual viewpoint image may be controlled to be the same size.

[0096] Although an example has been described in which the sizes of the virtual viewpoint object and the fixation point object on the second virtual viewpoint image are controlled to be the same regardless of the distance between the second virtual viewpoint and the fixation point, the present invention is not limited to this. For example, the sizes of the virtual viewpoint object and the fixation point object on the second virtual viewpoint image may be controlled to be within a predetermined range. Specifically, the range of the sizes of the virtual viewpoint object and the fixation point object on the second virtual viewpoint image is specified in advance. For example, a square frame surrounding the virtual viewpoint object and the fixation point object on the second virtual viewpoint image is set, and the range of the size of one side of the frame is set. Note that the size range is set in advance to be a range that is easy for the user to visually recognize. Here, it is set so that the size of one side is 5 to 10 pixels. A first threshold value and a second threshold value larger than the first threshold value are provided in advance according to the distance between the second virtual viewpoint and the fixation point. When the distance is less than the first threshold value, the size of one side of the frame is controlled to be 5 pixels. When the distance is equal to or greater than the first threshold value and less than the second threshold value, the size of one side of the frame is controlled to be within the range of 5 to 10 pixels. At this time, the distance and the size of one side may be in a correlation relationship, for example, a proportional relationship. When the distance is equal to or greater than the second threshold value, the size of one side of the frame is controlled to be 10 pixels. In other words, the above processing is an example of controlling the sizes of the virtual viewpoint object and the fixation point object on the second virtual viewpoint image to be within a predetermined range. Therefore, the difference between the size of the object indicating the fixation point on the second virtual viewpoint image when the distance between the second virtual viewpoint and the fixation point is the first distance and the size of the object indicating the fixation point on the second virtual viewpoint image when the distance is a second distance different from the first distance is within a predetermined range.

[0097] By the above processing, regardless of the distance between the second virtual viewpoint and the fixation point, it is possible to display the virtual viewpoint object and the fixation point object in a size that is easy for the user to visually recognize. In addition, since the sizes of the virtual viewpoint object and the fixation point object on the second virtual viewpoint image on the two-dimensional image are changed according to the distance between the second virtual viewpoint and the fixation point, it is possible to easily grasp the sense of distance between the second virtual viewpoint and the fixation point.

[0098] Although the present disclosure has been described based on a plurality of embodiments, the present disclosure is not limited to the above embodiments, and various modifications can be made based on the spirit of the present disclosure, and they are not excluded from the scope of the present disclosure.

[0099] Note that part or all of the control in the present embodiment may be supplied to an image processing system or the like via a network or various storage media by a computer program that realizes the functions of the above-described embodiments. Then, a computer (or a CPU, MPU, etc.) in the image processing system or the like may read and execute the program. In that case, the program and the storage medium storing the program will constitute the present disclosure.

[0100] Note that the disclosure of the present embodiment includes the following configurations, methods, and programs.

[0101] (Configuration 1) Acquisition means for acquiring information indicating the position of a first virtual viewpoint corresponding to a first virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices, and information indicating the position of a fixation point corresponding to the first virtual viewpoint; Determination means for determining the size of an object indicating the fixation point based on the distance from the position of the first virtual viewpoint to the position of the fixation point; Display control means for performing control to display a second virtual viewpoint image corresponding to a second virtual viewpoint determined based on the first virtual viewpoint, the second virtual viewpoint image including an object indicating the fixation point having the determined size and an object indicating the first virtual viewpoint; The determination means determines that the size of the object indicating the fixation point when the distance from the position of the first virtual viewpoint to the position of the fixation point is a second distance longer than the first distance is larger than the size of the object indicating the fixation point when the distance from the position of the first virtual viewpoint to the position of the fixation point is the first distance A display control device characterized by the above.

[0102] (Configuration 2) The size of the object indicating the fixation point is determined to be proportional to the distance from the position of the first virtual viewpoint to the position of the fixation point, according to the display control device described in Configuration 1.

[0103] (Configuration 3) The size of the object indicating the fixation point is determined to be a predetermined size when the distance from the position of the first virtual viewpoint to the position of the fixation point is less than a threshold value, and is determined to be larger as the distance from the position of the first virtual viewpoint to the position of the fixation point is larger when the distance is equal to or greater than the threshold value, according to the display control device described in Configuration 1.

[0104] (Configuration 4) The object indicating the first virtual viewpoint and the object indicating the fixation point are three-dimensional models, according to the display control device described in any one of Configurations 1 to 3.

[0105] (Configuration 5) The position of the second virtual viewpoint is a position at a predetermined distance from the position of the first virtual viewpoint, according to the display control device described in any one of Configurations 1 to 4.

[0106] (Configuration 6) Acquisition means for acquiring information indicating the position of a fixation point corresponding to a first virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices; Determination means for determining a larger size of an object indicating the fixation point on the second virtual viewpoint image when the distance from the position of the second virtual viewpoint, which is different from the first virtual viewpoint, to the position of the fixation point is a second distance that is longer than a first distance, compared to the size of the object indicating the fixation point on the second virtual viewpoint image when the distance from the position of the second virtual viewpoint to the position of the fixation point is the first distance; Display control means for performing control to display a second virtual viewpoint image corresponding to the second virtual viewpoint, which is different from the first virtual viewpoint, and including the object indicating the fixation point with the determined size; A display control device, characterized by comprising the above.

[0107] (Configuration 7) The determination means determines the size of the object indicating the fixation point on the second virtual viewpoint image corresponding to the second virtual viewpoint when the distance from the position of the second virtual viewpoint different from the first virtual viewpoint to the position of the fixation point is the first distance, and the size of the object indicating the fixation point on the second virtual viewpoint image when the distance from the position of the second virtual viewpoint to the position of the fixation point is a second distance longer than the first distance, and determines the size of the object indicating the fixation point so that they are the same size. The display control device according to Configuration 6, characterized in that.

[0108] (Method 1) An acquisition step of acquiring information indicating the position of a first virtual viewpoint corresponding to a first virtual viewpoint image generated based on a plurality of captured images obtained by capturing with a plurality of imaging devices, and information indicating the position of a fixation point corresponding to the first virtual viewpoint; A determination step of determining a larger size of the object indicating the fixation point when the distance from the position of the first virtual viewpoint to the position of the fixation point is large; A display control step of performing control to display a second virtual viewpoint image corresponding to a second virtual viewpoint different from the first virtual viewpoint, the second virtual viewpoint image including an object indicating the fixation point having the determined size; A display control method, characterized by comprising:

[0109] (Method 2) An acquisition step of acquiring information indicating the position of a fixation point corresponding to a first virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by capturing with a plurality of imaging devices; A determination step of determining a larger size of the object indicating the fixation point on the second virtual viewpoint image when the distance from the position of the second virtual viewpoint different from the first virtual viewpoint to the position of the fixation point is a second distance longer than the first distance, compared to the size of the object indicating the fixation point on the second virtual viewpoint image when the distance from the position of the second virtual viewpoint to the position of the fixation point is the first distance; A display control step for performing control to display a second virtual viewpoint image corresponding to a second virtual viewpoint different from the first virtual viewpoint, the second virtual viewpoint image including an object indicating the fixation point of the determined size. A display control method characterized by having the above.

[0110] (Program) A program for causing a computer to execute the display control device according to any one of Configurations 1 to 7.

Explanation of Signs

[0111] 134 First virtual viewpoint information determination unit 136 Virtual viewpoint object generation unit 137 Fixation point object generation unit 138 Object information holding unit 139 Second virtual viewpoint information determination unit

Claims

1. Acquisition means for acquiring information indicating the position of a first virtual viewpoint corresponding to a first virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices, and information indicating the position of a fixation point corresponding to the first virtual viewpoint; Determination means for determining the size of an object indicating the fixation point based on the distance from the position of the first virtual viewpoint to the position of the fixation point; Display control means for performing control to display a second virtual viewpoint image corresponding to a second virtual viewpoint determined based on the first virtual viewpoint, the second virtual viewpoint image including an object indicating the fixation point having the determined size and an object indicating the first virtual viewpoint; The determination means determines the size of the object indicating the fixation point to be larger when the distance from the position of the first virtual viewpoint to the position of the fixation point is a second distance longer than the first distance, than when the distance from the position of the first virtual viewpoint to the position of the fixation point is the first distance. A display control device characterized by the above.

2. The display control device according to claim 1, wherein the size of the object indicating the fixation point is determined to be proportional to the distance from the position of the first virtual viewpoint to the position of the fixation point.

3. The display control device according to claim 1, wherein the size of the object indicating the fixation point is determined to be a predetermined size when the distance from the position of the first virtual viewpoint to the position of the fixation point is less than a threshold value, and is determined to be larger as the distance is larger when the distance from the position of the first virtual viewpoint to the position of the fixation point is equal to or greater than the threshold value.

4. The display control device according to claim 1, wherein the object indicating the first virtual viewpoint and the object indicating the fixation point are three-dimensional models.

5. The display control device according to claim 1, wherein the position of the second virtual viewpoint is a position at a predetermined distance from the position of the first virtual viewpoint.

6. An acquisition unit that acquires information indicating the position of a fixation point corresponding to a first virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices; When the distance from the position of the second virtual viewpoint different from the first virtual viewpoint to the position of the fixation point is a first distance, the size of an object indicating the fixation point on the second virtual viewpoint image corresponding to the second virtual viewpoint is smaller than the size of an object indicating the fixation point on the second virtual viewpoint image when the distance from the position of the second virtual viewpoint to the position of the fixation point is a second distance longer than the first distance. A determination unit that determines to increase the size; A display control unit that performs control to display a second virtual viewpoint image corresponding to a second virtual viewpoint different from the first virtual viewpoint and including an object indicating the fixation point having a determined size; A display control device, comprising:

7. The display control device according to claim 6, wherein the determination unit determines the size of the object indicating the fixation point such that the size of the object indicating the fixation point on the second virtual viewpoint image corresponding to the second virtual viewpoint when the distance from the position of the second virtual viewpoint different from the first virtual viewpoint to the position of the fixation point is a first distance is the same as the size of the object indicating the fixation point on the second virtual viewpoint image when the distance from the position of the second virtual viewpoint to the position of the fixation point is a second distance longer than the first distance.

8. An acquisition step of acquiring information indicating the position of a first virtual viewpoint corresponding to a first virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices and information indicating the position of a fixation point corresponding to the first virtual viewpoint; A determination step of determining to increase the size of the object indicating the fixation point when the distance from the position of the first virtual viewpoint to the position of the fixation point is large; A display control step of performing control to display a second virtual viewpoint image corresponding to a second virtual viewpoint different from the first virtual viewpoint, the second virtual viewpoint image including an object indicating the fixation point of the determined size. A display control method characterized by comprising the above. **Claim 9** An acquisition step of acquiring information indicating the position of a fixation point corresponding to a first virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices. A determination step of determining to increase the size of an object indicating the fixation point on the second virtual viewpoint image corresponding to the second virtual viewpoint when the distance from the position of the second virtual viewpoint different from the first virtual viewpoint to the position of the fixation point is a second distance longer than a first distance, compared to the size of the object indicating the fixation point on the second virtual viewpoint image corresponding to the second virtual viewpoint when the distance from the position of the second virtual viewpoint to the position of the fixation point is the first distance. A display control step of performing control to display a second virtual viewpoint image corresponding to a second virtual viewpoint different from the first virtual viewpoint, the second virtual viewpoint image including an object indicating the fixation point of the determined size. A display control method characterized by comprising the above. **Claim 10** A program for causing a computer to execute the display control device according to any one of claims 1 to 7.

Images