Display control device, display control method, and program

Abstract

To provide a technique that allows a user to easily recognize a position of a gazing point of a virtual viewpoint in a virtual space.SOLUTION: A display control device 130 determines an object size indicating a gazing point when distance from a position of a first virtual viewpoint to a position of the gazing point is second distance, which is longer than a first distance, to be larger than a size of an object indicating the gazing point when distance from the position of the first virtual viewpoint to the position of the gazing point is a first distance, and performs control of displaying a second virtual viewpoint image corresponding to the second virtual viewpoint that is different from the first virtual viewpoint and where a distance from the position of the first virtual viewpoint to the position of the second virtual viewpoint is a predetermined distance, the second virtual viewpoint image including an object indicating the gazing point of the determined size and an object indicating the first virtual viewpoint.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a display control device, a display control method, and a program. [Background technology]

[0002] In recent years, a technology has been attracting attention in which a plurality of physical cameras are installed at different positions, capture images synchronously, and then use the plurality of images obtained by the capture to generate a virtual viewpoint image that would appear if the image were captured from a virtual camera viewpoint. Patent Document 1 discloses a technology for generating a virtual viewpoint image using images captured by a plurality of cameras arranged to surround a subject. Patent Document 1 also discloses that a user can specify the position of the virtual camera and the position of a gaze point that indicates where the virtual camera is gazing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-215828 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 has a problem in that it is difficult for the user to intuitively grasp the position of the point of interest in the virtual space.

[0005] In view of the above problems, the present disclosure aims to provide a technology that makes it easier for a user to grasp the position of the gaze point of a virtual viewpoint in a virtual space. [Means for solving the problem]

[0006] One aspect of the display control device of the present disclosure is a display control device including: an acquisition unit that acquires information indicating a position of a first virtual viewpoint corresponding to a first virtual viewpoint image generated based on a plurality of captured images obtained by capturing images with a plurality of imaging devices; and information indicating a position of a point of interest corresponding to the first virtual viewpoint; a first determination means for determining, based on information indicating the position of the first virtual viewpoint and information indicating the position of the point of gaze, a position of the second virtual viewpoint that is at a predetermined distance from the position of the first virtual viewpoint in a direction away from the point of gaze than the first virtual viewpoint, and such that a distance from the position of the second virtual viewpoint to the position of the point of gaze increases as the distance from the position of the first virtual viewpoint to the position of the point of gaze increases; and a second determination means for determining a size of an object indicating the point of gaze based on the distance from the position of the first virtual viewpoint to the position of the point of gaze. a display control means for controlling the display of a second virtual viewpoint image corresponding to the second virtual viewpoint, the second virtual viewpoint image including an object indicating the point of interest having the determined size and an object indicating the first virtual viewpoint; The second determination means determines a size of the object indicating the point of gaze when the distance from the position of the first virtual viewpoint to the position of the point of gaze is a second distance longer than the first distance, compared to a size of the object indicating the point of gaze when the distance from the position of the first virtual viewpoint to the position of the point of gaze is a first distance. . [Effects of the Invention]

[0007] According to the present disclosure, a user can easily grasp the position of the gaze point of a virtual viewpoint in a virtual space. [Brief explanation of the drawings]

[0008] [Figure 1] 1 shows an example of the configuration of an image processing system according to a first embodiment. [Figure 2] 2 shows an example of installation of the imaging device according to the first embodiment. [Figure 3] 1 shows a hardware configuration according to a first embodiment. [Figure 4] 1 shows a functional configuration of a display control device according to a first embodiment. [Figure 5] 10 shows an example in which a second virtual viewpoint image is displayed on the display unit according to the first embodiment. [Figure 6] 3 shows a flowchart illustrating the operation of the display control device according to the first embodiment. [Figure 7] 10 shows a graph illustrating the relationship between the distance from the first virtual viewpoint to the point of interest controlled by the display control device according to the first embodiment and the size coefficient of the point of interest object. [Figure 8] 10 shows the functional configuration of a display control device according to a second embodiment. [Figure 9] 10 shows a flowchart illustrating the operation of a display control device according to a second embodiment. [Figure 10] 10 shows the functional configuration of a display control device according to a third embodiment. [Figure 11]10 shows a functional configuration of a display control device according to a fourth embodiment. [Figure 12] 13 shows a display example of a guide object according to the fourth embodiment. [Figure 13] 10 shows the functional configuration of a display control device according to a fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the components described in the following embodiments are examples of embodiments, and the present disclosure is not limited to them.

[0010] A virtual viewpoint image is an image generated by a user and / or a dedicated operator freely manipulating the position and orientation of a virtual camera, and is also called a free viewpoint image or an arbitrary viewpoint image. While the present disclosure will mainly describe a case where a virtual viewpoint is specified by a user operation, the virtual viewpoint may also be specified automatically based on the results of image analysis, etc. Unless otherwise specified, the term "image" will be used in the following description to encompass both moving images and still images.

[0011] A virtual camera is a virtual camera that is different from the multiple imaging devices actually installed around the imaging area, and is a concept for conveniently explaining a virtual viewpoint related to the generation of a virtual viewpoint image. That is, a virtual viewpoint image can be considered to be an image captured from a virtual viewpoint set in a virtual space associated with the imaging area. The position and orientation of the viewpoint in the virtual image capture can be expressed as the position and orientation of the virtual camera. In other words, a virtual viewpoint image can be considered to be an image that simulates an image captured by a camera assuming that the camera exists at the position of the virtual viewpoint set in space. In this embodiment, the content of the transition of the virtual viewpoint over time is referred to as a virtual camera path. However, the concept of a virtual camera is not required to realize the configuration of this embodiment. That is, it is sufficient if at least information representing a specific position and information representing a direction in space are set, and a virtual viewpoint image is generated according to the set information.

[0012] The imaging device may have a physical camera. In addition to the physical camera, the imaging device may have various image processing functions. For example, the imaging device may have a processing unit that performs foreground / background separation processing. In addition, the imaging device may have a control unit that performs transmission control to transmit an image of a partial region of the captured image. In addition, the imaging device may have multiple physical cameras.

[0013] <Embodiment 1> FIG. 1 is a diagram showing an image processing system 100 according to this embodiment. The image processing system 100 includes a plurality of image capture devices 110, an image generation device 120, a display control device 130, and a display 140. The image capture devices 110, the image generation devices 120, and the display control device 130 are connected via communication cables such as LAN (Local Area Network) cables. In this embodiment, the communication cables are assumed to be LAN cables, but the communication cables are not limited to these. The image generation devices 120 and the display 140 are connected via a video signal transmission cable.

[0014] The imaging device 110 is, for example, a digital camera capable of capturing images (still images and video). FIG. 2 is a diagram showing an example of installation of the imaging devices 110. Each imaging device 110 is installed to surround a specific area, such as a stadium, and captures images (video) of a subject within the area. The captured images are transmitted from the imaging device 110 to the image generating device 120. Only an image corresponding to a partial area of ​​the captured image (for example, the area of ​​the subject) may be transmitted.

[0015] The image generating device 120 is, for example, a server device, and is equipped with a database function and an image processing function. The image generating device 120 stores, as background images, images captured in advance of a scene in which no subject is present, such as before the start of subject shooting in a stadium. The image generating device 120 also stores captured images obtained by the imaging device 110. When the image generating device 120 receives virtual viewpoint information and playback time information (e.g., a time code) through user operation of the display control device 130, it generates a virtual viewpoint image based on the stored captured images. Here, the virtual viewpoint information is information indicating the three-dimensional position of a virtual viewpoint (virtual viewpoint) in a virtual space, the line of sight direction from the virtual viewpoint, the angle of view, the position of the point of gaze, etc. The virtual viewpoint information includes at least the position relative to a predetermined origin position, such as the center of the stadium where the image was captured, i.e., position information for the front / back, left / right, and up / down directions relative to the origin position, and the orientation from that predetermined position, i.e., directional information on the angle around the front / back, left / right, and up / down axes. The virtual viewpoint information includes gaze point position information indicating which three-dimensional position is being watched from the virtual viewpoint position, and distance information from the gaze point position to the virtual viewpoint position.

[0016] The playback time information is time information at the time of shooting of a captured image, which includes the hour, minute, second, number of frames per second, etc., and by specifying the playback time, the scene at the recorded time is generated as a virtual viewpoint. Note that the number of frames per second is, for example, 60 frames. The multiple image capture devices 110 perform synchronized shooting based on a time server (not shown), and the time information at the time of shooting indicates the timing of shooting at the multiple image capture devices 110.

[0017] In addition, in a scene in which a subject is present, the image processing device 120 may perform image processing to separate the foreground of a specific object that is the subject as a specific object image. Note that the specific object may be not only a person but also an object such as a ball, for which an image pattern of the equipment is predetermined.

[0018] A virtual viewpoint image corresponding to the virtual viewpoint information is generated from a background image and a specific object image managed in a database. For example, model-based rendering (MBR) is used as a method for generating a virtual viewpoint image. MBR is a method for generating a virtual viewpoint image using a three-dimensional shape generated based on multiple captured images of a subject captured from multiple directions. Specifically, it is a technology for generating an image of the scene as seen from a virtual viewpoint using a three-dimensional shape (model) of a target scene obtained by a three-dimensional shape reconstruction method such as volume intersection or multi-view stereo (MVS). Note that a rendering method other than MBR may also be used to generate the virtual viewpoint image. The generated virtual viewpoint image is transmitted to the display 140 via a video signal transmission cable.

[0019] The display control device 130 is, for example, a PC (Personal Computer) or a tablet. The viewpoint controller 131 is a device for setting parameters such as the position and attitude of the virtual camera. For example, the viewpoint controller 131 is a mouse, keyboard, joystick, six-axis controller, touch panel, or game controller. The time controller 132 is a device for setting a playback time, and is, for example, an operation device equipped with a turntable. The viewpoint controller 131 and the time controller 132 are operated by the user. The display control device 130 receives information on user operations from the viewpoint controller 131 and the time controller 132. Then, the display control device 130 converts the information into virtual viewpoint information indicating the position and attitude of the virtual camera and playback time information according to the amount of operation, etc., and transmits the information to the image generation device 120.

[0020] Note that output from the display control device 130 using an operation device is not limited to continuous movement, and movement to a predetermined virtual viewpoint, such as a position in front of the subject in the virtual space, a position behind the subject, or a position looking down from above, is also possible. Also, by setting a playback time in advance, it is possible to instantly move the virtual viewpoint to that time. Furthermore, the display control device 130 displays objects in a three-dimensional space based on user operations on the screen via an application displayed on the display unit 305 by executing a control program described below.

[0021] 3 is a diagram showing the hardware configuration of the display control device 130. The display control device 130 has a CPU 301, a ROM 302, a RAM 303, an HDD 304, a display unit 305, an input unit 306, and a communication unit 307. The CPU 301 controls the entire display control device 130 using control programs and data stored in the ROM 302 and the RAM 303. Note that the display control device 130 may have one or more dedicated hardware components different from the CPU 301, and at least a portion of the processing by the CPU 301 may be executed by the dedicated hardware components. Examples of such dedicated hardware components include an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), and a DSP (digital signal processor).

[0022] ROM 302 stores programs that do not require modification. RAM 303 temporarily stores programs and data supplied from HDD 304, and data supplied from the outside via communication unit 307. RAM 303 is also used as a temporary storage area such as the main memory and work area of ​​CPU 301. HDD 304 stores various data, programs, etc.

[0023] The display unit 305 is configured with, for example, a liquid crystal display or LEDs, and displays various types of information. The input unit 306 can be connected to a keyboard, mouse, six-axis controller, etc., and accepts various operations by the user. The communication unit 307 performs communication processing with an external device via a network. An example of the network is Ethernet (registered trademark). As another example, the communication unit 307 may communicate with an external device wirelessly. The system bus 308 connects each unit of the display control device 130 and transmits information.

[0024] The functions and processes of the display control device 130, which will be described later, are realized by the CPU 301 reading and executing a program stored in the ROM 302 or the HDD 304. The hardware configuration of the image generation device 120 is the same as the hardware configuration of the display control device 130.

[0025] 4 is a diagram showing the functional configuration of the display control device 130. The controller operation acquisition unit 133 periodically acquires operation information for the virtual viewpoint acquired via the viewpoint controller 131 and the time controller 132.

[0026] The controller operation acquisition unit 133 converts operation information for the virtual viewpoint into virtual viewpoint movement amount information and playback time movement amount information, and outputs them to the first virtual viewpoint information determination unit 134. The virtual viewpoint movement amount is the amount of movement relative to the current virtual viewpoint position or the line of sight direction from the virtual viewpoint. The playback time movement amount is the amount of movement relative to the current playback time. These pieces of information are determined by a conversion coefficient set for the amount of user operation input to the viewpoint controller 131 or the time controller 132.

[0027] The first virtual viewpoint information determination unit 134 determines virtual viewpoint information corresponding to the position of the virtual viewpoint specified by a user operation and the line of sight direction from the virtual viewpoint based on the input virtual viewpoint movement amount information. The first virtual viewpoint information determination unit 134 then outputs the virtual viewpoint information to the virtual viewpoint object generation unit 136 as first virtual viewpoint information. The first virtual viewpoint information includes gaze point position information indicating the position of the gaze point corresponding to the first virtual viewpoint. Similarly, the first virtual viewpoint information determination unit 134 outputs the first virtual viewpoint information to the gaze point object generation unit 137, the second virtual viewpoint information determination unit 139, and the virtual viewpoint information transmission unit 142. The virtual space coordinate system is the same as the coordinate system of each imaging device 110. For example, the center of the coordinate system may be the center of a stadium or the like, or may be set appropriately by the user. The virtual viewpoint information is expressed as a three-dimensional position in this coordinate system. The gaze point in the present disclosure is a point located on the optical axis of the virtual camera and having three-dimensional coordinates. In addition, the distance between the gaze point and the virtual camera is determined based on a user operation.

[0028] The time code determination unit 135 determines playback time information specified by a user operation based on the input playback time movement amount, and outputs the playback time information to the virtual viewpoint information transmission unit 142. Note that the playback time is based on the date and time when shooting started by each image capture device 110. The playback time may be the shooting time itself, or may be expressed as the time elapsed since the shooting start time, with the shooting start time set to 0.

[0029] The virtual viewpoint object generation unit 136 acquires first virtual viewpoint information from the first virtual viewpoint information determination unit 134. The virtual viewpoint object generation unit 136 also acquires virtual viewpoint object information from the object information storage unit 138, which will be described later. Based on the first virtual viewpoint information and the virtual viewpoint object information, the virtual viewpoint object generation unit 136 generates a virtual viewpoint object that enables the user to recognize the position of the virtual viewpoint and the line of sight from the virtual viewpoint, which are determined by a user operation in the virtual space. The virtual viewpoint object includes data such as a shape for expressing the virtual viewpoint object, and information about its position in space. The virtual viewpoint object generation unit 136 also outputs the virtual viewpoint object to the second virtual viewpoint information determination unit 139.

[0030] The point of interest object generation unit 137 acquires first virtual viewpoint information from the first virtual viewpoint information determination unit 134. Furthermore, the point of interest object generation unit 137 acquires point of interest object information from the object information holding unit 138. Based on the point of interest position information included in the first virtual viewpoint information and the point of interest object information, the point of interest object generation unit 137 generates a point of interest object that enables the user to recognize the position of the point of interest of the virtual viewpoint determined by a user operation in virtual space. The point of interest object includes data such as a shape for expressing the point of interest object, and information on its position in space. Furthermore, the point of interest object generation unit 137 outputs the point of interest object to the second virtual viewpoint information determination unit 139.

[0031] The object information storage unit 138 stores object information such as virtual viewpoint object information and gaze point object information in advance, and outputs the corresponding object information. That is, the object information storage unit 138 outputs the virtual viewpoint object information to the virtual viewpoint object generation unit 136. The object information storage unit 138 also outputs gaze point object information to the gaze point object generation unit 137. Here, the object information is a three-dimensional model, and is data having three-dimensional coordinates created using dedicated software or the like, and is data that indicates a three-dimensional shape by connecting multiple three-dimensional coordinates (vertices). Furthermore, by storing color information and texture images associated with surfaces formed by connecting multiple vertices, the object becomes an object recognizable to the user. For example, the object information may be mesh data composed of multiple polygons.

[0032] The second virtual viewpoint information determination unit 139 acquires a virtual viewpoint object from the virtual viewpoint object generation unit 136 and acquires a gaze point object from the gaze point object generation unit 137. The second virtual viewpoint information determination unit 139 also acquires first virtual viewpoint information from the first virtual viewpoint information determination unit 134. The second virtual viewpoint information determination unit 139 determines second virtual viewpoint information that can display the virtual viewpoint object and the gaze point object based on the first virtual viewpoint information, and outputs the second virtual viewpoint information to the second virtual viewpoint image generation unit 140. Specifically, the second virtual viewpoint information is information that indicates the position of a virtual viewpoint other than the virtual viewpoint operated by the user and the line of sight from the virtual viewpoint. For example, the position of the virtual viewpoint indicated by the second virtual viewpoint information may be a position behind the first virtual viewpoint (the virtual viewpoint operated by the user) indicated by the first virtual viewpoint information. The line of sight from the second virtual viewpoint indicated by the second virtual viewpoint information may be a direction in which the gaze point object can be displayed. Furthermore, the position of the second virtual viewpoint and the line of sight direction from the virtual viewpoint may be a position and orientation that allows a virtual viewpoint object to be displayed. "Behind" does not mean that the second virtual viewpoint is connected to the first virtual viewpoint on a straight line, but rather that the second virtual viewpoint is located on the opposite side of the point of gaze relative to the first virtual viewpoint. Furthermore, the field of view of the first virtual viewpoint specified by the first virtual viewpoint information does not include the position of the second virtual viewpoint. On the other hand, the field of view of the second virtual viewpoint includes the position of the first virtual viewpoint and the position of the point of gaze.

[0033] Note that "displayable" means that the object can be displayed in a virtual viewpoint image generated on the display device based on the second virtual viewpoint information. Note that the virtual viewpoint object and the gaze point object do not have to be displayed simultaneously in this virtual viewpoint image. For example, the gaze point object may be displayed, but the virtual viewpoint object may not be displayed. Alternatively, for example, one of the objects may be hidden by a display instruction from the user. Furthermore, these objects may be displayed semi-transparently. In this case, these objects are prevented from occluding each other, making it easier for the user to recognize the objects. Furthermore, if a subject is present in the virtual viewpoint image, the subject is prevented from being occluded, making it easier for the user to recognize the subject.

[0034] The second virtual viewpoint image generation unit 140 acquires the second virtual viewpoint information from the second virtual viewpoint information determination unit 139, and generates a second virtual viewpoint image including a virtual viewpoint object and a focus point object based on the second virtual viewpoint information. This second virtual viewpoint image may be generated based on the imaging device 110, or may be generated based on a background model such as a stadium that is stored in advance in the image generation device 120. This second virtual viewpoint image may be generated by the display control unit 141. The second virtual viewpoint image generation unit 140 transmits the generated second virtual viewpoint image to the display control unit 141.

[0035] The display control unit 141 controls the display of the second virtual viewpoint image acquired from the second virtual viewpoint image generation unit 140. That is, the display unit 305 of the display control device 130 displays the position of the virtual camera (first virtual viewpoint) and its gaze point based on the user's virtual viewpoint operation. This display is performed in the second virtual viewpoint image generated based on the second virtual viewpoint information.

[0036] The virtual viewpoint information transmitting unit 142 transmits the first virtual viewpoint information input from the first virtual viewpoint information determining unit 134 and the playback time information input from the time code determining unit 135 to the image generating device 120. The virtual viewpoint image generated by the image generating device 120 based on the virtual viewpoint information and playback time information is output to and displayed on the display 140. The virtual viewpoint image generated by the image generating device 120 is a first virtual viewpoint image seen from the first virtual viewpoint, and is different from the second virtual viewpoint image seen from the second virtual viewpoint.

[0037] FIG. 5 shows an example of a second virtual viewpoint image displayed on the display unit 305. A virtual viewpoint object 501 and a gaze point object 502 are displayed on the screen of the display unit 305 of the display control device 130. In this embodiment, the larger (longer) the distance between the position of the virtual viewpoint and the position of the gaze point in the virtual space, the larger the size of the gaze point object. For example, FIG. 5(a) shows an example in which the distance between the position of the virtual viewpoint and the position of the gaze point is large but the size of the gaze point object is not changed. The gaze point object that is far away from the second virtual viewpoint appears small, reducing visibility. In FIG. 5(b), the size of the gaze point object is increased by processing described below, and the gaze point object appears large in the second virtual viewpoint image, improving visibility.

[0038] 6 is a flowchart showing the operation of the display control device 130 according to this embodiment. The CPU 301 reads out and executes a program stored in the ROM 302 or HDD 304, thereby performing the following processing.

[0039] In step S601, the controller operation acquisition unit 133 acquires movement amount information of a lever of a six-axis controller such as a joystick from the viewpoint controller 131. The controller operation acquisition unit 133 also acquires movement amount information of, for example, a wheel turned by the user from the time controller 132. Next, the controller operation acquisition unit 133 converts the movement amount information acquired from each of the viewpoint controller 131 and the time controller 132 into virtual viewpoint movement amount information and playback time movement amount information, and outputs them to the first virtual viewpoint information determination unit 134.

[0040] In step S602, first virtual viewpoint information determination unit 134 determines first virtual viewpoint information based on the input virtual viewpoint movement amount information. Then, first virtual viewpoint information determination unit 134 outputs the first virtual viewpoint information to virtual viewpoint object generation unit 136, gaze point object generation unit 137, and projected gaze point object generation unit 138. The first virtual viewpoint is a virtual viewpoint that is the object of operation by the user, and is a virtual viewpoint that corresponds to the virtual viewpoint image displayed on display 140.

[0041] In step S603, the virtual viewpoint object generation unit 136 generates a virtual viewpoint object. Specifically, the virtual viewpoint object generation unit 136 acquires virtual viewpoint information from the first virtual viewpoint information determination unit 134. The virtual viewpoint object generation unit 136 also acquires virtual viewpoint object information from the object information storage unit 139. Based on the virtual viewpoint information and the virtual viewpoint object information, the virtual viewpoint object generation unit 136 generates a virtual viewpoint object that enables the user to recognize the position of the first virtual viewpoint in the virtual space and the line of sight from the first virtual viewpoint. For example, the virtual viewpoint object may have the shape of a camera or another shape.

[0042] In step S604, the focus object generation unit 137 determines the size of the focus object. Specifically, the focus object generation unit 137 acquires virtual viewpoint information from the first virtual viewpoint information determination unit 134. The unit calculates the distance between the first virtual viewpoint and the focus position based on the positions of the first virtual viewpoint and the focus position included in the virtual viewpoint information. Next, the unit determines the size of the focus object based on the graph shown in FIG. 7, which represents the relationship between the distance between the first virtual viewpoint and the focus position and the size of the focus object. If the distance between the first virtual viewpoint and the focus position is less than a threshold D_th, the size coefficient of the focus object is set to 1.0. The size coefficient indicates how much the focus object is enlarged relative to its initial size. In other words, the acquired focus object is set to the same size. Here, the threshold D_th may be set separately by the user or may be determined based on the subject being photographed. For example, when photographing a baseball game and generating a virtual viewpoint image, the threshold D_th is set to 40 m, for example, based on the distance between home base and second base. Next, when the distance between the position of the first virtual viewpoint and the position of the point of interest is equal to or greater than a threshold D_th, a size coefficient for the point of interest object is determined linearly, and the size is determined by adding a numerical value obtained by multiplying the size coefficient by a predetermined value to the initial size of the point of interest object. However, the determination of the size coefficient is not limited to this, and a nonlinear method may be used so that the size increases as the distance increases. Alternatively, the size may be determined by multiplying the initial size of the point of interest object by a size coefficient. Note that the predetermined value by which the size coefficient is multiplied is separately set by the user. As a result, when the first virtual viewpoint is farther from the point of interest than the threshold D_th, the point of interest object is enlarged linearly (proportionally). Note that the design of the function for determining the size coefficient is not limited to this example, and a nonlinear function may also be used, as long as there is a positive correlation between the distance and the size coefficient. Furthermore, by setting the size coefficient for distances equal to or less than the threshold D_th to less than 1.0, the point of interest may be scaled down so that it does not appear too large when it approaches closer than the threshold D_th. However, since it is desirable for the size of the point of interest to change continuously from the viewpoint of continuity of operation, it is desirable for the function for determining the size coefficient to also be continuous.

[0043] In step S605, the point of interest object generation unit 137 generates a point of interest object. Specifically, the point of interest object generation unit 137 acquires point of interest object information from the object information storage unit 130. Based on the point of interest position information included in the virtual viewpoint information and the point of interest object information, the point of interest object generation unit 137 generates a point of interest object that enables the user to recognize the position of the point of interest of the first virtual viewpoint in the virtual space. The size of the generated point of interest object is the size determined in step S604. Note that the point of interest object may have a spherical shape or a rectangular prism shape.

[0044] In step S606, the second virtual viewpoint information determination unit 142 determines second virtual viewpoint information. Specifically, the second virtual viewpoint information determination unit 142 acquires first virtual viewpoint information from the first virtual viewpoint information determination unit 134. The second virtual viewpoint information determination unit 142 also acquires a virtual viewpoint object and a focus object. The second virtual viewpoint information determination unit 142 determines second virtual viewpoint information based on the acquired information. The second virtual viewpoint information indicates information about a second virtual viewpoint from which the virtual viewpoint object and the focus object can be virtually photographed. More specifically, the second virtual viewpoint information is determined so that the position of the second virtual viewpoint is a position a predetermined distance behind the position of the first virtual viewpoint, and the line of sight from the second virtual viewpoint is a direction from which the focus object can be virtually photographed. The second virtual viewpoint information determination unit 142 outputs the second virtual viewpoint information to the display control unit 143.

[0045] In step S607, the second virtual viewpoint image generation unit 140 generates a second virtual viewpoint image. Specifically, the second virtual viewpoint image generation unit 140 acquires second virtual viewpoint information from the second virtual viewpoint information determination unit 139, and generates a second virtual viewpoint image including a virtual viewpoint object and a focus point object based on the second virtual viewpoint information. The second virtual viewpoint image generation unit 140 outputs the generated second virtual viewpoint image to the display control unit 141.

[0046] In step S608, the display control unit 143 acquires the second virtual viewpoint image from the second virtual viewpoint image generation unit 140. Next, the display control unit 143 displays the second virtual viewpoint image including the virtual viewpoint object and the gaze point object on the display unit 305 of the display control device 130. This displays the positions in the virtual space of the first virtual viewpoint, its gaze point, and the projected gaze point based on the virtual viewpoint operation by the user. This allows the user to easily grasp the position of the gaze point in the virtual space.

[0047] As described above, in this embodiment, when the distance between the position of the first virtual viewpoint and the position of the gaze point is greater than a threshold, the size of the gaze point object is increased, thereby improving the visibility of the gaze point in the second virtual viewpoint image, which the user uses as a reference when operating the first virtual viewpoint.

[0048] In this embodiment, the second virtual viewpoint is set at a position a predetermined distance behind the virtual viewpoint, but this is not limited to this. For example, the second virtual viewpoint may be set near the gaze point, in which case the second virtual viewpoint is determined on the opposite side of the first virtual viewpoint with the gaze point as an axis so that the virtual viewpoint object is included in the angle of view. In this case, the size of the virtual viewpoint object is changed depending on the distance between the virtual viewpoint object and the gaze point object. Furthermore, the second virtual viewpoint may be determined at a position away from both the gaze point and the virtual viewpoint, in which case the sizes of both the gaze point object and the virtual viewpoint object are changed.

[0049] In addition, although the visibility of the gaze point or virtual viewpoint is improved by changing the size of each object in this embodiment, other parameters related to object display may also be changed. For example, the color of an object included in the parameters may be changed overall or partially, such as the outline. In addition, the size or color of an object may be periodically changed to improve visibility.

[0050] Additionally, the definition of the object information is not limited to a three-dimensional model, but may be held, input, and resized as an object that is always drawn facing the second virtual viewpoint, such as a two-dimensional image.

[0051] Furthermore, in the present embodiment, the display device 140 and the display unit 305 are described as separate blocks, but this is not limiting, and for example, the second virtual viewpoint image may be displayed on the display device 140. It is also possible to configure a multi-window configuration, configuring a window for displaying the first virtual viewpoint image and a window for displaying the second virtual viewpoint image, and displaying them simultaneously.

[0052] Although the virtual viewpoint information has been described using a three-dimensional position and a line-of-sight direction from the virtual viewpoint, it is not limited to this and may be information representing an attitude including rotation and an angle of view.

[0053] <Modification of the First Embodiment> In the first embodiment, the size of the fixation point object is changed according to the distance between the first virtual viewpoint and the fixation point, but it may be changed according to their respective three-dimensional positions. Hereinafter, a description of the same configuration as in the first embodiment will be omitted.

[0054] In step S505, each of the focus point object generation unit 137 and the virtual viewpoint object generation unit 136 determines the size of the object so that it will have a predetermined size when the three-dimensional position of each object is included in a predetermined subspace. For example, when a baseball is being photographed, if the three-dimensional position of the first virtual viewpoint belongs to the inside of a spherical subspace with a radius of 20 m from home base, the size coefficient for the process of determining the size of the focus point object is set to 1.0. If the three-dimensional position of the first virtual viewpoint belongs to outside the subspace and within a radius of 40 m from home base, the size coefficient is set to 2.0. If the three-dimensional position of the first virtual viewpoint belongs to outside the subspace and within a radius of 40 m from home base, the size coefficient is set to 3.0. As described above, by setting at least one predetermined subspace in the virtual space and setting parameters such as the size coefficient of the focus point object in each subspace, the size of the focus point object can be determined according to the position of the first virtual viewpoint.

[0055] As a result of the above processing, it is possible to improve the visibility of the gaze point, and at the same time, it becomes possible to intuitively grasp to which space the virtual viewpoint belongs.

[0056] <Embodiment 2> In the first embodiment, the size of the fixation point object is determined according to the distance between the first virtual viewpoint and the fixation point, but in the second embodiment, the size of the fixation point object is determined according to the distance between the second virtual viewpoint and the fixation point. Hereinafter, a description of the same configuration as in the first embodiment will be omitted.

[0057] FIG. 8 is a diagram showing the functional configuration of a display control device 130 according to the second embodiment.

[0058] The second virtual viewpoint information determination unit 801 acquires first virtual viewpoint information from the first virtual viewpoint information determination unit 134. Next, based on the first virtual viewpoint information, it determines second virtual viewpoint information indicating a second virtual viewpoint that includes the first virtual viewpoint and the point of interest within its angle of view. The position of the second virtual viewpoint and the line of sight direction from the second virtual viewpoint are the same as in the first embodiment, and therefore will not be described here. The second virtual viewpoint information determination unit 801 outputs the determined second virtual viewpoint information to the point of interest object generation unit 802.

[0059] The focus object generation unit 802 acquires focus object information from the object information storage unit 138. It also acquires first virtual viewpoint information from the first virtual viewpoint information determination unit 134. It also acquires second virtual viewpoint information from the second virtual viewpoint information determination unit 801. The focus object generation unit 802 determines the size of the focus object based on the focus position information and the second virtual viewpoint information included in the acquired first virtual viewpoint information. Specifically, it determines the size of the focus object based on the distance between the second virtual viewpoint and the focus. It then generates a focus object of the determined size and outputs it to the second virtual viewpoint image generation unit 804.

[0060] The virtual viewpoint object generation unit 803 acquires first virtual viewpoint information from the first virtual viewpoint information determination unit 134. Furthermore, the virtual viewpoint object generation unit 136 acquires virtual viewpoint object information from the object information storage unit 138, which will be described later. Based on the first virtual viewpoint information and the virtual viewpoint object information, the virtual viewpoint object generation unit 136 generates a virtual viewpoint object that enables the user to recognize the position of the virtual viewpoint in the virtual space determined by a user operation and the line of sight from the virtual viewpoint. The virtual viewpoint object generation unit 803 outputs the generated virtual viewpoint object to the second virtual viewpoint image generation unit 804.

[0061] The second virtual viewpoint image generation unit 804 acquires second virtual viewpoint information from the second virtual viewpoint information determination unit 801. The second virtual viewpoint image generation unit 804 also acquires a focus object from a focus object generation unit 802. The second virtual viewpoint image generation unit 804 also acquires a virtual viewpoint object from a virtual viewpoint object generation unit 803. The second virtual viewpoint image generation unit 804 generates a second virtual viewpoint image including the focus object and the virtual viewpoint object based on the second virtual viewpoint information. The second virtual viewpoint image generation unit 804 transmits the generated second virtual viewpoint image to the display control unit 141.

[0062] 9 is a flowchart showing the operation of the display control means 130 according to the second embodiment. The following processing is performed by the CPU 301 reading and executing a program stored in the ROM 302 or the HDD 304. Note that a description of steps that perform the same processing as in the flowchart shown in FIG. 6 will be omitted.

[0063] In step S901, the second virtual viewpoint information determination unit 801 acquires first virtual viewpoint information from the first virtual viewpoint information determination unit 134. Next, based on the first virtual viewpoint information, second virtual viewpoint information indicating a second virtual viewpoint that includes the first virtual viewpoint and the point of interest within its angle of view is determined. The position of the second virtual viewpoint and the line of sight direction from the second virtual viewpoint are the same as in the first embodiment, and therefore will not be described here. The second virtual viewpoint information determination unit 801 outputs the determined second virtual viewpoint information to the point of interest object generation unit 802.

[0064] In step S902, the gaze point object generation unit 802 generates a gaze point object. Specifically, the gaze point object generation unit 802 determines the size of the gaze point object based on the gaze point position information and the second virtual viewpoint information included in the acquired first virtual viewpoint information. Specifically, the size of the gaze point object is determined based on the distance between the second virtual viewpoint and the gaze point.

[0065] Through the above processing, the size of the fixation point object is determined based on the distance between the second virtual viewpoint and the fixation point, and the visibility of the fixation point in the second virtual viewpoint image can be improved.

[0066] <Embodiment 3> In the first embodiment, the size of the spherical gaze point object is determined according to the distance between the first virtual viewpoint and the gaze point, but in the third embodiment, an example will be described in which the type of gaze point object is replaced with a life-scale object. In this embodiment, when the distance between the first virtual viewpoint and the gaze point is closer than a threshold, it becomes possible to easily determine an appropriate distance in generating a virtual viewpoint image. Below, a description of the same configuration as in the first embodiment will be omitted.

[0067] The object shape storage unit 138 stores, for example, a sphere as described in the first embodiment as an example of a type of fixation point object, as well as a pre-created 3D model of a baseball player. In this case, the shape of the same type of 3D model may be deformed over time, and multiple 3D models of the same type may be stored. Note that the types of fixation point objects are not limited to those described above. For example, the color of the fixation point object may be changed depending on the distance between the first virtual viewpoint and the fixation point. Alternatively, when the distance between the first virtual viewpoint and the fixation point becomes equal to or less than a threshold, the outline of the fixation point object on the virtual viewpoint image may be emphasized.

[0068] FIG. 10 is a diagram showing the functional configuration of a display control device 130 according to this embodiment.

[0069] A first virtual viewpoint information determination unit 1001 determines virtual viewpoint information corresponding to the position of the virtual viewpoint specified by a user operation and the line of sight direction from the virtual viewpoint, based on the input virtual viewpoint movement amount information. The generated virtual viewpoint information is the same as in embodiment 1. Then, the first virtual viewpoint information determination unit 134 outputs the generated virtual viewpoint information to the object information storage unit 1002 as first virtual viewpoint information.

[0070] The object information holding unit 1002 acquires first virtual viewpoint information from the first virtual viewpoint information determination unit 1001. Based on the acquired first virtual viewpoint information, the object information holding unit 1002 determines a focus object to be displayed in the second virtual viewpoint image from among a plurality of focus objects stored in advance. Specifically, if the distance between the first virtual viewpoint and the focus is smaller than a threshold D_th2, the object information holding unit 1002 determines a stored three-dimensional model representing the shape of a baseball player as the focus object. If the distance between the first virtual viewpoint and the focus is equal to or greater than the threshold D_th2, the object information holding unit 1002 determines a three-dimensional model of a sphere as the focus object, as in the first embodiment. The object information holding unit 1002 outputs the determined object to the focus object generation unit 137.

[0071] Note that by generating a three-dimensional model of the baseball player in advance based on multiple captured images, when the distance between the first virtual viewpoint and the point of interest is less than D_th2, a three-dimensional model that is close to the size of a real baseball player can be set as the point of interest object. Alternatively, the three-dimensional model generated in advance may be set as a rectangular parallelepiped with a height of 1 meter. Note that the three-dimensional model generated in advance may be different from the three-dimensional model determined as the point of interest object when the distance between the first virtual viewpoint and the point of interest is equal to or greater than D_th2.

[0072] The above process results in a three-dimensional model in which the gaze point object displayed when the distance between the first virtual viewpoint and the gaze point is less than D_th2 and the gaze point object displayed when the distance between the first virtual viewpoint and the gaze point is equal to or greater than D_th2 are different. As a result, it becomes easier to intuitively grasp the distance between the first virtual viewpoint and the gaze point.

[0073] In this embodiment, an example has been described in which the type of the point-of-gain object is changed depending on the distance between the first virtual viewpoint and the point of gaze, but this is not limited to the above. For example, when the distance between the first virtual viewpoint and the point of gaze is equal to or greater than a threshold, text indicating the approximate location of the point of gaze may be displayed in the second virtual viewpoint image. Specifically, suppose that the subject of shooting is baseball, the threshold is 20 m, the distance between the first virtual viewpoint and the point of gaze is 25 m, and the point of gaze is located within 3 m of the three-dimensional position of home base. In this case, text such as "Around Home Base" may be displayed around the point-of-gain object on the second virtual viewpoint image. Note that the condition for displaying the text, such as "within 3 m of the three-dimensional position of home base," is set in advance for each text. In this way, the user can easily grasp the approximate location of the point of gaze.

[0074] <Embodiment 4> In the first embodiment, a method for improving the visibility of the point of gaze on the second virtual viewpoint image by increasing the size of the point of gaze object when the distance between the first virtual viewpoint and the point of gaze is large is described. In the fourth embodiment, a method is described in which the position of the point of gaze can be grasped more accurately by generating and displaying a guide object for supplementarily presenting background objects and their relative positional relationships. Hereinafter, a description of the same configuration as in the first embodiment will be omitted.

[0075] FIG. 11 shows the functional configuration of a display control device 130 according to this embodiment.

[0076] The first virtual viewpoint information determination unit 1101 outputs the generated first virtual viewpoint information to a guide object generation unit 1105 in addition to the function of the first virtual viewpoint information determination unit 134 described with reference to FIG.

[0077] The virtual viewpoint object generating unit 1102 outputs the generated virtual viewpoint object to the guide object generating unit 1105 in addition to the functions of the virtual viewpoint object generating unit 136 described with reference to FIG.

[0078] The gaze point object generating unit 1103 outputs the generated gaze point object to the guide object generating unit 1105 in addition to the function of the gaze point object generating unit 137 described with reference to FIG.

[0079] 4, the object information holding unit 1104 holds a background object, for example, a three-dimensional model of a baseball stadium. A plurality of types of background objects are held, and a background object selected by a user operation is output to the guide object generation unit 1105. Note that the present invention is not limited to this, and a background object to be output may be determined from a plurality of types of background objects based on a captured image.

[0080] The guide object generation unit 1105 receives the gaze point information, the virtual viewpoint information, and the background object, and generates a guide object. The generated guide object is output to the second virtual viewpoint image generation unit 1106. The guide object is, for example, a three-dimensional model of a line or a surface, as will be described later.

[0081] The second virtual viewpoint image generating unit 1106 generates a second virtual viewpoint image including the acquired guide object. The second virtual viewpoint image generating unit 1106 outputs the generated second virtual viewpoint image to the display control unit 141.

[0082] FIG. 12 shows a display example of a guide object according to this embodiment.

[0083] In FIG. 12(a), no guide object is displayed, and a three-dimensional model of a baseball stadium is displayed as a background object 1203, and a virtual viewpoint object 1201 and a gaze point object 1202 are displayed as in the first embodiment.

[0084] 12(b), a linear object 1206 directed from the first virtual viewpoint toward the point of gaze, and a plane 1205 slicing the three-dimensional virtual space in the vertical direction from the position and orientation of the first virtual viewpoint to include the linear object 1206 are displayed as guide objects. Here, plane 1205 is displayed with a limited angular range in the vertical direction according to the angle of view of the first virtual viewpoint. Furthermore, plane 1205 extends to the point where it intersects with background object 1203, and the dashed line includes the point of contact with background object 1203. These guide objects make it easier to intuitively grasp the orientation of the first virtual viewpoint, its positional relationship with background object 1203, and the line of sight.

[0085] 12(c), a linear object 1204 and a linear object 1207 are displayed as guide objects, extending vertically downward from the point of gaze and the first virtual viewpoint toward a background object 1203. Also displayed is a plane 1208 that slices the three-dimensional virtual space horizontally from the first virtual viewpoint toward the background object 1203. Like plane 1205, plane 1208 is displayed with a limited angular range depending on the angle of view of the first virtual viewpoint. Furthermore, because plane 1208 extends horizontally toward the background object 1203, it is in contact with the second-floor portion of the stadium seating of background object 1203. These guide objects make it easier to intuitively grasp the height of the point of gaze and the first virtual viewpoint and their positions within the infield.

[0086] As described above, by generating and displaying a guide object to assist in presenting the relative positions of each object, it becomes possible to more intuitively grasp the angle of view from the first virtual viewpoint, the point of gaze, and the position of the first virtual viewpoint.

[0087] Depending on the texture of the background, the color of each object may be set to a color with good visibility, for example, a complementary color to the average color of the background around the object.

[0088] <Embodiment 5> (Adjusting the size of the gaze point object on the virtual viewpoint image) In the first embodiment, the size of the spherical point-of-regard object is determined depending on the distance between the first virtual viewpoint and the point-of-regard. In the fifth embodiment, an example will be described in which the sizes of the virtual viewpoint object and the point-of-regard object on the second virtual viewpoint image are controlled to be the same regardless of the distance between the second virtual viewpoint and the point-of-regard.

[0089] FIG. 13 shows the functional configuration of a display control device 130 according to this embodiment.

[0090] The virtual viewpoint object generator 1301 outputs the generated virtual viewpoint object to the display form controller 1304 in addition to the functions of the virtual viewpoint object generator 136 described with reference to FIG.

[0091] 4, the gaze point object generation unit 1302 outputs the generated gaze point object to the display form control unit 1304. Note that, while the gaze point object generation unit 137 determined the size of the gaze point object in the first embodiment, the gaze point object generation unit 137 does not determine the size of the gaze point object in this embodiment.

[0092] The second virtual viewpoint information determination unit 1303 outputs the generated second virtual viewpoint information to the display form control unit 1304 in addition to the functions of the second virtual viewpoint information determination unit 139 described with reference to FIG.

[0093] The display form control unit 1304 acquires a virtual viewpoint object from the virtual viewpoint object generation unit 1301. The display form control unit 1304 acquires a focus point object from the focus point object generation unit 1302. The display form control unit 1304 acquires second virtual viewpoint information from the second virtual viewpoint information determination unit 1303. Based on the acquired information, the display form control unit 1305 controls the sizes of the virtual viewpoint object and the focus point object on the second virtual viewpoint image so that they are the same regardless of the distance between the second virtual viewpoint and the focus point. The sizes can be set by the user.

[0094] By the above processing, the sizes of the virtual viewpoint object and the focus object on the second virtual viewpoint image are controlled to be the same, so that the visibility of the virtual viewpoint object and the focus object can be improved.

[0095] Although the example has been described in which the sizes of the virtual viewpoint object and the focus object on the second virtual viewpoint image are controlled to be the same regardless of the distance between the second virtual viewpoint and the focus point, the present invention is not limited to this. For example, the sizes of the virtual viewpoint object and the focus point object on the second virtual viewpoint image may be controlled to be the same regardless of the distance between the first virtual viewpoint and the focus point. Alternatively, the sizes of either the virtual viewpoint object or the focus point object on the second virtual viewpoint image may be controlled to be the same.

[0096] Although the above description is an example in which the sizes of the virtual viewpoint object and the gaze point object on the second virtual viewpoint image are controlled to be the same regardless of the distance between the second virtual viewpoint and the gaze point, the present invention is not limited to this. For example, the sizes of the virtual viewpoint object and the gaze point object on the second virtual viewpoint image may be controlled to be within a predetermined range. Specifically, a size range of the virtual viewpoint object and the gaze point object on the second virtual viewpoint image is specified in advance. For example, a square frame is set to surround the virtual viewpoint object and the gaze point object on the second virtual viewpoint image, and a range of the size of one side of the frame is set. Note that the size range is set in advance to a range of sizes that is easy for the user to view. In this example, the size of one side is set to be 5 to 10 pixels. A first threshold and a second threshold greater than the first threshold are set in advance for the distance between the second virtual viewpoint and the gaze point, and when the distance is less than the first threshold, the size of one side of the frame is controlled to be 5 pixels. When the distance is equal to or greater than the first threshold but less than the second threshold, the size of one side of the frame is controlled to be within a range of 5 to 10 pixels. In this case, the distance and the size of one side are correlated, and may be proportional, for example. If the distance is equal to or greater than a second threshold, the size of one side of the frame is controlled to 10 pixels. In other words, the above process is an example of controlling the sizes of the virtual viewpoint object and the gaze point object on the second virtual viewpoint image to be within a predetermined range. Therefore, the difference between the size of the object representing the gaze point on the second virtual viewpoint image when the distance between the second virtual viewpoint and the gaze point is a first distance and the size of the object representing the gaze point on the second virtual viewpoint image when the distance is a second distance different from the first distance falls within a predetermined range.

[0097] The above process makes it possible to display the virtual viewpoint object and the gaze point object in sizes that are easy for the user to view, regardless of the distance between the second virtual viewpoint and the gaze point. In addition, since the sizes of the virtual viewpoint object and the gaze point object on the second virtual viewpoint image on the two-dimensional image are changed according to the distance between the second virtual viewpoint and the gaze point, the user can easily grasp the sense of distance between the second virtual viewpoint and the gaze point.

[0098] The present disclosure has been described above based on multiple embodiments, but the present disclosure is not limited to the above embodiments, and various modifications are possible based on the spirit of the present disclosure, and these modifications are not excluded from the scope of the present disclosure.

[0099] Note that a computer program that realizes part or all of the control in this embodiment and the functions of the above-described embodiment may be supplied to an image processing system or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the image processing system or the like may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present disclosure.

[0100] The disclosure of this embodiment includes the following configurations, methods, and programs.

[0101] (Configuration 1) An acquisition means for acquiring information indicating the position of a first virtual viewpoint corresponding to a first virtual viewpoint image generated based on a plurality of captured images obtained by capturing images with a plurality of imaging devices, and information indicating the position of a point of interest corresponding to the first virtual viewpoint; a determining means for determining a size of an object indicating the point of interest based on a distance from the position of the first virtual viewpoint to the position of the point of interest; a display control means for controlling the display of a second virtual viewpoint image corresponding to a second virtual viewpoint determined based on the first virtual viewpoint, the second virtual viewpoint image including an object indicating the point of interest having a determined size and an object indicating the first virtual viewpoint; The determining means determines a size of the object indicating the point of gaze when the distance from the position of the first virtual viewpoint to the position of the point of gaze is a second distance longer than the first distance, compared to a size of the object indicating the point of gaze when the distance from the position of the first virtual viewpoint to the position of the point of gaze is a first distance. A display control device comprising:

[0102] (Configuration 2) The display control device according to configuration 1, wherein the size of the object indicating the point of gaze is determined to be proportional to the distance from the position of the first virtual viewpoint to the position of the point of gaze.

[0103] (Configuration 3) A display control device according to configuration 1, characterized in that the size of the object indicating the gaze point is determined to be a predetermined size when the distance from the position of the first virtual viewpoint to the position of the gaze point is less than a threshold, and when the distance from the position of the first virtual viewpoint to the position of the gaze point is equal to or greater than a threshold, the size of the object is determined to be larger the greater the distance.

[0104] (Configuration 4) 4. The display control device according to any one of configurations 1 to 3, wherein the object representing the first virtual viewpoint and the object representing the point of gaze are three-dimensional models.

[0105] (Configuration 5) 5. The display control device according to any one of configurations 1 to 4, wherein the position of the second virtual viewpoint is a position at a predetermined distance from the position of the first virtual viewpoint.

[0106] (Configuration 6) an acquisition means for acquiring information indicating a position of a point of interest corresponding to a first virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by capturing images with a plurality of imaging devices; a determining means for determining a size of the object indicating the point of gaze on the second virtual viewpoint image corresponding to the second virtual viewpoint when a distance from the position of the second virtual viewpoint different from the first virtual viewpoint to a position of the point of gaze is a second distance longer than the first distance; a display control means for controlling the display of a second virtual viewpoint image corresponding to a second virtual viewpoint different from the first virtual viewpoint, the second virtual viewpoint image including an object indicating the point of interest of a determined size; A display control device comprising:

[0107] (Configuration 7) The display control device described in configuration 6, characterized in that the determination means determines the size of the object indicating the point of gaze so that the size of the object indicating the point of gaze on the second virtual viewpoint image corresponding to the second virtual viewpoint when the distance from the position of the second virtual viewpoint different from the first virtual viewpoint to the position of the point of gaze is a first distance is the same as the size of the object indicating the point of gaze on the second virtual viewpoint image when the distance from the position of the second virtual viewpoint to the position of the point of gaze is a second distance longer than the first distance.

[0108] (Method 1) an acquisition step of acquiring information indicating a position of a first virtual viewpoint corresponding to a first virtual viewpoint image generated based on a plurality of captured images obtained by capturing images with a plurality of imaging devices, and information indicating a position of a point of interest corresponding to the first virtual viewpoint; a determining step of determining a size of an object indicating the point of gaze to be large when a distance from the position of the first virtual viewpoint to the position of the point of gaze is large; a display control step of controlling the display of a second virtual viewpoint image corresponding to a second virtual viewpoint different from the first virtual viewpoint, the second virtual viewpoint image including an object indicating the point of gaze of a determined size; A display control method comprising:

[0109] (Method 2) an acquisition step of acquiring information indicating a position of a point of interest corresponding to a first virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by imaging with a plurality of imaging devices; a determining step of determining a size of the object indicating the point of gaze on the second virtual viewpoint image corresponding to the second virtual viewpoint when a distance from the position of the second virtual viewpoint different from the first virtual viewpoint to a position of the point of gaze is a second distance longer than the first distance; a display control step of controlling the display of a second virtual viewpoint image corresponding to a second virtual viewpoint different from the first virtual viewpoint, the second virtual viewpoint image including an object indicating the point of gaze of a determined size; A display control method comprising:

[0110] (program) A program for causing a computer to execute the display control device according to any one of configurations 1 to 7. [Explanation of symbols]

[0111] 134 First virtual viewpoint information determination unit 136 Virtual viewpoint object generation unit 137 Point of interest object generation unit 138 Object Information Storage Unit 139 Second virtual viewpoint information determination unit

Claims

1. an acquisition means for acquiring information indicating a position of a first virtual viewpoint corresponding to a first virtual viewpoint image generated based on a plurality of captured images obtained by capturing images with a plurality of imaging devices, and information indicating a position of a point of interest corresponding to the first virtual viewpoint; a first determination means for determining, based on information indicating the position of the first virtual viewpoint and information indicating the position of the point of gaze, a position of the second virtual viewpoint that is at a predetermined distance from the position of the first virtual viewpoint in a direction away from the point of gaze relative to the first virtual viewpoint, and such that as the distance from the position of the first virtual viewpoint to the position of the point of gaze increases, the distance from the position of the second virtual viewpoint to the position of the point of gaze increases; second determination means for determining a size of an object indicating the point of interest based on a distance from the position of the first virtual viewpoint to the position of the point of interest; a display control means for controlling display of a second virtual viewpoint image corresponding to the second virtual viewpoint, the second virtual viewpoint image including an object indicating the point of interest having a determined size and an object indicating the first virtual viewpoint; The second determination means determines a size of the object indicating the point of gaze when the distance from the position of the first virtual viewpoint to the position of the point of gaze is a second distance longer than the first distance, compared to a size of the object indicating the point of gaze when the distance from the position of the first virtual viewpoint to the position of the point of gaze is a first distance. A display control device comprising:

2. The display control device according to claim 1 , wherein a size of the object indicating the point of gaze is determined to be proportional to a distance from the position of the first virtual viewpoint to the position of the point of gaze.

3. The display control device according to claim 1, characterized in that the size of the object indicating the point of gaze is determined to be a predetermined size when the distance from the position of the first virtual viewpoint to the position of the point of gaze is less than a threshold, and when the distance from the position of the first virtual viewpoint to the position of the point of gaze is equal to or greater than a threshold, the size of the object is determined to be larger the greater the distance.

4. The display control device according to claim 1 , wherein the object representing the first virtual viewpoint and the object representing the point of gaze are three-dimensional models.

5. an acquiring step of acquiring information indicating a position of a first virtual viewpoint corresponding to a first virtual viewpoint image generated based on a plurality of captured images obtained by capturing images with a plurality of imaging devices, and information indicating a position of a point of interest corresponding to the first virtual viewpoint; a first determination step of determining, based on information indicating the position of the first virtual viewpoint and information indicating the position of the point of gaze, a position of the second virtual viewpoint that is at a predetermined distance from the position of the first virtual viewpoint in a direction away from the point of gaze than the first virtual viewpoint, and such that as the distance from the position of the first virtual viewpoint to the position of the point of gaze increases, the distance from the position of the second virtual viewpoint to the position of the point of gaze increases; a second determination step of determining a size of an object indicating the point of gaze based on a distance from the position of the first virtual viewpoint to the position of the point of gaze; a display control step of controlling the display of a second virtual viewpoint image corresponding to the second virtual viewpoint, the second virtual viewpoint image including an object indicating the point of interest having a determined size and an object indicating the first virtual viewpoint, The second determination step determines a size of the object indicating the point of gaze when the distance from the position of the first virtual viewpoint to the position of the point of gaze is a second distance longer than the first distance, compared to a size of the object indicating the point of gaze when the distance from the position of the first virtual viewpoint to the position of the point of gaze is a first distance. A display control method comprising:

6. A program for causing a computer to execute the display control device according to any one of claims 1 to 4.

Images