Image display apparatus, control method, and program

Abstract

To improve convenience when viewing or operating a virtual viewpoint image.SOLUTION: An image display apparatus displays a virtual viewpoint image, which is a virtual viewpoint image generated from a multi-viewpoint image and in which at least any of the posture and the position of an attention point at a virtual viewpoint can be set. The image display apparatus receives a first parameter that designates at least any of the posture and the position of the attention point at the virtual viewpoint through user operation, acquires a preset second parameter that designates at least any of the posture and the position of the attention point at the virtual viewpoint, and generates a virtual viewpoint image on the basis of at least any of the first parameter and the second parameter to display it on a display section.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] Virtual viewpoint image reproduction processing [Background technology]

[0002] In recent years, attention has been focused on a virtual viewpoint image generation technique that can generate an image seen from an arbitrary viewpoint from a plurality of images captured by a plurality of cameras.

[0003] An arbitrary viewpoint used to generate a virtual viewpoint image is called a virtual camera (or virtual viewpoint), and unlike a physical camera, a virtual camera can move, rotate, and perform various other movements in three-dimensional space without being subject to physical constraints.

[0004] Patent Document 1 discloses a method for controlling various operations of a virtual camera in a configuration in which a virtual viewpoint image is displayed on an image display device equipped with a touch panel, depending on the number of fingers used to tap on the touch panel. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-56978 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, for a user viewing a virtual viewpoint image for the first time, including the method of operating the virtual viewpoint image described in Patent Document 1, it is an unknown experience. When a virtual viewpoint image is demonstrated at an exhibition or the like for such a user, it is sometimes difficult to attract the user's interest because the user does not know what characteristics the image has and does not know how to operate the virtual camera.

[0007] In view of the above problems, an object of the present invention is to improve convenience when viewing or operating a virtual viewpoint image. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, one aspect of an image display device according to the present invention is a virtual viewpoint image generated from a multi-viewpoint image, An image display device that displays a virtual viewpoint image in which at least one of an attitude of a virtual viewpoint and a position of a point of interest can be set, a receiving means for receiving a first parameter that specifies at least one of an attitude of a virtual viewpoint and a position of a point of interest through a user operation; an acquisition means for acquiring a second parameter set in advance, the second parameter specifying at least one of an attitude and a position of a point of interest in a virtual viewpoint; a display unit that generates the virtual viewpoint image based on at least one of the first parameter and the second parameter and displays the virtual viewpoint image on a display unit; The present invention is characterized by having the following. Effect of the Invention

[0009] According to the present invention, it is possible to improve convenience when viewing or operating a virtual viewpoint image. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a virtual viewpoint image generating system. [Diagram 2] FIG. 1 is a diagram illustrating an example of a configuration of an image display device. [Diagram 3] 3A to 3C are diagrams illustrating an example of the position, orientation, and focus point of a virtual camera. [Figure 4] FIG. 11 is a diagram illustrating an example of an operation of the virtual camera by automatic operation. [Diagram 5] 11A and 11B are diagrams illustrating an example of a manual operation screen and manual operation information. [Figure 6] FIG. 1 illustrates an example of a use case. [Figure 7] 11 is a flowchart showing an example of a process for operating a virtual camera. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] First Embodiment In this embodiment, a system will be described that generates a virtual viewpoint image that represents a view from a specified virtual viewpoint, based on a plurality of images captured by a plurality of imaging devices and a specified virtual viewpoint.

[0013] In this embodiment, the term "virtual camera" is used for explanation. The virtual camera is a virtual camera different from the multiple imaging devices actually installed around the imaging area, and is a concept for conveniently explaining the virtual viewpoint related to the generation of the virtual viewpoint image. The virtual viewpoint image can be considered as an image captured from a virtual viewpoint set in a virtual space associated with the imaging area. The position and orientation of the viewpoint in the virtual imaging can be expressed as the position and orientation of the virtual camera. In other words, the virtual viewpoint image can be said to be an image simulating an image captured by a camera when it is assumed that the camera exists at the position of the virtual viewpoint set in the virtual space.

[0014] The order of explanation of this embodiment is as follows: the entire virtual viewpoint image generating system is explained with reference to Fig. 1, and the image display device 104 therein is explained with reference to Fig. 2. This image display device 104 performs virtual camera control processing using a plurality of different parameters, which is a feature of this invention. In this embodiment, manual operation and automatic operation are used as a method of specifying a plurality of different parameters for the virtual camera. The control processing of the virtual camera is explained with reference to Fig. 6, and an example of virtual camera control and a display example of a virtual viewpoint image are explained with reference to Fig. 7. The configuration of the virtual camera, and the manual operation and automatic processing required for these explanations are explained with reference to Figs. 3 to 5.

[0015] (Configuration of the virtual viewpoint image generation system) The configuration of a virtual viewpoint image generation system 100 according to this embodiment will be described with reference to Fig. 1(a). The virtual viewpoint image generation system 100 has n sensor systems, from sensor system 101a to sensor system 101n, and each sensor system has at least one camera, which is an image capturing device. Hereinafter, unless otherwise specified, the n sensor systems will not be distinguished from one another and will be referred to as multiple sensor systems 101.

[0016] FIG. 1(b) is a diagram showing an example of installation of a plurality of sensor systems 101. The plurality of sensor systems 101 are installed so as to surround an area 120, which is an area to be photographed, and each of them photographs the area 120 from a different direction. In the example of this embodiment, the area to be photographed 120 is the field of a stadium where a soccer match is played, and n (e.g., 100) sensor systems 101 are installed so as to surround the field. However, the number of sensor systems 101 to be installed is not limited, and the area to be photographed 120 is not limited to the field of a stadium. For example, the area 120 may include stadium seating, or the area 120 may be an indoor studio, stage, or the like.

[0017] Furthermore, the multiple sensor systems 101 do not have to be installed all around the area 120, and may be installed only in a part of the periphery of the area 120 depending on restrictions on installation locations, etc. Furthermore, the multiple cameras of the multiple sensor system 101 may include imaging devices with different functions, such as a telephoto camera and a wide-angle camera.

[0018] The multiple cameras of the multiple sensor systems 101 capture images synchronously. The multiple images obtained by capturing images with these cameras are called multi-view images. Note that each of the multi-view images in this embodiment may be a captured image, or may be an image obtained by performing image processing, such as a process of extracting a predetermined area, on the captured image.

[0019] The multiple sensor systems 101 may have a microphone (not shown) in addition to a camera. The microphones of the multiple sensor systems 101 synchronously pick up sound. Based on the picked up sound, an acoustic signal can be generated that is played back together with the display of an image on the image display device 104. In the following, for the sake of simplicity, a description of sound will be omitted, but it is assumed that images and sound are basically processed together.

[0020] The image recording device 102 acquires multi-viewpoint images from multiple sensor systems 101, and stores them together with the time code used for shooting in a database 103. The time code is information for uniquely identifying the time when the image was captured by the image capturing device, and can be specified in a format such as "day:hour:minute:second.frame number".

[0021] The image display device 104 displays images based on the multi-viewpoint images corresponding to the time code from the database 103 and the user's manual operation of the virtual camera. The image display device 104 is, for example, a portable terminal such as a tablet terminal or a smartphone. In particular, if the image display device 104 has a touch panel, it is more convenient to implement this embodiment.

[0022] The virtual camera 110 is set in a virtual space associated with the area 120 and can view the area 120 from a different viewpoint than any of the real cameras of the multiple sensor systems 101. The virtual camera 110 and its operation will be described in detail below with reference to FIG.

[0023] In this embodiment, manual operation and automatic operation are used as methods for specifying a plurality of different parameters for virtual camera 110, and automatic operation information for the virtual camera will be described later in Fig. 4, and a manual operation method for virtual camera 110 will be described later in Fig. 5. The parameters set by manual operation are an example of first parameters, and the parameters set by automatic operation are an example of second parameters.

[0024] The image display device 104 executes a virtual camera control process that combines automatic and manual operations in an effective manner, and generates a virtual viewpoint image from a multi-viewpoint image based on the controlled virtual camera and time code. This virtual camera control process is described in detail in FIG. 6, and an example of this control and an example of a virtual viewpoint image generated using the virtual camera are described in detail in FIG. 7.

[0025] The virtual viewpoint image generated by the image display device 104 is an image that represents the view from the virtual camera 110. The virtual viewpoint image in this embodiment is also called a free viewpoint image, and is displayed on the touch panel, liquid crystal display, or the like of the image display device 104.

[0026] The configuration of the virtual viewpoint image generating system 100 is not limited to the example shown in Fig. 1(a). The image display device 104 may be configured to be separate from the operation device and the display device. Alternatively, a plurality of display devices may be connected to the image display device 104, and a virtual viewpoint image may be output to each of them.

[0027] In the example of FIG. 1(a), the database 103 and the image display device 104 are described as separate devices, but the database 103 and the image display device 104 may be integrated into one device.

[0028] The image display device 104 may switch between allowing access to all time codes or allowing access to only some of the time codes. In the case of some of the time codes, a 3D model of some of the time codes may be copied to the image display device 104 in advance.

[0029] In this embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image.

[0030] (Functional configuration of image display device) Next, the configuration of the image display device 104 will be described with reference to FIG.

[0031] 2(a) is a diagram showing an example of the functional configuration of the image display device 104. The image display device 104 uses the functions shown in the figure to execute virtual camera control processing using manual operation and automatic operation as a method of specifying multiple parameters for the virtual camera.

[0032] The image display device 104 includes a manual operation unit 201 , an automatic operation unit 202 , a model generation unit 203 , an image generation unit 204 , a sensor control unit 205 , and a virtual camera control unit 206 .

[0033] The manual operation unit 201 is a receiving unit for receiving input information manually operated by the user for the virtual camera 110 or the time code. A touch panel, a joystick, a keyboard, or the like is used for manually operating the virtual camera. Note that the input information may be acquired through the operation of another input device. Details of the manual operation screen will be described later with reference to FIG. 5.

[0034] The automatic operation unit 202 automatically operates the virtual camera 110 and the time code. An example of the automatic operation in this embodiment will be described in detail with reference to Fig. 4, in which the image display device 104 operates or sets the virtual camera or the time code separately from the user's manual operation. The automatic operation allows the user to experience the virtual viewpoint image as if it were automatically operated separately from the operation performed by the user.

[0035] Information used by the automatic operation unit 202 for automatic operation is called automatic operation information. For example, the coordinates of three-dimensional models related to players and the ball, which will be described later, and position information measured by a GPS or the like are used. Note that the automatic operation information is not limited to these, and may be anything that can specify information related to a virtual camera and a time code without relying on user operation.

[0036] Model generation unit 203 generates a 3D model representing the 3D shape of an object in area 120 based on a multi-view image acquired by specifying a time code from database 103. Specifically, a foreground image in which a foreground area corresponding to an object such as a ball or a person is extracted, and a background image in which a background area other than the foreground area is extracted are acquired from the multi-view image. Then, model generation unit 203 generates a 3D model of the foreground based on the multiple foreground images.

[0037] The three-dimensional model is generated by a shape estimation method such as visual hull method, and is three-dimensional shape data composed of a group of points. However, the format of the three-dimensional shape data representing the shape of the object is not limited to this. The three-dimensional model of the background may be acquired in advance by an external device. The model generation unit 203 may be configured to be included in the image recording device 102 instead of the image display device 104. In this case, the three-dimensional model is recorded in the database 103, and the image display device 104 reads the three-dimensional model from the database 103. For the three-dimensional model of the foreground, the coordinates of each ball, person, and object may be calculated and stored in the database 103. The coordinates of each object may be specified as automatic operation information, which will be described later with reference to FIG. 4(c), and used for automatic operation.

[0038] The image generating unit 204 generates a virtual viewpoint image from the three-dimensional model based on the controlled virtual camera. Specifically, for each point constituting the three-dimensional model, appropriate pixel values ​​are acquired from the multi-viewpoint image and coloring processing is performed. The colored three-dimensional model is then placed in a three-dimensional virtual space, projected onto the virtual viewpoint, and rendered to generate a virtual viewpoint image. However, the method of generating the virtual viewpoint image is not limited to this, and various methods can be used, such as a method of generating a virtual viewpoint image by projective transformation of a captured image without using a three-dimensional model.

[0039] The sensor detection unit 205 detects physical movement and rotation of the image display device 104. For example, it detects a sensor value of an acceleration sensor or the like and determines whether the image display device 104 is held by a user or placed on a desk or the like. Note that the sensor is not limited to an acceleration sensor as long as it can detect physical movement and rotation of the image display device 104.

[0040] The virtual camera control unit 206 controls the virtual camera and the time code by appropriately combining user input information acquired by the manual operation unit 201 and automatic operation information used by the automatic operation unit 202. Details of the processing by the virtual camera control unit 206 will be described later with reference to Fig. 6, and an example of a virtual viewpoint image generated by the controlled virtual camera will be described later with reference to Fig. 7.

[0041] Furthermore, the virtual camera control unit 206 sets virtual camera operations to be the subject of automatic operation and manual operation, which are called operation settings, and will be described in detail later with reference to FIGS. 4(a) and (b).

[0042] (Hardware configuration of image display device) Next, the hardware configuration of the image display device 104 will be described with reference to Fig. 2(b). The image display device 104 includes a CPU (Central Processing Unit) 211, a RAM (Random Access Memory) 212, and a ROM (Read Only Memory) 213. The image display device 104 also includes an operation input unit 214, a display unit 215, and an external interface 216.

[0043] The CPU 211 performs processing using programs and data stored in the RAM 212 and the ROM 213. The CPU 211 controls the overall operation of the image display device 104, and executes processing for realizing each function shown in Fig. 2(a). Note that the image display device 104 may have one or more pieces of dedicated hardware different from the CPU 211, and at least a part of the processing by the CPU 211 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

[0044] The ROM 213 holds programs and data. The RAM 212 has a work area for temporarily storing programs and data read from the ROM 213. The RAM 212 also provides a work area used by the CPU 211 when it executes each process.

[0045] The operation input unit 214 is, for example, a touch panel and acquires information operated by a user. For example, the operation input unit 214 receives operations for a virtual camera or a time code. The operation input unit 214 may be connected to an external controller and receive input information related to the operation.

[0046] The external controller is, for example, a three-axis controller such as a joystick, a mouse, etc. However, the external controller is not limited to these.

[0047] The display unit 215 is a touch panel or a screen, and displays the generated virtual viewpoint image. In the case of a touch panel, the operation input unit 214 and the display unit 215 are integrated into one unit.

[0048] The external interface 216 transmits and receives information to and from the database 103 via, for example, a LAN or the like. Information may also be transmitted to an external screen via an image output port such as HDMI (registered trademark) or SDI. Image data may also be transmitted via Ethernet or the like.

[0049] The sensor 217 is an acceleration sensor, a gyro sensor, or the like, and detects the physical movement and rotation of the image display device 104. Note that the sensor is not limited to these, and it is sufficient if it can detect the physical movement and rotation of the image display device 104.

[0050] (Virtual camera movement) Next, the operation of virtual camera 110 (or virtual viewpoint) will be described with reference to Fig. 3. In order to explain the operation, the position, orientation, view frustum, point of interest, etc. of the virtual camera will first be described.

[0051] The virtual camera 110 and its operation are specified using a coordinate system, which is a general three-dimensional Cartesian coordinate system consisting of an X-axis, a Y-axis, and a Z-axis, as shown in FIG.

[0052] This coordinate system is set for the subject and used. The subject is a stadium field, a studio, etc. As shown in Fig. 3(b), the subject includes the entire stadium field 391, as well as a ball 392 and players 393 on the field. The subject may also include spectator seats around the field.

[0053] The coordinate system for the subject is set with the center of field 391 as the origin (0,0,0). The X axis is the long side direction of field 391, the Y axis is the short side direction of field 391, and the Z axis is the vertical direction to the field. Note that the settings of the coordinate system are not limited to these.

[0054] Next, the virtual camera will be described with reference to Figures 3(c) and 3(d). The virtual camera serves as a viewpoint for drawing a virtual viewpoint image. In the quadrangular pyramid shown in Figure 3(c), the vertex represents the position 301 of the virtual camera, and the vector extending from the vertex represents the attitude 302 of the virtual camera. The position of the virtual camera is expressed by the coordinates (x, y, z) in a three-dimensional space, and the attitude is expressed by a unit vector with the components of each axis as scalars.

[0055] The virtual camera attitude 302 passes through the center points of the front clip plane 303 and the rear clip plane 304. The space 305 sandwiched between the front clip plane 303 and the rear clip plane is called the virtual camera's viewing frustum, which is the range in which the image generator 203 generates a virtual viewpoint image (or the range in which the virtual viewpoint image is projected and displayed; hereinafter, this is referred to as the display area of ​​the virtual viewpoint image). The virtual camera attitude 302 is expressed by a vector, and is also called the optical axis vector of the virtual camera.

[0056] The movement and rotation of the virtual camera will be explained with reference to Fig. 3(d). The virtual camera moves and rotates in a space expressed in three-dimensional coordinates. The movement 306 of the virtual camera is the movement of the virtual camera position 301, and is expressed by the components of each axis (x, y, z). The rotation 307 of the virtual camera is expressed by Yaw, which is a rotation around the Z axis, Pitch, which is a rotation around the X axis, and Roll, which is a rotation around the Y axis, as shown in Fig. 3(a).

[0057] These allow the virtual camera to move and rotate freely in the three-dimensional space of the subject (field), and any area of ​​the subject can be generated as a virtual viewpoint image. In other words, the shooting position and shooting direction of the virtual camera can be controlled by specifying the X, Y, and Z coordinates (x, y, z) of the virtual camera and the rotation angles of the X, Y, and Z axes (pitch, roll, yaw).

[0058] Next, the position of the target point of the virtual camera will be described with reference to Fig. 3(e) , which shows a state in which the attitude 302 of the virtual camera is directed toward a target plane 308.

[0059] The plane of interest 308 is, for example, the XY plane (field plane, Z=0) described in Fig. 3(b). The plane of interest 308 is not limited to this, and may be a plane parallel to the XY plane and at a height corresponding to the player's viewpoint (for example, Z=1.7m). Note that the plane of interest 308 does not have to be parallel to the XY plane.

[0060] The position (coordinates) of the virtual camera's point of interest indicates the intersection 309 of the virtual camera's orientation 302 (vector) and the plane of interest 308. The coordinates 309 of the virtual camera's point of interest can be calculated as unique coordinates once the virtual camera's position 301, orientation 302, and plane of interest 308 are determined. Note that calculation of the intersection of the vector and plane used in this case (virtual camera's point of interest 309) is omitted because it can be achieved by a known method. Note that a virtual viewpoint image may be generated so that the coordinates 309 of the virtual camera's point of interest is at the center of the drawn virtual viewpoint image.

[0061] In this embodiment, the movement of the virtual camera that can be realized by a combination of the movement and rotation of the virtual camera is called a virtual camera operation. There are an infinite number of movements of the virtual camera that can be realized by a combination of the above-mentioned movements and rotations, and among them, a combination that is easy for the user to see or operate may be defined in advance as a virtual camera operation. There are a plurality of representative virtual camera operations according to this embodiment, including enlargement (zoom in), reduction (zoom out), translation of the point of interest, and horizontal or vertical rotation around the point of interest that is different from the above rotation. These operations will be described next.

[0062] Scaling is the operation of enlarging or reducing the display of an object in the display area of ​​the virtual camera. Scaling of an object is achieved by moving the position 301 of the virtual camera back and forth along the virtual camera's attitude direction 302 (optical axis direction). When the virtual camera moves forward along the attitude direction (optical axis direction) 302, it approaches the object in the display area without changing its direction, and as a result, the object is displayed as enlarged. Conversely, when the virtual camera moves backward along the attitude direction (optical axis direction), the object is displayed as reduced. Note that scaling is not limited to these methods, and changes in the focal length of the virtual camera, etc. may also be used.

[0063] The translation of the attention point is an operation of moving the attention point 309 of the virtual camera without changing the attitude 302 of the virtual camera. This operation will be described with reference to Fig. 3(f), where the translation of the attention point 309 of the virtual camera is a trajectory 320, and refers to the movement of the virtual camera from the attention point 309 to the attention point 319. In the translation of the attention point 320, the position of the virtual camera changes but the attitude vector 302 does not change, so that the virtual camera operates by moving only the attention point and continuing to view the subject from the same direction. Note that since the attitude vector 302 does not change, the translation 320 of the attention point 309 of the virtual camera and the translation 321 of the position 301 of the virtual camera have the same movement trajectory (movement distance).

[0064] Next, horizontal rotation of the virtual camera around the point of interest will be described with reference to FIG. 3(g). Horizontal rotation around the point of interest is an operation in which the virtual camera rotates 322 around the point of interest 309 along a parallel plane of the plane of interest 308, as shown in FIG. 3(g). In the horizontal rotation around the point of interest 309, the attitude vector 302 changes while continuing to face the point of interest 309, and the coordinates of the point of interest 309 do not change. Since the point of interest 309 does not move due to the horizontal rotation around the point of interest, the user can view the surrounding subject from various angles without changing the height from the plane of interest. This operation does not shift the point of interest and does not change the height distance from the subject (plane of interest), so the user can easily view the target to be watched from various angles.

[0065] Note that the trajectory 322 of the virtual camera position 301 during rotation is on a plane parallel to the plane of interest 308, and the center of rotation is 323. That is, assuming that the XY coordinates of the point of interest are (X1, Y1) and the XY coordinates of the virtual camera are (X2, Y2), the position of the virtual camera is determined so that it moves on a circle of radius R, where R^2=(X1-X2)^2+(Y1-Y2)^2. Also, by determining the attitude so that the attitude vector is in the XY coordinate direction of the point of interest from the virtual camera position, the subject can be viewed from various angles without shifting the point of interest.

[0066] Next, vertical rotation of the virtual camera around the point of interest will be described with reference to FIG. 3(h). Vertical rotation around the point of interest is an operation in which the virtual camera rotates 324 around the point of interest 309 along a plane perpendicular to the point of interest 308. In vertical rotation around the point of interest, the attitude vector 302 changes while continuing to face the point of interest 309, and the coordinates of the point of interest 309 do not change. Unlike the above-mentioned horizontal rotation 322, this is an operation in which the height of the virtual camera from the point of interest changes. In this case, the position of the virtual camera can be determined so that the distance between the virtual camera position and the point of interest does not change on a plane that includes the virtual camera position and the point of interest and is perpendicular to the XY plane. In addition, by determining the attitude so that the attitude vector is from the virtual camera position to the point of interest position, the subject can be viewed from various angles without shifting the point of interest.

[0067] By combining the above-described horizontal rotation 322 and vertical rotation 324 about the point of interest, the virtual camera can realize an operation that allows the subject to be viewed from various angles of 360 degrees without changing the point of interest 309. An example of virtual camera operation control using this operation will be described later with reference to FIG.

[0068] The virtual camera operations are not limited to these, and may be any operations that can be realized by a combination of the movement and rotation of the virtual camera. The virtual camera operations described above may be performed in the same manner whether they are automatic operations or manual operations.

[0069] (Operation settings) In this embodiment, the virtual camera operations described in Fig. 3 can be freely assigned to automatic operations and manual operations. This assignment is called an operation setting.

[0070] Next, the operation settings will be described with reference to FIG. 4(a) and FIG. 4(b).

[0071] Fig. 4(a) is a list in which identifiers are assigned to virtual camera operations, and is used for operation setting. In Fig. 4(a), the virtual camera operations described in Fig. 3 are set for each operation identifier (ID). For example, identifier=2 is set to translation of the attention point described with reference to Fig. 3(f), identifier=3 is set to horizontal rotation about the attention point described with reference to Fig. 3(g), and identifier=4 is set to vertical rotation about the attention point described with reference to Fig. 3(h). Other identifiers can be set in a similar manner.

[0072] The virtual camera control unit 206 uses this identifier to specify the virtual camera operation to be assigned to the automatic operation or manual operation. Note that, in one example, multiple operation identifiers may be assigned to the same operation. For example, the translation of the attention point described with reference to FIG. 3(f) with a movement distance of 10 m may be set to identifier=5, and the translation of the attention point described with reference to FIG. 3(f) with a movement distance of 20 m may be set to identifier=6.

[0073] Figure 4(b) is an example of operation settings. In Figure 4(b), the translation of the virtual camera's focal point (identifier = 2) is set as the target of automatic operation. In addition, zoom in / out (identifier = 1) and horizontal rotation (identifier = 3) / vertical rotation (identifier = 4) around the focal point are set as targets of manual operation.

[0074] The virtual camera control unit 206 of the image display device 104 controls the virtual camera operation by automatic operation and manual operation according to this operation setting. In the example of Fig. 4(b), the attention point of the virtual camera is translated in accordance with the time code by automatic operation, and the subject can be viewed from various angles with the attention point as the center by the user's manual operation.

[0075] A detailed description of the virtual camera operation and the display example of the virtual viewpoint image according to this operation setting example will be given later with reference to Fig. 4(c), Fig. 4(d) and Fig. 7. The combinations that can be set for the operations are not limited to the example in Fig. 4(b), and the contents of the operation settings can be changed midway. In addition, the number of operations that can be set for automatic operation or manual operation is not limited to one, and multiple virtual camera operations can be set for each.

[0076] The combination of operations to be set may be preset by the image display device 104 as setting candidates, or may be set by accepting from the user a selection operation of the virtual camera operations to be assigned to automatic operation and manual operation.

[0077] (Automated Operation Information) Now, regarding the operation of the virtual camera that is automatically operated using the operation settings in Figures 4(a) and (b), information that indicates specific details for each time code is called automatic operation information. Here, the automatic operation information will be explained with reference to Figures 4(c) and (d).

[0078] As described above, the identifier (type) of the virtual camera operation specified by the automatic operation information is determined by the operation settings in Figures 4(a) and (b). Figure 4(c) is an example of automatic operation information for the operation setting example in Figure 4(b).

[0079] In the example of automatic operation information in Fig. 4(c), the position and orientation of the virtual camera's focus point are specified for each time code as the virtual camera operation. Like the time codes from the first line to the last line in Fig. 4(c), the continuous time codes (2020-02-02 13:51:11.020 to 2020-02-02 13:51:51.059) are called a time code section. In addition, the start time (13.51.11.020) is called the start time code, and the end time (13:51:51.059) is called the end time code.

[0080] The coordinates of the attention point are specified by (x, y, z), where z is a constant value, and only (x, y) change. If the continuous values ​​of these coordinates are illustrated in the three-dimensional space 411 shown in FIG. 4(d), it will look like a trajectory 412 that moves in parallel at a constant height from the field surface (identifier=2 in FIG. 4(b)). On the other hand, for the attitude of the virtual camera, only the initial value is specified in the corresponding time code section. Therefore, the automatic operation using the automatic operation information 401 operates the virtual camera so that the attitude does not change from the initial value (-Y direction) in the specified time code section, and the attention point follows the trajectory 412 (virtual viewpoints 421 to 423).

[0081] For example, if the trajectory 412 is set to track the position of the ball in a field sport such as rugby (position of the focal point = position of the ball), this automatic operation will cause the virtual camera to always track the ball and the subjects around it without changing its posture.

[0082] Examples of virtual viewpoint images when the virtual camera is at virtual viewpoints 421 to 423 using the automatic operation information shown in Fig. 4(d) are shown in Fig. 4(e) to Fig. 4(g), respectively. As shown in this series of figures, although the subject that appears in the angle of view of the virtual camera changes as the time code and position change, all virtual viewpoint images have the same attitude vector, i.e., the -Y direction, and are oriented as if the field is being viewed from the side.

[0083] Note that the coordinates of continuous values ​​such as the trajectory 412 may be the coordinates of the ball or players, which are foreground objects, calculated when the 3D model is generated as described in Fig. 2. Note that data acquired from a device other than the image display device 104 may be used for the automatic operation information.

[0084] For example, a positioning tag may be attached to the ball or the player's clothing included in the subject, and the position information may be used as a point of interest. In addition, the coordinates of the foreground object calculated when the 3D model is generated or the position information obtained by the positioning tag may be used as is for the automatic operation information, or information calculated separately from such information may be used.

[0085] For example, with regard to the coordinate values ​​and positioning values ​​of a foreground object of a three-dimensional model, the z value varies depending on the posture of a player and the position of the ball, but z may be a fixed value and only the x and y values ​​may be used. The z value may be set to the height of the field (z=0) or a height to which the viewer can naturally direct their eyes (e.g., z=1.7 m). Note that the device for acquiring the automatic operation information is not limited to the positioning tag, and, for example, information received from an image display device other than the image display device 104 currently being displayed may be used. For example, an operator who operates the automatic operation information may input the automatic operation information by operating an image display device other than the image display device 104.

[0086] The time code section that can be specified in the automatic operation information may be the entire game in a field sport, or may be a part of it.

[0087] The automatic operation information is not limited to the above-described examples. The items of the automatic operation information are not limited as long as they are information related to the virtual camera operation. Other examples of the automatic operation information will be described later.

[0088] (Manual operation screen) Manual operation of the virtual camera according to this embodiment will be described with reference to Fig. 5. The virtual camera operation specified by manual operation is determined by the operation settings shown in Figs. 4(a) and (b).

[0089] Here, a manual operation method for the virtual camera operation assigned in the operation settings and a manual operation method for the time code will be described.

[0090] 5(a) is a diagram for explaining the configuration of a manual operation screen 501 of the operation input unit 214 of the image display device 104. The manual operation screen 501 is broadly composed of two areas: a virtual camera operation area 502 and a time code operation area 503. The virtual camera operation area 502 is a GUI (Graphical User Interface) that accepts a user's manual operation of the virtual camera, and the time code operation area 503 is a GUI that accepts a user's manual operation of the time code.

[0091] First, a description will be given of the virtual camera operation area 502. The virtual camera operation area 502 executes the virtual camera operation set in the operation settings in Fig. 4(a) and (b) according to the type of operation received.

[0092] In the example of FIG. 5, since a touch panel is used, the types of operations are touch operations such as tapping and swiping. On the other hand, when a mouse is used to input user operations, a click operation may be performed instead of a tap operation, and a drag operation may be performed instead of a swipe operation. To each of these touch operations, operations such as translation of the focus point and horizontal / vertical rotation around the focus point, which are virtual camera operations shown in FIG. 4, are assigned. A table that determines such assignments is called a manual operation setting, and is shown in FIG. 5(b).

[0093] The manual operation settings will be described with reference to Fig. 5(b). In the figure, the items include touch operation, number of touches, touch area, and virtual camera operation identifier (ID).

[0094] The touch operation item lists touch operations possible in the virtual camera operation area 502. For example, tapping, swiping, etc. are included. The number of touches item defines the number of fingers required for the touch operation. The touch area item specifies the area to be processed by the touch operation. For example, the virtual camera operation area (whole) or the virtual camera operation area (right end) is specified. The contents of the manual operation accepted from the user are distinguished according to the contents of the three items of the touch operation item, the number of touches item, and the touch area item. Then, by assigning a virtual camera operation identifier (ID) to each manual operation, one of the virtual camera operations is executed when each manual operation is accepted. The virtual camera operation identifier (ID) identifies the type of virtual camera operation described in FIG. 3, as described with reference to FIG. 4(a).

[0095] In the allocation example of Fig. 5(b), for example, when a horizontal swipe operation with one finger is accepted on the virtual camera operation area 502 by specifying the second line (No. 2), horizontal rotation (action identifier = 3) is executed around the focus point. Also, when a vertical swipe operation with one finger is accepted on the virtual camera operation area 502 by specifying the third line (No. 3), vertical rotation (action identifier = 4) is executed around the focus point.

[0096] As for the type of touch operation used for manual operation of the virtual camera, gesture operations according to the number of fingers may be used, or a relatively simple gesture operation shown in a third embodiment described later may be used. Either of these can be specified in the manual operation setting in Fig. 5(b). Note that the operations that can be specified in the manual operation setting are not limited to touch operations, and operations using a device other than a touch panel can also be specified.

[0097] Next, the time code operation area 503 will be described. The time code operation area 503 is made up of elements 512 to 515 for operating the time code. The main slider 512 can be operated for all time codes of the shooting data. Any time code in the main slider 512 can be specified by selecting the position of the knob 522 by a dragging operation or the like.

[0098] The sub-slider 513 displays an enlarged portion of the entire time code, and allows the user to operate the time code at a finer granularity than the main slider 512. The user can select the position of the knob 523 by dragging or the like to specify the time code. The main slider 512 and the sub-slider 513 have the same length on the screen, but the width of the time code that can be selected is different. For example, the main slider 512 allows the user to select from three hours, which is the length of one game, whereas the sub-slider 513 allows the user to select from 30 seconds, which is a part of the game. In other words, the scales of the sliders are different, and the sub-slider 513 allows the user to specify the time code at a finer granularity (unit), such as in units of frames. In one example, the sub-slider 513 allows the user to specify from 15 seconds before to 15 seconds after the time at which the knob 522 is located.

[0099] The time code specified using the knob 522 of the main slider 521 or the time code specified using the knob 523 of the sub-slider 513 may be displayed as a number in the format "day:hour:minute:second.frame number".

[0100] Note that the sub-slider 513 does not have to be displayed all the time. For example, it may be displayed after a display instruction is received, or when a specific operation such as pausing is instructed. The time code section that can be selected by the sub-slider 513 may be variable. When a specific operation such as pausing is received, a section of about 15 seconds before and after the time of the knob 523 at the time the pause instruction was received may be displayed.

[0101] The slider 514 for specifying the playback speed can specify playback speeds such as normal playback or slow playback. The count-up interval of the time code is controlled according to the playback speed selected with the knob 524. An example of controlling the count-up interval of the time code will be described with reference to the flowchart of FIG.

[0102] The cancel button 515 may be used to cancel each operation related to the time code, or to clear the pause and return to normal playback. Note that the button is not limited to cancel, as long as it is a button for performing manual operations related to the time code.

[0103] The manual operation screen may have an area other than the virtual camera operation area 502 and the time code operation area 503. For example, match information may be displayed in area 511 at the top of the screen. Match information may include the venue, date and time, match cards, and score status. However, the match information is not limited to these.

[0104] Also, an exceptional operation may be assigned to the area 511 at the top of the screen. For example, when the area 511 at the top of the screen receives a double tap, the operation may be to move the position and orientation of the virtual camera to a position where the entire subject can be viewed from above. Manual operation of the virtual camera may be difficult for an inexperienced user, and the user may become unclear as to where he or she is. In such a case, an operation may be assigned to return to a bird's-eye view point where the position and orientation are easy for the user to understand. The bird's-eye view image is a virtual viewpoint image looking down on the subject from the Z axis as shown in FIG. 3(b).

[0105] Note that the configuration is not limited to these as long as the virtual camera or the time code can be operated, and the virtual camera operation area 502 and the time code operation area 503 do not have to be separated. For example, a double tap on the virtual camera operation area 502 may be processed as an operation on the time code, such as pausing. Note that although the case where the operation unit input unit 214 is a tablet 500 has been described, the operation and display device are not limited to this. For example, a double tap on the right half of the display area 502 may be set to fast forward 10 seconds, and a double tap on the left half may be set to rewind 10 seconds.

[0106] (Example of a use case) A virtual camera control process using manual operation and automatic operation as a method of specifying a plurality of different parameters for the virtual camera according to this embodiment will be described with reference to the use case example in FIG. 6 and the flowchart in FIG.

[0107] First, an example of a use case in this embodiment will be described with reference to FIG.

[0108] 6(a) shows a situation in which a demo video of a virtual viewpoint image is displayed on the display unit 215 of the image display device 104. The user assumed in this situation is not necessarily in front of the image display device 104, but may be, for example, a person passing by in front of the image display device 104, such as a passerby. Such a user is assumed to be unfamiliar with virtual viewpoint images and methods for viewing virtual viewpoint images.

[0109] In such a situation, the demo video played back on the image display device 104 becomes a virtual viewpoint image in which the virtual camera is automatically operated.

[0110] As an example of the automatic operation, the automatic operation information shown in Fig. 4(c) is used. Therefore, the image display device 104 repeatedly plays back the virtual viewpoint image for the time code section in which the automatic operation information is set, and the display unit 215 of the image display device 104 displays a moving image of the virtual viewpoint image as shown in Fig. 4(e) to (g). The example in Fig. 6(a) is an example in which the time code in Fig. 4(f) is displayed.

[0111] In such a situation where the user is not necessarily in front of the image display device 104, it is possible to hold a demonstration that will attract the user by automatically playing back the highlights of the virtual viewpoint video.

[0112] 6(b) shows a situation in which the image display device 104 is lifted by the user. In response to being lifted by the user, the image display device 104 switches from a state in which the virtual camera is automatically operated to a state in which the virtual camera is manually operated.

[0113] In FIG. 6(b), the image display device 104 displays the manual operation screen 501 shown in FIG. 5 together with the virtual viewpoint image. Here, the time code does not change from the time when the user lifts it up. Alternatively, the time code may be paused when lifting starts, and the time code may be restarted counting up after the user starts manual operation. For example, the transition from FIG. 6(a) to (b) shows an example of lifting by the image display device 104, but these are continuous time codes. This allows the user to seamlessly switch from the demo video being automatically operated to manual operation, and enables the user to start operation by himself from a scene that interests him.

[0114] The user can view a virtual viewpoint image from any position and orientation of the virtual camera by performing operations via the manual operation screen 501. The operation method at this time follows the operation settings for manual operations shown in Figures 4(a) and (b).

[0115] Note that the image display device 104 may maintain the position and orientation of the virtual camera when transitioning from FIG. 6(a) to FIG. 6(b).

[0116] Alternatively, when the user lifts the image display device 104 during the transition, the position and orientation of the virtual camera may be changed according to values ​​acquired from the sensor 217, such as an acceleration sensor or a gyro sensor. For example, Fig. 6(a) shows a virtual camera looking from the side of the field, but in Fig. 6(b) the user lifts the image display device 104 so that it is parallel to the floor, and according to this orientation, the virtual camera may also be changed to one looking down from above the field.

[0117] 6(b), the image display device 104 may repeatedly play back the virtual viewpoint image for the time code section set in the automatic operation information, or may play back the virtual viewpoint image for the time code section not set in the automatic operation information. In this case, in the time code operation area 503, the start position and end position of the main slider 512 may be associated with time codes corresponding to the time code section in which the virtual viewpoint image can be played back in the manual operation state.

[0118] 6(c) shows, for example, a state in which the user stops operating the virtual viewpoint image and places the image display device 104 on a desk. In response to being placed by the user, the image display device 104 switches from a state in which the virtual camera of the virtual viewpoint image is manually operated to a state in which the virtual viewpoint image is automatically operated. In other words, in response to being placed by the user, the image display device 104 returns to a mode in which a demo video is displayed.

[0119] Here, the image display device 104 may perform control so as to play back the demo video from the beginning. For example, the image display device 104 plays back the virtual viewpoint image from the earliest time code among the time codes recorded in the automatic operation information shown in Fig. 4(c) at the focus point of the virtual camera and the attitude of the virtual camera of that time code.

[0120] Thereafter, the processes shown in FIGS. 6(a) to 6(c) are repeated.

[0121] As shown in Fig. 6(a) and (c), the image display device 104 does not display part of the operation UI while the demo video is being displayed. For example, the time code area 503 (512 to 515, 522 to 524) in Fig. 5 is not displayed. Also, as shown in Fig. 6(b), the image display device 104 displays the operation UI in response to some operation by the user.

[0122] In Fig. 6(b), the image display device 104 may start a tutorial before displaying the manual operation screen for a user who has never operated a virtual camera of a virtual viewpoint image. The tutorial may be, for example, a display of an operation method using a video or a pop-up to explain an operation method according to the operation settings of Fig. 4(a) and (b). This allows the user to understand how to operate the virtual camera.

[0123] 6(a) may be switched depending on the user viewing the demo video. For example, the display device 104 may recognize the gender, age, etc. of the user from the appearance captured by an imaging means (not shown), and switch to demo video or automatic operation information according to the recognition result. This enables the display device 104 to play back demo video that is more attractive to the user.

[0124] In the situation shown in Fig. 6(b), the image display device 104 may temporarily suspend the playback of the virtual viewpoint image. Furthermore, the image display device 104 may also change the angle of view to the telephoto side, thereby displaying a wider area to be played back in the virtual viewpoint image. This makes it easier for the user to find the desired position and orientation of the virtual camera, and enables the user to play back the virtual viewpoint image after changing the virtual camera to that position and orientation.

[0125] 6(b), the image display device 104 does not switch to a state in which the virtual camera is manually operated simply by being lifted, but may switch to a state in which the virtual camera is manually operated in response to an operation by the user on the screen, etc. In this way, the user's operation experience can be improved by switching the state in response to the user expressing an intention to operate.

[0126] In the situation shown in Fig. 6(c), the image display device 104 may wait in a paused state until a predetermined time has elapsed since the user placed it on the screen. This allows the user to immediately pick up the image display device 104 again and resume the operation if he or she intends to continue the operation on the virtual viewpoint image. In this case, the image display device 104 transitions to a state of playing the demo video after the predetermined time has elapsed.

[0127] An example of a use case in this embodiment has been described above with reference to FIG.

[0128] (Flowchart of virtual camera control process) Next, an example of processing by the image display device 104 for implementing the use case of FIG. 6 will be described with reference to the flowchart of FIG.

[0129] As a method of specifying a plurality of different parameters for the virtual camera, the virtual camera operations set for each of the automatic operation and manual operation are determined by the operation settings described above. Furthermore, the automatic operation is performed according to the automatic operation information described in Fig. 4, and the manual operation is performed according to the manual setting contents described in Fig. 5. This flowchart describes a virtual camera control process that meaningfully combines both of these operations.

[0130] The flowchart shown in FIG. 7 is implemented by the CPU 211 using the RAM 212 as a workspace and executing a program stored in the ROM 213.

[0131] In S701, the virtual camera control unit 206 starts playing back the virtual viewpoint image. Here, in this embodiment, the initial time code is the start time code described in FIG. 4(c). Here, the virtual camera control unit 206 executes processing related to the time code and the playback speed. For example, when the frame rate of the virtual viewpoint image is 59.94 FPS, the virtual viewpoint image is displayed at intervals of about 16.6 [ms]. Note that the interval of one loop may be realized by setting the update rate (refresh rate) of the image display of a touch panel or the like to 59.94 FPS in the image display device and performing a synchronous process therewith.

[0132] In S702, the virtual camera control unit 206 acquires a three-dimensional model of the current time code via the model generation unit 203. Here, the model generation unit 203 generates a three-dimensional model of the subject from the multi-viewpoint images, as described in FIG.

[0133] In S703, the virtual camera control unit 206 determines whether or not the virtual camera is operated by the user. The user operation may be a case where a user operation on the operation input unit 214 is detected, or a case where the sensor 217 such as an acceleration sensor detects a physical movement of the image display device 104.

[0134] If it is determined that the image display device 104 is not operated by the user (No in S703), the virtual camera control unit 206 executes the process of S704. In this case, for example, the image display device 104 is placed on a desk. If it is determined that the image display device 104 is operated by the user (Yes in S703), the virtual camera control unit 206 executes the process of S705. In this case, for example, the image display device 104 is held by the user. Thus, in S703, the image display device 104 changes between automatic operation and manual operation by detecting the user's intention to operate.

[0135] In S704, the virtual camera control unit 206 automatically operates the virtual camera using the automatic operation information at the corresponding time code. The automatic operation of the virtual camera has been described with reference to FIG. 4, and therefore will not be described here.

[0136] In S705, the virtual camera control unit 206 executes manual operation of the virtual camera. The manual operation of the virtual camera has been described in FIG. 5, and therefore will not be described here. When the time code is changed by manual operation, the virtual camera control unit 206 updates the time code to be displayed to the time code designated by the user's manual operation. The manual operation for designating the time code has been described in FIG. 5, and therefore will not be described here.

[0137] In S706, the virtual camera control unit 206 generates and draws a virtual viewpoint image when capturing an image at a position and attitude of the virtual camera operated by at least one of automatic operation and manual operation. The drawing of the virtual viewpoint image has been described with reference to FIG. 2, and therefore will not be described here.

[0138] In S707, the virtual camera control unit 206 determines whether the time code being processed has reached the end time code. If the end time code has not been reached (No in S611), the virtual camera control unit 206 executes the process of S708. If the end time code has been reached (Yes in S707), the virtual camera control unit 206 executes the process of S709.

[0139] In S708, the virtual camera control unit 206 counts up the time code. In S709, the virtual camera control unit 206 changes the time code to a start time code. Note that the time code according to this embodiment can be specified by "day:hour:minute:second:frame number" as explained in FIG. 1, and the count up is executed on a frame-by-frame basis. In other words, the loop processing of S702 to S710 is executed on a frame-by-frame basis. Also, the processing of S709 causes the virtual viewpoint image to be repeatedly played back.

[0140] When the time code is counted up manually, the interval at which the time code is counted up may be changed according to the selected value 514 of the playback speed slider described in Fig. 5. For example, when a playback speed of 1 / 2 is specified, the time code may be counted up once for every two loop processes of steps S702 to S710.

[0141] In S710, the virtual camera control unit 206 determines whether or not there is an instruction to end the playback of the virtual viewpoint image. For example, the virtual camera control unit 206 determines that the playback is to end when the power is turned off by the power button. If the playback is not to end, the virtual camera control unit 206 executes the process of S702 again and performs the playback process of the virtual viewpoint image.

[0142] The operation control process of this embodiment has been described above.

[0143] The present invention can improve the convenience when viewing or operating a virtual viewpoint image. For example, even for a user who has never experienced a virtual viewpoint image, an image display device can automatically display an interesting virtual viewpoint image when exhibited, and when the user picks up the image display device, a virtual viewpoint image can be displayed by manual operation according to the user's interest. Then, when the user's operation is finished, the display is switched to the automatic display of the virtual viewpoint image, which can attract the interest of the next user.

[0144] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0145] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0146] The disclosure of this embodiment includes the following configurations and methods.

[0147] (Configuration 1) a receiving means for receiving a first parameter that specifies at least one of an attitude of a virtual viewpoint and a position of a point of interest through a user operation; an acquisition means for acquiring a second parameter set in advance, the second parameter designating at least one of an attitude and a position of a point of interest in the virtual viewpoint; a generating means for generating a virtual viewpoint image using the first parameter when operated by a user, and for generating the virtual viewpoint image using the second parameter when not operated by a user; A display unit for displaying the virtual viewpoint image on a display unit; An image display device comprising:

[0148] (Configuration 2) 2. The image display device according to configuration 1, further comprising detection means for detecting a user operation by an acceleration sensor or a gyro sensor.

[0149] (Configuration 3) The image display device according to configuration 1 or 2, characterized in that when switching from a virtual viewpoint image generated using the second parameters to a virtual viewpoint image generated using the first parameters, the generation means changes the time code to a start time code.

[0150] (Configuration 4) The image display device according to any one of configurations 1 to 3, characterized in that when switching from a virtual viewpoint image generated using the second parameters to a virtual viewpoint image generated using the first parameters, the generation means performs processing to pass on the second parameters to the first parameters.

Claims

1. a receiving means for receiving a first parameter that specifies at least one of an attitude of a virtual viewpoint and a position of a point of interest through a user operation; an acquisition means for acquiring a second parameter set in advance, the second parameter designating at least one of an attitude and a position of a point of interest in the virtual viewpoint; a generating means for generating a virtual viewpoint image using the first parameter when operated by a user, and for generating the virtual viewpoint image using the second parameter when not operated by a user; A display unit for displaying the virtual viewpoint image on a display unit; An image display device comprising:

2. 2. The image display device according to claim 1, further comprising a detection unit that detects a user operation by an acceleration sensor or a gyro sensor.

3. The image display device described in claim 1 or 2, characterized in that when switching from a virtual viewpoint image generated using the second parameters to a virtual viewpoint image generated using the first parameters, the generation means changes the time code to a start time code.

4. An image display device as described in any one of claims 1 to 3, characterized in that when switching from a virtual viewpoint image generated using the second parameters to a virtual viewpoint image generated using the first parameters, the generation means performs processing to pass on the second parameters to the first parameters.

5. a receiving step of receiving a first parameter that specifies at least one of an attitude of a virtual viewpoint and a position of a point of interest through a user operation; an acquisition step of acquiring a second parameter set in advance, the second parameter specifying at least one of an attitude and a position of a point of interest in the virtual viewpoint; a generating step of generating a virtual viewpoint image using the first parameter when the virtual viewpoint image is operated by a user, and generating the virtual viewpoint image using the second parameter when the virtual viewpoint image is not operated by a user; a display step of displaying the virtual viewpoint image on a display unit; 13. A method for controlling an image display device comprising:

6. a receiving step of receiving a first parameter that specifies at least one of an attitude of a virtual viewpoint and a position of a point of interest through a user operation; an acquisition step of acquiring a second parameter set in advance, the second parameter specifying at least one of an attitude and a position of a point of interest in the virtual viewpoint; a generating step of generating a virtual viewpoint image using the first parameter when the virtual viewpoint image is operated by a user, and generating the virtual viewpoint image using the second parameter when the virtual viewpoint image is not operated by a user; a display step of displaying the virtual viewpoint image on a display unit; A program for causing a computer to execute the following.

Images