Information processing apparatus, image processing system, information processing method, and storage medium

Abstract

To easily obtain a desired virtual viewpoint image by a simple operation when controlling a virtual camera arranged in a virtual space to generate the virtual viewpoint image.SOLUTION: An information processing apparatus that controls a virtual camera arranged in a virtual space to generate a virtual viewpoint image. A first camera parameter set in advance and indicating an attitude of a virtual camera and a position of a gaze point corresponding to the virtual camera is stored. A second camera parameter which is different from the first camera parameter and indicates the position and attitude of the virtual camera is acquired. Based on the second camera parameter and the first camera parameter, control is performed to change the position and the orientation of the virtual camera indicated by the second camera parameter. The changed position and orientation of the virtual camera are determined based on the second camera parameters and the first camera parameters.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an image processing system, an information processing method, and a program, and in particular to specifying a virtual viewpoint for generating a virtual viewpoint image. [Background technology]

[0002] In recent years, a system has been proposed that generates a virtual viewpoint video of an imaging space from a virtual viewpoint specified by a user, based on a plurality of captured images obtained by a plurality of imaging devices in the imaging space. Patent Document 1 discloses a technology for generating such a virtual viewpoint video.

[0003] Multiple captured images obtained by multiple imaging devices can be stored in an image processing device such as a server. A virtual viewpoint represents the viewpoint of a virtual camera (hereinafter referred to as a virtual camera) that can move freely in three-dimensional space. An image processing device can generate a virtual viewpoint video consisting of multiple virtual viewpoint images (frames) by rendering virtual viewpoint images from such a virtual camera. The virtual viewpoint video is displayed on a display device. A user can view the virtual viewpoint video on the display device. The virtual viewpoint can be specified by the creator of the virtual viewpoint video or the viewer of the virtual viewpoint image. Such virtual viewpoint video technology is used to create more realistic video representations in sports broadcasts and the like. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-211828 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to quickly move the virtual camera from its current position to a predetermined position, the position and orientation of the virtual camera can be registered (preset) in advance. When a user inputs a command to move the virtual camera to a preset position, a camera path indicating the movement of the virtual camera is set so that the virtual camera moves smoothly from its current position to the preset position. At this time, the camera path is set so that the orientation of the virtual camera changes smoothly from the current orientation of the virtual camera to the preset orientation. The virtual camera then moves to the preset position according to the set camera path so as to have the preset orientation. This function of moving the virtual camera to have the registered position and orientation is called a preset function. Using the preset function, frequently used positions and orientations of the virtual camera can be registered. For example, it is possible to register the positions and orientations of the virtual camera that capture frequently captured objects, such as baseball bases or soccer goals in sports broadcasts. Using the preset function makes it easy to move the virtual camera so that such objects are captured, making it easier to create virtual viewpoint footage of a desired play.

[0006] With such a preset function, after the virtual camera is moved, a virtual viewpoint image of an object at a desired position can be created from the virtual camera according to the preset position and orientation. However, depending on the scene, a user may want to move the virtual camera in a different manner. For example, a user may want to move the virtual camera so that an object at a desired position is captured for a longer period of time. Alternatively, a user may want to move the virtual camera so that an object moving to a desired position is more likely to be captured.

[0007] An object of the present disclosure is to facilitate obtaining a desired virtual viewpoint video with a simple operation when controlling a virtual camera placed in a virtual space to generate a virtual viewpoint video. [Means for solving the problem]

[0008] An information processing device according to an embodiment of the present disclosure has the following configuration: An information processing device that controls a virtual camera arranged in a virtual space to generate a virtual viewpoint video, a storage means for storing first camera parameters that are set in advance and indicate the attitude of a virtual camera and the position of a point of interest corresponding to the virtual camera; an acquisition means for acquiring second camera parameters that are different from the first camera parameters and indicate the position and orientation of a virtual camera; a control means for performing control to change the position and attitude of the virtual camera indicated by the second camera parameters based on the second camera parameters and the first camera parameters, The information processing device, wherein the position and orientation of the virtual camera after the change are determined based on the second camera parameters and the first camera parameters. Equipped with. [Effects of the Invention]

[0009] When controlling a virtual camera placed in a virtual space to generate a virtual viewpoint video, a desired virtual viewpoint video can be easily obtained with a simple operation. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating the overall configuration of an image processing system according to an embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the configuration of an information processing apparatus according to an embodiment. [Figure 3] FIG. 4 is a diagram showing an example of registration of preset information. [Figure 4] FIG. 10 is a diagram showing an example of a preset operation. [Figure 5] FIG. 10 is a diagram showing an example of a preset operation. [Figure 6] 1 is a flowchart of an information processing method according to an embodiment. [Figure 7] FIG. 10 is a diagram showing an example of a preset operation. [Figure 8] FIG. 10 is a diagram showing an example of a preset operation. [Figure 9]FIG. 2 is a diagram showing an example of the hardware configuration of a computer used in an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the present disclosure, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] (Image Processing System) 1 is a schematic diagram illustrating an example of an image processing system 101 according to the present disclosure. The image processing system 101 includes a plurality of cameras 102, a plurality of camera control devices 103 each connected to the camera 102, an image processing server 104, and a virtual camera control device 105.

[0013] The multiple cameras 102 are arranged to surround an imaging range 109. The imaging range 109 defines an imaging space to be imaged. One or more subjects may be present in the imaging range 109. Examples of subjects include people such as players and objects such as a ball.

[0014] The camera control device 103 performs image processing on the captured image obtained by the connected camera 102. In this specification, the image processing performed by the camera control device 103 is referred to as pre-processing. The pre-processing includes processing to extract the subject as the foreground from the captured image and generate a silhouette image of the foreground. The silhouette image indicates the foreground area in the captured image. The camera control device 103 sends the captured image and silhouette image obtained by the camera 102 to the image processing server 104.

[0015] The image processing server 104 generates a three-dimensional model of the subject based on images captured by the multiple cameras 102. To this end, the image processing server 104 collects the captured images obtained by each of the multiple cameras 102 and silhouette images obtained based on the captured images. The image processing server 104 then generates a virtual viewpoint video based on these images. In this specification, the image processing performed by the image processing server 104 is referred to as post-stage processing. This processing can be performed by the model generation unit 141, which will be described later.

[0016] For example, the image processing server 104 generates a three-dimensional model of the subject based on multiple silhouette images. The image processing server 104 can create the 3D model of the subject by applying Visual Hull. When multiple subjects are present in the imaging range 109, the image processing server 104 can generate a three-dimensional model for each of the multiple subjects. The image processing server 104 stores the generated three-dimensional models of the subjects in a model DB 142, which will be described later.

[0017] Furthermore, the image processing server 104 generates a virtual viewpoint image of the image capture space from the virtual camera. This processing can be performed by the image generation unit 143, which will be described later. For example, the image processing server 104 can render a virtual viewpoint image of a three-dimensional model of a subject placed in the virtual space from the virtual camera. At this time, the image processing server 104 can place one or more three-dimensional models of subjects in the virtual space in the same manner as the placement of the subjects in the image capture range 109. The placement of the subjects in the image capture range 109 can be determined based on a silhouette image by applying, for example, a visual hull.

[0018] The multiple cameras 102 can synchronously capture images of the imaging range 109 at each of multiple times. The camera control device 103 can perform pre-processing on the captured images obtained at each of the multiple times. The image processing server 104 can then generate a 3D model of the subject at each of the multiple times. Time codes can be assigned to these captured images and 3D models. The assigned time codes may be set based on the capture times of the captured images. In this case, the image processing server 104 can generate a virtual viewpoint image from a virtual camera of the 3D model of the subject at a specific time. The virtual viewpoint image generated in this manner corresponds to one frame of the virtual viewpoint video. The image processing server 104 can generate the virtual viewpoint video by generating virtual viewpoint images for each of the multiple times in this manner.

[0019] The virtual camera control device 105 controls a virtual camera placed in a virtual space to generate a virtual viewpoint image. The virtual camera control device 105 can set camera parameters of the virtual camera according to user input, as described below. In this embodiment, the virtual camera control device 105 sets the camera parameters of the virtual camera for each of a plurality of times. That is, the virtual camera control device 105 can control the movement of the virtual camera. A time code can be assigned to the camera parameters of the virtual camera. In this case, the image processing server 104 can generate a virtual viewpoint image from the virtual camera corresponding to the specific time code of a three-dimensional model of a subject corresponding to the specific time code.

[0020] Furthermore, the virtual camera control device 105 may be capable of displaying a virtual viewpoint image generated by the image processing server 104. The user can operate the virtual camera via the virtual camera control device 105 while checking the virtual viewpoint image. For example, the user can operate the position of the virtual camera. Furthermore, the user can control the orientation of the virtual camera by operating the attitude of the virtual camera.

[0021] The image processing system 101 shown in FIG. 1 has a star-shaped configuration in which multiple camera control devices 103, each connected to a camera 102, are connected to an image processing server 104. However, the configuration of the image processing system 101 is not limited to this configuration. For example, the image processing system 101 may have a configuration in which multiple camera control devices 103 are daisy-chained. In this case, one of the camera control devices 103 can be connected to the image processing server 104. Although FIG. 1 shows ten cameras 102, the number of cameras 102 is not particularly limited. Furthermore, the camera control devices 103 and the cameras 102 do not need to be separate devices. For example, the function of the camera control device 103 may be realized by an image processing unit within the camera 102. Alternatively, the camera control device 103 may be omitted. In this case, the image processing server 104 can perform both pre-processing and post-processing.

[0022] (Information processing device) 2(A) is a schematic diagram showing the appearance of virtual camera control device 105. Virtual camera control device 105 has control terminal 150, operation display 159, video display 160, and controller 158. Controller 158 includes operation controller 158a and setting controller 158b. Controller 158 is installed in front of operation display 159 and video display 160.

[0023] The steering controller 158a has sticks 51a and 51b. The sticks 51a and 51b each have an operating axis with three degrees of freedom. By operating the stick 51a, the virtual camera can be translated along the X, Y, and Z axes. Furthermore, by operating the stick 51b, the virtual camera can be rotated in the pan, tilt, and roll directions. The steering controller 158a also has a switch 52. The switch 52 has two degrees of freedom. In the example of FIG. 2(A), the switch 52 is a lever-type zoom switch with two degrees of freedom. The focal length of the virtual camera can be changed by flipping the switch 52 to the plus or minus side. The focal length of the virtual camera may be changeable within a predetermined focal length range.

[0024] The setting controller 158b has a group of keys for mode setting and a group of keys for preset registration. The group of keys for mode setting includes a plurality of mode keys 53. Each mode key 53 is associated with a specific preset mode. By pressing a mode key 53, the preset mode can be switched to the preset mode corresponding to the pressed mode key 53. FIG. 2(A) shows mode keys 53a to 53d. The group of keys for preset registration includes a plurality of preset keys 54. Preset information is registered in each preset key 54. By pressing a preset key 54, the virtual camera is controlled in accordance with the preset information. FIG. 2(A) shows preset keys 54a to 54d. The preset modes and their switching, preset information and its registration, and virtual camera control in accordance with the preset information will be described in detail below. The setting controller 158b may also have a numeric keypad 55 and an entry key 56.

[0025] 2(B) is a block diagram showing an example of the functional configuration of an image processing system 101 including a virtual camera control device 105. The camera group 120 includes the multiple cameras 102 and the camera control device 103 shown in FIG. 1. The image processing server 104 includes a model generation unit 141, a model DB 142, and an image generation unit 143. As described above, the virtual camera control device 105 includes the control terminal 150, the controller 158, the operation display 159, and the image display 160. The control terminal 150 is an information processing device according to an embodiment of the present disclosure. The control terminal 150 includes an operation detection unit 151, a parameter setting unit 152, a preset unit 153, a preset recording unit 154, a mode setting unit 155, a UI generation unit 156, and an information transmission unit 157.

[0026] The virtual camera control device 105 is used by the user to operate the virtual camera used to generate the virtual viewpoint video. To this end, the control terminal 150 can control the virtual camera. Specifically, the control terminal 150 can control the camera parameters of the virtual camera at each time. In this embodiment, the camera parameters refer to information indicating the state of the virtual camera. In one embodiment, the camera parameters indicate at least the position and orientation of the virtual camera. For example, the camera parameters can include external parameters such as the position or orientation of the virtual camera. The camera parameters can also include internal parameters such as the focal length of the virtual camera. The camera parameters can also include information calculated from the external parameters and internal parameters, such as the position of the gaze point, which will be described later.

[0027] The operation detection unit 151 acquires a user input. In this embodiment, the operation detection unit 151 detects an operation of the controller 158 and transmits the detection result to the parameter setting unit 152, the mode setting unit 155, and the preset unit 153.

[0028] The parameter setting unit 152 sets the camera parameters of the virtual camera. For example, the parameter setting unit 152 can set camera parameters that indicate at least the position and orientation of the virtual camera. In this embodiment, the camera parameters set by the parameter setting unit 152 include information on the position, orientation, and focal length of the virtual camera. The parameter setting unit 152 can set the camera parameters based on the detection result of a user operation on the controller 158.

[0029] As an example, when the position of the virtual camera is expressed by three-dimensional coordinates [X, Y, Z], the parameter setting unit 152 sets values ​​such as X=4.0, Y=9.0, and Z=1.5. In this embodiment, the coordinate unit is [m]. The origin of the coordinates is the center of the 3D model generation range. When the X axis is specified to be parallel to the ground surface, the Y axis is specified to be parallel to the ground surface and perpendicular to the X axis, and the Z axis is specified to be perpendicular to the ground surface.

[0030] Furthermore, when the attitude of the virtual camera is expressed by three angles, [Pan, Tilt, Roll], the parameter setting unit 152 sets values ​​such as Pan=20.0, Tilt=10.0, and Roll=2.0. In this embodiment, the unit of attitude is [degrees]. The range of values ​​representing the attitude is from -180 to 180. Pan is the angle of rotation parallel to the ground surface, and Tilt is the angle of rotation perpendicular to the ground surface. Roll is the angle of rotation around the optical axis of the virtual camera.

[0031] Furthermore, the angle of view of the virtual camera can be expressed by the focal length Zoom. In this case, the parameter setting unit 152 can set a value such as Zoom=6.0. In this embodiment, the unit of the focal length is [mm].

[0032] In this embodiment, the user can directly control camera parameters such as the position and attitude of the virtual camera via the controller 158. In this case, the parameter setting unit 152 sets the camera parameters of the virtual camera according to a user input via the controller 158. On the other hand, as will be described later, the user can also instruct the control terminal 150 to control the movement of the virtual camera according to preset information. In this specification, control of the movement of the virtual camera based on preset information is referred to as a preset operation. When performing a preset operation, the parameter setting unit 152 sets the camera parameters of the virtual camera according to the camera path set by the preset unit 153.

[0033] Using the above method, parameter setting unit 152 can set the camera parameters of the virtual camera at a specific time. Information transmission unit 157 transmits the camera parameters of the virtual camera set by parameter setting unit 152 to image generation unit 143. Thereafter, image generation unit 143 generates a virtual viewpoint image at the specific time from the virtual viewpoint indicated by the camera parameters transmitted by information transmission unit 157.

[0034] The UI generation unit 156 generates an operation UI that is presented to the user via the operation display 159. The user can check the operation UI on the operation display 159. Fig. 3(B) shows an example of the operation UI.

[0035] The mode setting unit 155 sets a preset mode. In this embodiment, the operation detection unit 151 can acquire a user input indicating a preset mode selected from a plurality of preset modes. The user can operate the mode key 53 of the setting controller 158b to select a preset mode. The mode setting unit 155 can set a preset mode in accordance with the operation of the mode key 53 of the setting controller 158b detected via the operation detection unit 151. In this embodiment, a camera position preset mode is registered in the mode key 53a shown in FIG. 2(A), and a camera gaze point preset mode is registered in the mode key 53b. Furthermore, a preset mode different from the mode keys 53a and 53b may be registered in the mode keys 53c and 53d, respectively. The preset unit 153, which will be described later, can set a camera path according to the preset mode indicated by the user input.

[0036] The preset recording unit 154 stores preset information. The preset information indicates camera parameters (sometimes referred to as first camera parameters in this specification) that indicate the orientation of the virtual camera and the position of the point of interest corresponding to the virtual camera. As described above, the preset recording unit 154 stores preset information in association with the preset keys 54. Furthermore, the preset recording unit 154 can store preset information indicating different positions and orientations of virtual cameras in association with each of the multiple preset keys 54. Note that the preset information may include information indicating at least one of the position of the point of interest of the virtual camera and the distance between the virtual camera and the point of interest. Furthermore, the preset information may be information indicating the position and orientation of the virtual camera. As will be described later, the preset information indicating the orientation of the virtual camera, the position of the point of interest, and the distance between the virtual camera and the point of interest can indicate the position and orientation of the virtual camera. Note that, as will be described later, the specific type of information included in the camera parameters is not specified. For example, the camera parameters included in the preset information may include information indicating the position and orientation of the virtual camera and the distance between the virtual camera and the point of interest. The camera parameters including such information can also indicate the posture of the virtual camera and the position of the gaze point corresponding to the virtual camera.

[0037] The preset unit 153 sets a camera path indicating the movement of the virtual camera based on preset information stored in the preset recording unit 154. As will be described later, the preset unit 153 can perform control to change the position and orientation of the virtual camera indicated by camera parameters (sometimes referred to as second camera parameters in this specification) different from the first camera parameters. The camera path set by the preset unit 153 can indicate the movement of the virtual camera from the position and orientation indicated by the second camera parameters. That is, the second camera parameters can indicate the position and orientation of the virtual camera at the start point of the camera path. For example, the second camera parameters can indicate the currently set position and orientation of the virtual camera. That is, the second camera parameters can indicate the position and orientation of the virtual camera when a preset operation is started or when the preset key 54 is operated. Hereinafter, the currently set position and orientation of the virtual camera indicated by the second camera parameters will be referred to as the initial position and initial orientation of the virtual camera. The camera path can indicate the position and orientation of the virtual camera at each time. Furthermore, the camera path set by the preset unit 153 may indicate the camera parameters of the virtual camera at each time. Here, the preset unit 153 can set the camera path based on the initial position and initial orientation of the virtual camera and the gaze point of the virtual camera arranged according to preset camera parameters.

[0038] For example, the preset unit 153 detects the operation of the preset key 54 of the setting controller 158b via the operation detection unit 151. When the preset key 54 is pressed, the preset unit 153 reads out preset information corresponding to the preset key 54 from the preset recording unit 154. Then, the preset unit 153 sets a camera path according to the initial position and initial orientation of the virtual camera, the read preset information, and the current preset mode. The preset unit 153 further transmits the set camera path to the parameter setting unit 152.

[0039] The parameter setting unit 152 can set the position and orientation of the virtual camera at each time according to the camera path, and can also set the camera parameters of the virtual camera at each time according to the camera path.

[0040] In this embodiment, the virtual camera moves smoothly along the camera path. For example, upper limits may be set on the movement speed and attitude change speed of the virtual camera. By smoothly moving the virtual camera in this manner, the user can easily recognize where the virtual camera is moving. However, with this configuration, a time lag occurs until the virtual camera moves to the preset position by the preset operation so as to have the preset attitude.

[0041] (Preset operation) Next, the preset operation will be described. Fig. 3 is a diagram for explaining the preset operation and the camera path. In the example shown in Fig. 3, the subject to be photographed is a baseball game.

[0042] First, preset registration will be explained. Fig. 3(A) is a schematic diagram illustrating the position and orientation of the virtual camera on the field (hereinafter, abbreviated as camera position and orientation) and preset information. Fig. 3(B) shows the UI used when registering presets, which is displayed on the operation display 159. In Fig. 3(B), the display area 15a on the left side of the screen displays the number and preset information for each preset key. Furthermore, the display area 15b on the right side of the screen displays the field and the position of the virtual camera.

[0043] 3(A), the user operates the operation controller 158b to move the virtual camera so that it has a camera position and orientation 61a and faces the point of interest 61b. When the user then presses the entry key 56 and then the preset key 54a, the camera parameters of the virtual camera are registered in the preset recording unit 154 via the preset unit 153. In this example, the camera position and orientation 61a of the virtual camera, the point of interest 61b of the virtual camera, and the radius of the preset sphere 61c are registered in the preset recording unit 154 as preset A.

[0044] In this specification, the point of gaze is the position at which the virtual camera focuses. In one embodiment, the point of gaze is a point along the line of sight from the virtual camera at a distance corresponding to the focal length indicated by the camera parameters. In this case, the preset unit 153 can determine the point of gaze based on the focal length, position, and attitude of the virtual camera. For example, the point of gaze may be a point on the optical axis of the virtual camera. The distance from the virtual camera to the point of gaze may be a value determined according to the focal length of the virtual camera. For example, the distance from the virtual camera to the point of gaze may be calculated by multiplying the focal length of the virtual camera by a predetermined coefficient. As an example, the distance from the virtual camera to the point of gaze corresponding to the focal length may be determined so that the length of an image of a subject located at the point of gaze and having a predetermined length extending in a direction perpendicular to the optical axis in the virtual viewpoint video matches the vertical length of the virtual viewpoint video.

[0045] In this embodiment, information on the position and orientation of the virtual camera and the radius of the preset sphere are registered as preset information. The position of the virtual camera indicated by the preset information will be referred to hereinafter as the preset position. Furthermore, the orientation of the virtual camera indicated by the preset information will be referred to hereinafter as the preset orientation. Below, the preset position and preset orientation may be collectively referred to as the preset position and orientation. Furthermore, the gaze point of the virtual camera indicated in the first camera parameter indicated by the preset information will be referred to hereinafter as the preset gaze point. In this embodiment, the preset gaze point may be the gaze point of the virtual camera positioned according to the preset information. Furthermore, in this embodiment, the preset sphere is a sphere whose center is at the preset gaze point and whose radius is the distance from the preset position to the preset gaze point. The radius of the preset sphere can be calculated based on the preset position and the position of the preset gaze point.

[0046] The UI shown in Fig. 3(B) displays preset information about registered preset A. In Fig. 3(B), preset A is selected as the display target, as indicated by highlight 15c in display area 15a. At this time, display area 15b displays the position (x, y, z) and orientation (u, v, w) of the virtual camera at the time the preset was registered, as well as the radius (r) of the preset sphere. Display area 15b also displays the position of the point of interest at the time the preset was registered.

[0047] Furthermore, after moving the virtual camera to a desired position, the user can register preset information corresponding to another preset key by pressing the entry key 56 and then pressing a preset key other than the preset key 54a. In response to the registration of preset information, the registered preset information and the corresponding preset key name are displayed in the display area 15a. Note that it is also possible to register new preset information to a preset key to which preset information has already been registered.

[0048] In this embodiment, preset information corresponding to the camera position and orientation 62a and the gaze point 62b shown in FIG. 3(A) is registered in the preset key 54b. Furthermore, preset information corresponding to the camera position and orientation 63a and the gaze point 62b shown in FIG. 3(A) is registered in the preset key 54c. Note that the preset spheres corresponding to the preset keys 54b and 54c are not shown in FIG. Note that the gaze points indicated by the preset information registered in each of the multiple preset keys may be the same. For example, the gaze point from the camera position and orientation 62a corresponding to the preset key 54b may be the same as the gaze point from the camera position and orientation 63a corresponding to the preset key 54c.

[0049] The method for registering the preset information is not particularly limited. For example, the user may select the position and orientation of the camera to be preset on the UI of the operation display 159. Alternatively, the user may directly input each value of the preset information using the numeric keypad of the setting controller 158b.

[0050] Next, an example of virtual camera movement according to a preset will be described. As described above, the preset unit 153 can set a camera path according to a preset mode selected from a plurality of preset modes. In this embodiment, the plurality of preset modes include the following camera position preset mode and gaze point preset mode. First, a method of setting a camera path according to a camera position preset mode will be described.

[0051] In the camera position preset mode, the preset unit 153 sets a camera path so that the end point of the movement of the virtual camera along the camera path is a position indicated by the camera parameters of the virtual camera that have been preset in advance. At the end point of the movement of the virtual camera along the camera path, the virtual camera is at a preset position and has a preset attitude.

[0052] In Fig. 3(A), it is assumed that the current position and orientation of the virtual camera is camera position and orientation 63a, and the current point of gaze of the virtual camera is point of gaze 62b. When preset key 54a is pressed in this state, the virtual camera moves along a camera path indicated by trajectory 64b and assumes camera position and orientation 61a. In Fig. 3(A), camera position and orientation 64a indicates the position and orientation of the virtual camera during movement. Furthermore, point of gaze 64c indicates the point of gaze of the virtual camera at camera position and orientation 64a. Furthermore, trajectory 64d indicates the movement of the point of gaze of the virtual camera when the virtual camera moves according to trajectory 64b.

[0053] As described above, in this specification, the currently set position and orientation of the virtual camera is referred to as the initial position and orientation. Also, the point of gaze of the virtual camera indicated by the preset information is referred to as the preset point of gaze. In the example of FIG. 3(A), the camera position and orientation 63a, which is the position and orientation of the virtual camera when the preset key 54a is pressed, corresponds to the initial position and orientation. Also, the camera position and orientation 61a indicated by the preset information registered in the preset key 54a corresponds to the preset position and orientation. And the point of gaze 61b of the virtual camera indicated by the preset information registered in the preset key 54a corresponds to the preset point of gaze.

[0054] In this case, the trajectory 64d of the virtual camera's point of interest is a straight line connecting the position of the point of interest 62b when the preset key 54a is pressed and the position of the point of interest 61b, which is the preset point of interest. When the preset key 54a is pressed, the point of interest of the virtual camera moves at a constant speed along the trajectory 64d. Furthermore, the virtual camera moves at a constant speed from the position indicated by the camera position and orientation 63a, which is the initial position, to the position indicated by the camera position and orientation 61a, which is the preset position and orientation, so as to follow the moving point of interest. The orientation of the moving virtual camera can be controlled according to the position and point of interest of the virtual camera. By smoothly moving the virtual camera in this way, the user can easily recognize where the virtual camera is moving. Furthermore, it also becomes easier to operate the virtual camera after the preset operation.

[0055] As another example, the virtual camera may move at a constant speed along the trajectory 64d from the position indicated by the camera position and orientation 63a, which is the initial position and orientation, to the position indicated by the camera position and orientation 61a, which is the preset position and orientation. Furthermore, the virtual camera may move while changing its orientation by rotating at a constant speed from the orientation indicated by the camera position and orientation 63a, which is the initial position and orientation, to the orientation indicated by the camera position and orientation 61a, which is the preset position and orientation.

[0056] As yet another example, the virtual camera may move at a constant speed along a straight line connecting the position indicated by the camera position and orientation 63a, which is the initial position and orientation, to the position indicated by the camera position and orientation 61a, which is the preset position and orientation. In this case, the virtual camera can also move while rotating at a constant speed and changing its orientation. Thus, in the camera position preset mode, the virtual camera is at a preset position and has a preset orientation at the end point of the camera path. However, the movement trajectory of the virtual camera is not particularly limited.

[0057] According to the camera position preset mode, after the virtual camera moves along the camera path, a virtual viewpoint image with a field of view corresponding to the preset information registered in advance can be obtained. However, it may be desirable to control the virtual camera using a different method. FIG. 4 is a diagram illustrating the preset operation of the virtual camera in another scene of a baseball game. In the scene shown in FIG. 4, a ball 69a hit by a batter 66a is caught by a fielder 67, and then the fielder 67 throws the ball to another fielder 68 at first base, while a batter 66b is running toward first base. In FIG. 4, batter 66a and batter 66b represent the same player at different times. Also, ball 69a and ball 69b represent the same ball at different times.

[0058] The camera position and orientation 70a indicates the initial position and orientation of the virtual camera. The focus of the virtual camera at the initial position is the position of the fielder 67. When the fielder 67 catches the ball and throws it toward first base, the user presses the preset key 54a. In this case, the virtual camera moves toward the position indicated by the preset position and orientation 71a and captures the batter 66b and ball 69b reaching first base. However, when operating in accordance with the camera position preset mode described above, it is possible that the ball 69 or the batter 66 reaches first base before the virtual camera reaches the preset position. Furthermore, it is possible that first base will not be included in the field of view of the moving virtual camera until it approaches the preset position. In this case, it may be impossible to obtain a virtual viewpoint image showing the batter 66 running through first base or the first baseman catching the ball. It may also be impossible to obtain a virtual viewpoint image showing which of the batter 66 or the ball 69 reaches first base first.

[0059] Therefore, the control terminal 150 according to this embodiment can control the camera parameters of the virtual camera in different ways according to the preset information. As described above, in this embodiment, the user selects a preset mode, and the control terminal 150 performs a preset operation according to the selected preset mode. This configuration makes it easy to set a desired camera path. In one embodiment, the preset mode includes a gaze point preset mode. The following describes how the preset unit 153 sets the camera path of the virtual camera according to the gaze point preset mode.

[0060] In the point of interest preset mode, the preset unit 153 acquires second camera parameters indicating the position and orientation of the virtual camera. Then, based on the second camera parameters and the preset point of interest, the preset unit 153 performs control to change the position and orientation of the virtual camera indicated by the second camera parameters. In this embodiment, the preset unit 153 performs such control by setting a camera path of the virtual camera based on the second camera parameters and the preset point of interest. FIG. 5 is a diagram illustrating the preset operation and camera path in the point of interest preset mode. FIG. 5 shows a baseball scene similar to that shown in FIG. 4. As in FIG. 4, a camera position and orientation 70a indicates the initial position and orientation of the virtual camera. The point of interest of the virtual camera at the initial position is located at the position of a fielder 67. Then, when the fielder 67 catches the ball and throws it toward first base, the user presses the preset key 54a.

[0061] At this time, the preset unit 153 acquires preset information corresponding to the preset key 54a from the preset recording unit 154. The preset information corresponding to the preset key 54a indicates a preset gaze point 71b of the virtual camera at a preset position and orientation 71a. In this embodiment, the preset information indicates the position, orientation, and focal length of the virtual camera, and the preset gaze point 71b can be obtained based on this information. Then, the preset unit 153 sets a camera path of the virtual camera based on the initial position and initial orientation of the virtual camera and the preset gaze point 71b of the virtual camera indicated by the preset information corresponding to the preset key 54a.

[0062] The preset unit 153 can set a camera path so that the virtual camera moves from the initial position toward the preset gaze point. For example, the preset unit 153 can set the camera path of the virtual camera as follows.

[0063] In one embodiment, the presetting unit 153 sets a camera path so that the virtual camera moves on a straight line connecting the initial position and the preset point of interest. For example, the presetting unit 153 first calculates a straight line 70b connecting the initial position of the virtual camera indicated in the camera position and orientation 70a and the preset point of interest 71b. The presetting unit 153 can set the camera path so that the virtual camera moves on the straight line 70b.

[0064] Furthermore, the presetting unit 153 can set the camera path so that a point away from the point of gaze becomes the end point of the movement of the virtual camera. For example, the presetting unit 153 can set the camera path so that a point a predetermined distance away from the point of gaze becomes the end point of the movement of the virtual camera. This predetermined distance can be the distance between the preset point of gaze and the position of the virtual camera indicated by the camera parameters of the virtual camera (i.e., the preset position). Specifically, the presetting unit 153 calculates the intersection 72a between the above-mentioned straight line 70b and a preset sphere 71c centered on the preset point of gaze 71b. As already described, the distance between the preset point of gaze and each point on the preset sphere coincides with the distance between the preset position and the preset point of gaze. The intersection 72a calculated in this manner is used as the end point of the camera path. That is, the presetting unit 153 can set the camera path so that the virtual camera moves from the initial position to the intersection 72a.

[0065] 5, the camera path is set so that the virtual camera moves on a straight line 70b from the initial position to the intersection 72a. The preset unit 153 can set the camera path so that the virtual camera moves from the initial position to the intersection 72a at a constant speed.

[0066] Furthermore, the preset unit 153 can set the attitude of the moving virtual camera. The preset unit 153 can set a camera path so that the virtual camera moves while changing its attitude in a direction facing the preset point of interest. In the example shown in FIG. 5, the preset unit 153 can control the attitude of the virtual camera moving along the camera path so that the virtual camera faces the preset point of interest 71b. Furthermore, the preset unit 153 can set the camera path so that the virtual camera faces the preset point of interest 71b at the end point.

[0067] As described above, the preset unit 153 can set a camera path so that the virtual camera moves from its initial position toward the preset point of interest. When the virtual camera moves in this manner, by changing the attitude of the virtual camera in a direction toward the preset point of interest during the movement, the time during which the preset point of interest is included in the angle of view of the virtual camera becomes longer. This makes it less likely that a desired scene will be missed. Also, as described above, the preset unit 153 can set a camera path so that a point away from the point of interest becomes the end point of the movement of the virtual camera. When the virtual camera moves in this manner, by changing the attitude of the virtual camera in a direction toward the preset point of interest during the movement, the time during which the entire subject present at the preset point of interest is included in the angle of view of the virtual camera becomes longer. This makes it less likely that a desired scene will be missed.

[0068] In one embodiment, when moving from the initial position to the end point, the virtual camera can rotate at a constant speed so as to change from the orientation at the initial position to the orientation at the end point. In another embodiment, the virtual camera can rotate at a faster speed during the first half of the movement from the initial position to the end point compared to the second half of the movement. In yet another embodiment, the presetting unit 153 can set a camera path so that the virtual camera faces a preset gaze point during the movement of the virtual camera. Note that, in one embodiment, the virtual camera facing a preset gaze point means that the preset gaze point is located on the optical axis of the virtual camera. In the example of FIG. 5, the presetting unit 153 can set a camera path so that the virtual camera faces the preset gaze point 71b during the movement to the end point. For example, in FIG. 5, the moving virtual cameras 70c and 70d face the preset gaze point 71b. Here, the presetting unit 153 may set a camera path so that the orientation of the virtual camera changes as it moves and faces the preset gaze point 71b during the movement. Furthermore, the preset unit 153 may set the camera path so that the virtual camera starts moving from the initial position after completing the change in the virtual camera's attitude so that it faces the preset point of interest 71b. In this way, during the movement period from the initial position to the end point, the change in the virtual camera's attitude can be completed before the virtual camera reaches the end point. With this configuration, in the example of FIG. 5, the virtual camera moves toward first base and then moves to the end point on the preset sphere. This makes it less likely that a scene at the preset point of interest will be missed. It also makes it easier to create a realistic virtual viewpoint video.

[0069] In the camera position preset mode, the position and orientation of the virtual camera at the end point of the camera path are uniquely determined by preset information, regardless of the initial position of the virtual camera. The camera position preset mode is effective when you want to move the virtual camera to a fixed position to obtain a virtual viewpoint image with a fixed angle of view, such as when shooting from behind the batter. The focus point preset mode is effective when shooting a series of plays, such as when shooting as if chasing the ball, as described with reference to Figure 5.

[0070] On the other hand, in the gaze point preset mode, the preset unit 153 can perform control so that the position and orientation of the virtual camera after the position and orientation are changed are determined based on the second camera parameters and the first camera parameters. For example, the preset unit 153 can set a camera path so that the end point of the movement of the virtual camera along the camera path changes depending on the initial position of the virtual camera and the position of the preset gaze point. Furthermore, the preset unit 153 can set a camera path so that the orientation of the virtual camera at the end point of the camera path changes depending on the initial position of the virtual camera and the position of the preset gaze point. The gaze point preset mode is also effective for shooting subjects such as ball games other than baseball or sports other than ball games. For example, in soccer, the gaze point preset mode is effective when the ball moves a long way, such as when centering the ball in front of the goal.

[0071] The end point of the camera path may be determined on the XY plane. In this case, by projecting the preset position and the preset gaze point onto the XY plane, a preset circle can be defined that is centered on the preset gaze point and passes through the preset position. Also, by projecting the trajectory from the initial position of the virtual camera to the preset position onto the XY plane, the intersection point between the trajectory and the preset circle can be determined. In this case, the XY coordinates of the determined intersection point can be used as the XY coordinates of the end point of the camera path.

[0072] (Processing flow) Next, the flow of processing performed by the virtual camera control device 105 in this embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the operation performed by the virtual camera control device 105. When performing the following processing, a three-dimensional model of the subject corresponding to each time (time code) is generated sequentially. Then, a virtual viewpoint video of the three-dimensional model of the subject, configured from frames corresponding to each time, is displayed. The user can set the virtual camera while playing back the virtual viewpoint video. However, a three-dimensional model of the subject at each time may also be generated in advance.

[0073] In S601, the operation detection unit 151 detects whether or not a virtual camera movement operation has been performed using the operation controller 158a. If the operation detection unit 151 detects an operation, the process proceeds to S610. If not, the process proceeds to S602.

[0074] In S602, the operation detection unit 151 detects whether a preset key of the setting controller 158b has been pressed. If the operation detection unit 151 detects that a preset key has been pressed, the process proceeds to S603. If not, the process returns to S601, and the process according to FIG. 6 is continued.

[0075] In S603, the preset unit 153 determines the current preset mode. In this embodiment, the preset unit 153 determines whether the current preset mode is the gaze point preset mode or the camera position preset mode. If it is determined that the current preset mode is the gaze point preset mode, the process proceeds to S604. If it is determined that the current preset mode is the camera position preset mode, the process proceeds to S605.

[0076] In S604, the preset unit 153 determines a camera path in the gaze point preset mode. The preset unit 153 determines the camera path based on the current position and orientation of the virtual camera and the preset information corresponding to the pressed preset key, using the method described above.

[0077] In S605, the preset unit 153 determines a camera path in the camera position preset mode. The preset unit 153 determines the camera path from the current position of the virtual camera to the preset position indicated by the preset information corresponding to the pressed preset key, using the method described above.

[0078] In S606, the parameter setting unit 152 updates the camera parameters of the virtual camera according to the camera path set in S604 or S605. Here, the parameter setting unit 152 can set the camera parameters of the virtual camera at a specific time (time code). Then, the parameter setting unit 152 transmits the set virtual camera parameters to the video generation unit 143 via the information transmission unit 157.

[0079] In S607, the video generation unit 143 generates a virtual viewpoint video of the subject from the virtual camera based on the position and orientation of the virtual camera at each time and a three-dimensional model of the subject corresponding to each time. For example, the video generation unit 143 can generate a virtual viewpoint video in accordance with the received virtual camera parameters based on the three-dimensional model of the subject stored in the model DB 142. The video generation unit 143 can generate a frame image of the virtual viewpoint video corresponding to a specific time (time code) using the three-dimensional model of the subject at the specific time.

[0080] In S608, the video generation unit 143 causes the video display 160 to display the generated virtual viewpoint video.

[0081] The processes of S606 to S608 are repeated until the virtual camera reaches the end point along the camera path. That is, by repeating the loop of S606 to S608, parameter setting unit 152 can sequentially set camera parameters of the virtual camera at each of a plurality of times. Then, video generation unit 143 can sequentially generate virtual viewpoint images at each of a plurality of times.

[0082] In S609, the preset unit 153 determines whether or not to end the processing. For example, the preset unit 153 can determine to end the processing when a power switch (not shown) on the setting controller 158b is operated or when an end button (not shown) on the operation UI is clicked. If it is determined to end the processing, the flow according to Fig. 6 ends. If it is determined not to end the processing, the process returns to S601, and the process according to Fig. 6 is repeated.

[0083] In S610, parameter setting unit 152 calculates the camera parameters of the virtual camera in accordance with the operation of operation controller 158a detected in S601. Here, parameter setting unit 152 can set the camera parameters of the virtual camera at a specific time (time code). Then, parameter setting unit 152 transmits the set virtual camera parameters to video generation unit 143 via information transmission unit 157.

[0084] In S611, similarly to S607, the video generation unit 143 generates a virtual viewpoint video in accordance with the received virtual camera parameters, based on the three-dimensional model of the subject stored in the model DB 142. The video generation unit 143 can generate a frame image of the virtual viewpoint video corresponding to a specific time (time code) using the three-dimensional model of the subject at the specific time.

[0085] In S612, the video generation unit 143 displays the generated virtual viewpoint video on the video display 160. Thereafter, the process proceeds to S609.

[0086] 6, even if it is determined in S601 and S602 that the controller 158 has not been operated, the video generation unit 143 can generate frame images of the virtual viewpoint video in the same manner as in S611. In this case, the video generation unit 143 can generate frame images of the virtual viewpoint video according to the camera parameters of the current virtual camera using a three-dimensional model of the subject at a new specific time. The video generation unit 143 displays the generated virtual viewpoint video on the video display 160 in the same manner as in S612. In other words, the virtual viewpoint video can be played back without moving the virtual camera.

[0087] As described above, the virtual camera control device according to this embodiment can perform preset operations according to preset information in accordance with the preset mode. Furthermore, in one embodiment, the user can change the preset mode. This configuration allows the user to generate a desired virtual viewpoint video with simple operations.

[0088] (Another example of camera path settings in the gaze preset mode) In the gaze point preset mode, the preset unit 153 may set a camera path different from the camera path described above. For example, the preset unit 153 can set a camera path so that the virtual camera does not move through a three-dimensional model of the subject. As an example, if another subject is on the trajectory of the virtual camera set by the above method, the preset unit 153 can generate a camera path by the following method.

[0089] FIG. 7 is a diagram illustrating another camera path in the focus point preset mode. FIG. 7 shows a baseball scene similar to that shown in FIG. 4. The scene in FIG. 7 differs from FIG. 4 in that a fielder 69, different from fielders 67 and 68, is present on trajectory 70b shown in FIG. 4. In this case, the fielder 69 located on trajectory 70b may be included in the field of view of the virtual camera moving along trajectory 70b while changing its posture to face first base. In this way, if an obstacle or subject exists on trajectory 70b of the camera path, the preset unit 153 can set a camera path that bypasses the obstacle or subject. For example, the preset unit 153 can set a camera path indicated by trajectory 70e. By moving the virtual camera along the camera path indicated by trajectory 70e, it is possible to prevent the area around first base from being obscured by the fielder 69 in the virtual viewpoint video.

[0090] In this case, the attitude of the virtual camera can be changed so that it faces the preset point of interest immediately after the virtual camera starts moving along the camera path. Furthermore, as shown by camera position and attitude 70f, the attitude of the virtual camera can be adjusted so that it faces the preset point of interest 71b while the virtual camera is moving along the camera path indicated by trajectory 70e. In this example, the end point of the camera path is the intersection of trajectory 70e and preset sphere 71c.

[0091] In one embodiment, the preset unit 153 can determine whether a three-dimensional model of a subject exists within a predetermined range from a straight line connecting the initial position and the preset gaze point. For example, the preset unit 153 can determine whether an obstacle or subject that needs to be detoured exists based on whether a three-dimensional model of a subject in the foreground is located within a predetermined range from the trajectory 70b in the virtual space in which the three-dimensional model is located. This predetermined range can be set, for example, between the initial position of the virtual camera indicated by the camera position and orientation 70a and the preset sphere 71c.

[0092] If a three-dimensional model of the subject does not exist within the predetermined range, the preset unit 153 can set a camera path such as trajectory 70b so that the virtual camera moves on a straight line connecting the initial position and the point of interest. On the other hand, if a three-dimensional model of the subject exists within the predetermined range, the preset unit 153 can set a camera path so that the virtual camera moves on a path connecting the initial position and the preset point of interest, bypassing the three-dimensional model of the subject. For example, if a three-dimensional model exists within a predetermined range from trajectory 70b, the preset unit 153 can set a curve that does not pass through a certain size of area centered on the three-dimensional model. This curve is a curve extending from the current position of the virtual camera to the preset point of interest and may be, for example, a circular arc. In the example of FIG. 7, the preset unit 153 generates trajectory 70e, which avoids the obstacle, fielder 69, by shifting trajectory 70b in the X and Y directions. However, the preset unit 153 may also generate a camera path in which the virtual camera moves in the Z direction to avoid the obstacle.

[0093] Regardless of whether the virtual camera moves along a straight line or a curve, the preset unit 153 can set the camera path so that the end point of the movement of the virtual camera along the camera path is closer to the initial position of the virtual camera than the preset point of interest. When the virtual camera moves in this way, by changing the attitude of the virtual camera so that it faces the preset point of interest while moving, the time during which the preset point of interest is included in the field of view of the virtual camera becomes longer. This makes it less likely that you will miss capturing a desired scene.

[0094] (Other preset modes) The preset modes selectable by the user are not limited to the gaze point preset mode and the camera position preset mode described above. Below, a posture change preset mode and a position change preset mode will be described as other examples of the preset modes selectable by the user.

[0095] Below, with reference to FIGS. 8(A) to 8(C), a method for setting a camera path in each preset mode will be described. FIGS. 8(A) to 8(C) show a scene in which an outfielder 81 catches a ball 83a and throws it toward home base, and a runner 82 also runs toward home base. The ball 83a arrives at home base via positions 83b and 83c. Balls 83a to 83c each show the same ball at a different time. In this example, the preset position and orientation indicated by the camera position and orientation 75a is behind home base. The preset point of interest 75b is set near home base. Meanwhile, the initial position and orientation of the virtual camera is indicated by the camera position and orientation 76a. The point of interest in the camera position and orientation 76a is located at the outfielder 81. When the outfielder 81 throws the ball 83 toward home base, the preset unit 153 sets the camera path in response to the user pressing the preset key 54.

[0096] Setting of the camera path in the posture change preset mode will be described with reference to Fig. 8(A). In the posture change preset mode, the preset unit 153 sets the camera path so that the posture of the virtual camera is changed in a direction facing the preset gaze point without moving from the initial position.

[0097] In FIG. 8(A), the initial position of the virtual camera is indicated by camera position and orientation 76a. In the orientation change preset mode, the preset unit 153 sets a camera path that changes only the orientation of the virtual camera so that it faces the preset gaze point 75b without changing the position of the virtual camera. The end point of the camera path in FIG. 8(A) is indicated by camera position and orientation 76b. In FIG. 8(A), for the sake of explanation, the position indicated by camera position and orientation 76a and the position indicated by camera position and orientation 76b are slightly offset from each other. In reality, however, only the orientation and gaze point of the virtual camera are changed without changing the position of the virtual camera. By moving the virtual camera according to such a camera path, it is possible to generate a virtual viewpoint video of a play in which the ball arrives at home base from the viewpoint of an outfielder. Note that in the orientation change preset mode, the camera parameters of the virtual camera may be controlled so that the focal length gradually increases.

[0098] Next, setting of the camera path in the position change preset mode will be described with reference to Fig. 8(B). In Fig. 8(B) as well, the initial position of the virtual camera is indicated by the camera position and attitude 76a. In the position change preset mode, the preset unit 153 sets the camera path so that the virtual camera does not change its attitude from the initial attitude and faces the gaze point at the end point of the movement of the virtual camera along the camera path.

[0099] For example, the preset unit 153 can set a camera path so that the virtual camera moves from the position indicated by the camera position and orientation 76a to the position indicated by the camera position and orientation 76d without changing the attitude of the virtual camera. For example, the preset unit 153 can set a camera path so that the virtual camera moves at a constant speed along a trajectory 76e represented as a line segment from the position indicated by the camera position and orientation 76a to the position indicated by the camera position and orientation 76d. As indicated by the camera position and orientation 76c, the virtual camera moves without changing its attitude. The camera position and orientation 76d indicating the end point of the camera path is set so that the attitude of the virtual camera matches the initial attitude and the point of gaze of the virtual camera is located at the preset point of gaze 75b. In other words, the position indicated by the camera position and orientation 76d is located on the preset sphere 75c.

[0100] FIG. 8(C) shows a method for setting a camera path in the gaze point preset mode. Setting of the camera path in the gaze point preset mode is performed as already described. In the example shown in FIG. 8(C), the position and orientation of the virtual camera at the end point of the camera path is represented by camera position and orientation 76f. In this example, the virtual camera can move at a constant speed along a trajectory 76g from the position indicated by camera position and orientation 76a to the position indicated by camera position and orientation 76f. Meanwhile, the orientation of the virtual camera can be changed so that it faces the preset gaze point 75b as soon as the preset operation starts.

[0101] In this way, the preset unit 153 may set a camera path that changes only the camera attitude or only the camera position based on the preset information. By using such a preset mode, the user can create a variety of free viewpoint videos with simple operations.

[0102] After the preset operation, the parameter setting unit 152 can control the camera parameters of the virtual camera in accordance with a user input. In this case, constraints may be imposed on the position or attitude of the virtual camera. For example, the operation detection unit 151 may acquire a user input to move the virtual camera on a line connecting the initial position of the virtual camera and the preset point of interest while fixing the attitude of the virtual camera while the virtual camera is facing the preset point of interest. For example, after changing the attitude of the virtual camera in the attitude change preset mode so that it faces the preset point of interest, the virtual camera may be moved closer to the preset point of interest in accordance with a user instruction via the operation controller 158a. Furthermore, while the virtual camera is facing the preset point of interest, the parameter setting unit 152 may automatically change the attitude of the virtual camera so that the virtual camera always faces the preset point of interest, even if the virtual camera is moved by a user operation.

[0103] As already described, the user can switch preset modes. To switch preset modes, the user can operate the mode key 53. On the other hand, to start a preset operation, the user can press the mode key 53 and the preset key 54 in combination. As described above, a preset mode is set for each of the multiple mode keys 53. Furthermore, camera parameters of a virtual camera are set for each of the multiple preset keys 54. The user can press one of the multiple mode keys 53 corresponding to the preset mode to be selected and one of the multiple preset keys 54 corresponding to the camera parameters to be selected in combination. At this time, the preset unit 153 can set a camera path based on the preset mode corresponding to the pressed preset mode key and the camera parameters of the virtual camera corresponding to the pressed preset key.

[0104] For example, a camera position and orientation 75a having a point of interest near home base may be registered in the preset key 54a. In this case, the preset position and orientation 75a, the point of interest 75b, and the radius of the preset sphere 75c can be registered as preset information corresponding to the preset key 54a. Also, a camera position preset mode can be registered in the mode key 53a. An orientation change preset mode can be registered in the mode key 53b. A position change preset mode can be registered in the mode key 53c. By simultaneously pressing the mode key 53 and the preset key 54, the user can start a preset operation corresponding to each preset mode and preset information.

[0105] For example, when the user presses the preset key 54a together with the mode key 53b, the preset unit 153 sets the camera path according to the posture change preset mode as described with reference to FIG. 8(A). When the user presses the preset key 54a together with the mode key 53c, the preset unit 153 sets the camera path according to the position change preset mode as described with reference to FIG. 8(B). When the user presses the preset key 54a together with the mode keys 53b and 53c, the preset unit 153 sets the camera path according to the gaze point preset mode as described with reference to FIG. 8(C). Note that the gaze point preset mode may be registered in the mode key 53d.

[0106] (Variation) In the above-described embodiment, the camera parameters of the virtual camera include information on the position, orientation, and focal length of the virtual camera. However, for example, once the position, orientation, and focal length of the virtual camera are determined, the gaze point and the distance from the virtual camera to the gaze point can be calculated using the above-described method. Similarly, the orientation, focal length, and the distance from the virtual camera position to the gaze point can be calculated from the virtual camera position and gaze point. In other words, once some parameters are determined, other parameters can be calculated. In other words, each parameter has a complementary relationship. Therefore, the camera parameters of the virtual camera may include information on the position and gaze point of the virtual camera. Alternatively, the camera parameters of the virtual camera may include information on the position, orientation, and focal length of the virtual camera. In this way, the camera parameters of the virtual camera include one or more of the virtual camera position, the virtual camera orientation, the virtual camera focal length, the virtual camera gaze point, and the distance from the virtual camera position to the gaze point.

[0107] In one embodiment, the camera parameters of the virtual camera are information that allows calculation of at least the position, orientation, and point of interest of the virtual camera. Such information can be said to be information indicating the position, orientation, and point of interest of the virtual camera. For example, the information indicating the position and point of interest of the virtual camera can be said to be information indicating the position, orientation, and point of interest of the virtual camera.

[0108] In the above-described embodiment, the point of gaze of the virtual camera is information that can be calculated based on the position and orientation of the virtual camera and the focal length of the virtual camera. However, the method for determining the point of gaze is not limited to this method. For example, the point of gaze may be set regardless of the focal length. For example, the user may be able to set the point of gaze of the virtual camera. Furthermore, the point of gaze of the virtual camera may be a point on the optical axis of the virtual camera that is a predetermined distance away from the virtual camera. Furthermore, the point of gaze of the virtual camera may be located at the position of a three-dimensional model of a subject (e.g., a base) on the optical axis of the virtual camera. In such a case, the point of gaze of the virtual camera can be calculated based on the position and orientation of the virtual camera. That is, in one embodiment, the camera parameters of the virtual camera may be information that indicates the position and orientation of the virtual camera. However, the camera parameters of the virtual camera may include other information.

[0109] Similarly, the preset information may be information similar to the camera parameters of the virtual camera. The preset information may indicate the orientation of the virtual camera and the position of the point of interest corresponding to the virtual camera. In the above-described embodiment, the preset information includes the position and orientation of the preset virtual camera and the radius of the preset sphere (i.e., the distance from the preset position to the preset point of interest). On the other hand, the preset information may also include information indicating the orientation of the virtual camera and information indicating the position of the point of interest. For example, the preset information may include information indicating the orientation of the virtual camera, information indicating the position of the point of interest, and information indicating the distance between the virtual camera and the point of interest. The preset information may also include information indicating the position of the preset virtual camera, the position of the preset point of interest, and the radius of the preset sphere. Here, the information indicating the radius of the preset sphere may be the focal length or may be set regardless of the focal length. The preset information may also indicate only the position and orientation of the preset virtual camera. Based on this preset information, the position and orientation of the preset virtual camera and the point of interest can be calculated as described above. On the other hand, the preset information may also include information on other camera parameters.

[0110] In the above-described embodiment, the virtual camera control device 105 mainly controls the position and orientation of the virtual camera at each time. However, the virtual camera control device 105 may generate a camera path so as to control camera parameters other than the position and orientation of the virtual camera.

[0111] In the above-described embodiment, the user can select a desired preset mode from a plurality of preset modes. However, it is not essential that the user can select a preset mode. For example, the virtual camera control device 105 may perform a preset operation according to only one of the gaze point preset mode, the position change preset mode, or the attitude change preset mode.

[0112] Preset operations in the gaze point preset mode, position change preset mode, and attitude change preset mode can be performed based on the initial position and initial attitude of the virtual viewpoint and the preset gaze point. These preset operations do not require the use of a preset position and attitude. Therefore, only preset gaze point information may be registered as preset information in a preset key. For example, a user can register only preset information for the gaze point preset mode. In this case, the user can select the gaze point to be registered on the UI of the operation display 159, for example, and further input the radius of the preset sphere. In this case, only the gaze point and the radius of the preset sphere are registered as preset information. However, in the gaze point preset mode, the preset operation can be performed using only this information.

[0113] Each device shown in FIG. 2(B) can be realized using a computer. Examples of computers include a general-purpose desktop computer, a laptop computer, a tablet PC, or a smartphone. For example, the functions of each processing unit of each device shown in FIG. 2(B) can be realized by a computer. However, at least some of the processing units may be realized by dedicated hardware. Furthermore, each device may be configured by multiple information processing devices connected via a network, for example. For example, the functions of each image processing device may be provided as a cloud service. Furthermore, one computer may realize the functions of two or more devices shown in FIG. 2(B).

[0114] FIG. 9 is a diagram showing the basic configuration of a computer. In FIG. 9, a processor 910 is, for example, a CPU, and controls the operation of the entire computer. A memory 920 is, for example, a RAM, and temporarily stores programs, data, etc. A computer-readable storage medium 930 is, for example, a hard disk or a CD-ROM, and stores programs, data, etc. long-term. In this embodiment, a program that realizes the function of each unit, which is stored in the storage medium 930, is read into the memory 920. Then, the processor 910 operates in accordance with the program on the memory 920, thereby realizing the function of each unit.

[0115] 9, input interface 940 is an interface for acquiring information from an external device. Output interface 950 is an interface for outputting information to an external device. Bus 960 connects the above-mentioned components and enables data exchange.

[0116] (Other Examples) The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0117] The disclosure of this specification includes the following information processing device, image processing system, information processing method, and program. (Item 1) An information processing device that controls a virtual camera arranged in a virtual space to generate a virtual viewpoint video, a storage means for storing first camera parameters that are set in advance and indicate the attitude of a virtual camera and the position of a point of interest corresponding to the virtual camera; an acquisition means for acquiring second camera parameters that are different from the first camera parameters and indicate the position and orientation of a virtual camera; a control means for performing control to change the position and attitude of the virtual camera indicated by the second camera parameters based on the second camera parameters and the first camera parameters, The position and orientation of the virtual camera after the change are determined based on the second camera parameters and the first camera parameters. An information processing device comprising: (Item 2) The information processing device described in item 1, characterized in that the gaze point is a point along the line of sight from the virtual camera positioned according to the first camera parameters by a distance corresponding to the focal length indicated by the first camera parameters. (Item 3) 3. The information processing device according to any one of items 1 to 2, wherein the control means performs the control so that the virtual camera moves while changing its posture in a direction facing the point of interest. (Item 4) 4. The information processing device according to any one of items 1 to 3, wherein the control means performs the control so that the virtual camera faces the point of interest while the virtual camera is moving. (Item 5) 5. The information processing device according to any one of items 1 to 4, characterized in that the control means performs the control so that the virtual camera moves from a position indicated by the second camera parameter toward the point of interest. (Item 6) 6. The information processing device according to item 5, wherein the control means performs the control so that the virtual camera moves on a straight line connecting the position indicated by the second camera parameter and the point of interest. (Item 7) the control means determines whether or not a three-dimensional model of the subject exists within a predetermined range from a straight line connecting the position indicated by the second camera parameter and the point of interest; When a three-dimensional model of the subject does not exist within the predetermined range, the control is performed so that the virtual camera moves on a straight line connecting the position indicated by the second camera parameter and the point of gaze; Item 6. The information processing device according to item 5, characterized in that, when a three-dimensional model of the subject is present within the specified range, the control is performed so that the virtual camera bypasses the three-dimensional model of the subject and moves on a path connecting the position indicated by the second camera parameter and the point of interest. (Item 8) 8. The information processing device according to any one of items 5 to 7, wherein the control means performs the control so that a point away from the gaze point becomes an end point of the movement of the virtual camera. (Item 9) The information processing device described in any one of items 5 to 8, characterized in that the control means performs the control so that a point a predetermined distance away from the point of gaze becomes the end point of the movement of the virtual camera, and the predetermined distance is the distance between the position of the virtual camera indicated by the first camera parameter and the point of gaze. (Item 10) The information processing device described in any one of items 1 to 9, characterized in that the control means performs the control so that the end point of the movement of the virtual camera changes depending on the position indicated by the second camera parameter and the position of the gaze point. (Item 11) The information processing device described in any one of items 1 to 10, characterized in that the control means performs the control so that the end point of the movement of the virtual camera is closer to the position indicated by the second camera parameter than the point of interest. (Item 12) The information processing device described in any one of items 1 to 11, characterized in that the control means performs the control so that the virtual camera changes its posture in a direction facing the gaze point without moving from the position indicated by the second camera parameter. (Item 13) Item 13. The information processing device described in item 12, further comprising a means for acquiring user input that moves the virtual camera on a straight line connecting the position indicated by the second camera parameter and the point of interest while fixing the attitude of the virtual camera after changing the attitude of the virtual camera in a direction facing the point of interest. (Item 14) The information processing device described in any one of items 1 to 13, characterized in that the control means performs the control so that the virtual camera does not change its orientation from the orientation indicated by the second camera parameter, and so that the virtual camera is facing the gaze point at the end point of the movement of the virtual camera. (Item 15) means for receiving a user input indicating a selected preset mode from a plurality of preset modes; the control means performs the control in accordance with a preset mode indicated by the user input; The plurality of preset modes include: a first preset mode in which the control is performed based on the position and orientation indicated by the second camera parameters and the gaze point of the virtual camera indicated by the first camera parameters; a second preset mode in which the control is performed so that an end point of the movement of the virtual camera is the position of the virtual camera indicated by the first camera parameter; 15. The information processing device according to any one of items 1 to 14, comprising: (Item 16) the user input is performed by pressing one of a plurality of mode keys, each of which has a preset mode set thereto, in combination with one of a plurality of preset keys, each of which has a camera parameter of the virtual camera set thereto; Item 16. The information processing device described in item 15, characterized in that the control means performs the control based on a preset mode corresponding to the pressed mode key and camera parameters of the virtual camera corresponding to the pressed preset key. (Item 17) An image processing system including the information processing device according to any one of items 1 to 16, a parameter setting means for setting the position and attitude of the virtual camera at each time in accordance with the control; an image generating means for generating a virtual viewpoint image of the subject from the virtual camera based on the position and orientation of the virtual camera at each time and a three-dimensional model of the subject corresponding to each time; An image processing system comprising: (Item 18) Multiple cameras and a model generation means for generating a three-dimensional model of the subject based on images captured by the plurality of cameras; Item 18. The image processing system according to item 17, further comprising: (Item 19) An information processing method performed by an information processing device that controls a virtual camera arranged in a virtual space to generate a virtual viewpoint video, acquiring, from a storage means, first camera parameters that are set in advance and indicate the attitude of a virtual camera and the position of a gaze point corresponding to the virtual camera; acquiring second camera parameters that are different from the first camera parameters and indicate a position and orientation of a virtual camera; and performing a control step of changing the position and attitude of the virtual camera indicated by the second camera parameters based on the second camera parameters and the first camera parameters, An information processing method, characterized in that the position and orientation of the virtual camera after the change are determined based on the second camera parameters and the first camera parameters. (Item 20) 17. A program for causing a computer to function as the information processing device according to any one of items 1 to 16.

[0118] The present disclosure is not limited to the above-described embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present disclosure. Accordingly, the following claims are appended to apprise the public of the scope of the present disclosure. [Explanation of symbols]

[0119] 101: Image processing system, 102: Camera, 103: Camera control device, 104: Image processing server, 105: Virtual camera control device, 151: Operation detection unit, 152: Parameter setting unit, 153: Preset unit, 154: Preset recording unit, 155: Mode setting unit, 156: UI generation unit, 157: Information transmission unit

Claims

1. An information processing device that controls a virtual camera arranged in a virtual space to generate a virtual viewpoint video, a storage means for storing first camera parameters that are set in advance and indicate the attitude of a virtual camera and the position of a gaze point corresponding to the virtual camera; an acquisition means for acquiring second camera parameters that are different from the first camera parameters and indicate a position and orientation of a virtual camera; a control means for performing control to change the position and attitude of the virtual camera indicated by the second camera parameters based on the second camera parameters and the first camera parameters, The position and orientation of the virtual camera after the change are determined based on the second camera parameters and the first camera parameters. An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the gaze point is a point that is separated from the virtual camera positioned in accordance with the first camera parameters along the line of sight by a distance corresponding to a focal length indicated by the first camera parameters.

3. The information processing apparatus according to claim 1 , wherein the control means performs the control so that the virtual camera moves while changing its attitude in a direction facing the point of interest.

4. 2. The information processing apparatus according to claim 1, wherein the control means performs the control so that the virtual camera faces the point of interest while the virtual camera is moving.

5. The information processing apparatus according to claim 1 , wherein the control means performs the control so that the virtual camera moves from the position indicated by the second camera parameter toward the point of interest.

6. 6. The information processing apparatus according to claim 5, wherein the control means performs the control so that the virtual camera moves on a straight line connecting the position indicated by the second camera parameter and the point of interest.

7. the control means determines whether or not a three-dimensional model of the subject exists within a predetermined range from a straight line connecting the position indicated by the second camera parameter and the point of interest; When a three-dimensional model of the subject does not exist within the predetermined range, the control is performed so that the virtual camera moves on a straight line connecting the position indicated by the second camera parameter and the point of interest; 6. The information processing device according to claim 5, characterized in that, when a three-dimensional model of the subject is present within the specified range, the control is performed so that the virtual camera detours around the three-dimensional model of the subject and moves on a path connecting the position indicated by the second camera parameter and the point of gaze.

8. 6. The information processing apparatus according to claim 5, wherein said control means performs said control so that a point away from said point of interest becomes an end point of movement of said virtual camera.

9. 6. The information processing device according to claim 5, wherein the control means performs the control so that a point a predetermined distance away from the point of gaze becomes the end point of movement of the virtual camera, and the predetermined distance is the distance between the position of the virtual camera indicated by the first camera parameter and the point of gaze.

10. 2. The information processing apparatus according to claim 1, wherein the control means performs the control so that an end point of movement of the virtual camera changes depending on the position indicated by the second camera parameter and the position of the point of interest.

11. 2. The information processing apparatus according to claim 1, wherein the control means performs the control so that an end point of the movement of the virtual camera is closer to the position indicated by the second camera parameter than the point of gaze.

12. 2. The information processing apparatus according to claim 1, wherein the control means performs the control so that the virtual camera changes its posture in a direction facing the point of interest without moving from the position indicated by the second camera parameter.

13. 13. The information processing device according to claim 12, further comprising means for acquiring a user input for moving the virtual camera on a straight line connecting the position indicated by the second camera parameter and the point of interest while fixing the attitude of the virtual camera after changing the attitude of the virtual camera in a direction facing the point of interest.

14. 2. The information processing device according to claim 1, wherein the control means performs the control so that the virtual camera does not change its orientation from the orientation indicated by the second camera parameter, and so that the virtual camera faces the gaze point at the end point of the movement of the virtual camera.

15. means for receiving a user input indicating a selected preset mode from a plurality of preset modes; the control means performs the control in accordance with a preset mode indicated by the user input; The plurality of preset modes include: a first preset mode in which the control is performed based on the position and orientation indicated by the second camera parameters and the gaze point of the virtual camera indicated by the first camera parameters; a second preset mode in which the control is performed so that an end point of the movement of the virtual camera is the position of the virtual camera indicated by the first camera parameter; The information processing device according to claim 1 , further comprising:

16. the user input is performed by pressing one of a plurality of mode keys, each of which has a preset mode set thereto, in combination with one of a plurality of preset keys, each of which has a camera parameter of the virtual camera set thereto; 16. The information processing apparatus according to claim 15, wherein the control means performs the control based on a preset mode corresponding to the pressed mode key and camera parameters of the virtual camera corresponding to the pressed preset key.

17. An image processing system including the information processing device according to claim 1, a parameter setting means for setting the position and attitude of the virtual camera at each time in accordance with the control; an image generating means for generating a virtual viewpoint image of the subject from the virtual camera based on the position and orientation of the virtual camera at each time and a three-dimensional model of the subject corresponding to each time; An image processing system comprising:

18. Multiple cameras and a model generation means for generating a three-dimensional model of the subject based on images captured by the plurality of cameras; 18. The image processing system of claim 17, further comprising:

19. An information processing method performed by an information processing device that controls a virtual camera arranged in a virtual space to generate a virtual viewpoint video, acquiring, from a storage means, first camera parameters that are set in advance and indicate the attitude of a virtual camera and the position of a gaze point corresponding to the virtual camera; acquiring second camera parameters that are different from the first camera parameters and indicate a position and orientation of a virtual camera; and performing control to change the position and attitude of the virtual camera indicated by the second camera parameters based on the second camera parameters and the first camera parameters, An information processing method, characterized in that the position and orientation of the virtual camera after the change are determined based on the second camera parameters and the first camera parameters.

20. A program for causing a computer to function as the information processing device according to any one of claims 1 to 16.

Images