Information processing apparatus, information processing method, and program

Abstract

To solve the problem in which a user cannot respond to the motion of a subject with a preset change speed of a virtual camera.SOLUTION: A virtual camera control apparatus 110 is configured to :acquire input information corresponding to an input corresponding to an operation performed on a first member, for changing virtual camera parameters including a parameter corresponding to a position of a virtual camera or a position of a point-of-gaze corresponding to the virtual camera, a parameter corresponding to an orientation of the virtual camera, and a parameter corresponding to an angle of field of the virtual camera; acquire a first adjustment parameter specified based on an operation performed on a second member; acquire a second adjustment parameter changed depending on the amount of operation of a third member; and determine the amount of change of each of the virtual camera parameters based on the input, the first adjustment parameter, and the second adjustment parameter.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a technology for generating virtual viewpoint images. [Background technology]

[0002] A technology that generates a virtual viewpoint image that represents a view from an arbitrary viewpoint (virtual camera) in a virtual space by installing multiple imaging devices at different positions and synchronously capturing images from multiple viewpoints, using the multiple captured images obtained by the capturing, is attracting attention. A user who operates a virtual camera operates the virtual camera with reference to the generated virtual viewpoint image. Therefore, when a user wants to generate a virtual viewpoint image for live distribution, the user considers the composition in the virtual viewpoint image of how to capture the subject, predicts the movement of the subject in advance, and controls the position, direction, and angle of view of the virtual camera. At this time, in order to facilitate the operation of the virtual camera, there is a technology that changes the change speed per frame when controlling the position and direction of the virtual camera according to the user's preference. Furthermore, Patent Document 1 describes that the change speed of the virtual camera is set to change in advance depending on whether the generated virtual viewpoint image is on air or not. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-190917 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when a virtual viewpoint image is generated using a plurality of captured images of a subject, the movement of the subject may be unpredictable, and the preset virtual camera change speed may not be able to handle the movement. For example, when generating an image of gradually approaching a subject, if the subject suddenly moves significantly, the preset virtual camera change speed may not be able to handle the movement, and an image showing the subject in an appropriate position may not be generated. In addition, in sports such as baseball and soccer, where the moving speeds of subjects such as players and balls are significantly different, when the virtual camera change speed is preset, there is a risk that the virtual camera change speed suitable for both subjects may not be set.

[0005] Therefore, an object of the present disclosure is to facilitate operation of a virtual camera in response to the movement of a subject. [Means for solving the problem]

[0006] In order to achieve the above object, the image processing device of the present disclosure has the following configuration: That is, a first acquisition means for acquiring input information corresponding to an input corresponding to an operation on a first member, the input changing virtual camera parameters including a parameter corresponding to a position of a virtual camera or a position of a gaze point corresponding to the virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by capturing an object with a plurality of imaging devices, a parameter corresponding to an orientation of the virtual camera, and a parameter corresponding to an angle of view of the virtual camera; a second acquisition means for acquiring a first adjustment parameter that adjusts a change amount of the virtual camera parameter corresponding to an operation amount of the operation, the first adjustment parameter being specified based on an operation on a second member, and that retains the first adjustment parameter acquired in a previous input when there is no operation on the second member; a third acquisition means for adjusting a change amount of the virtual camera parameter corresponding to an operation amount of the operation, and acquiring a second adjustment parameter that is specified based on an operation on a third member, and that is changed to an initial value when there is no operation on the third member; a determination means for determining a change amount of the virtual camera parameter based on the input information, the first adjustment parameter, and the second adjustment parameter; has. Effect of the Invention

[0007] According to the present disclosure, it is possible to easily operate a virtual camera in response to the movement of a subject. [Brief description of the drawings]

[0008] [Figure 1] 1 is a system configuration diagram of an image processing system 100 according to a first embodiment. [Diagram 2] 1 is a hardware configuration diagram of a virtual camera control device 110 according to the first embodiment. [Diagram 3] 4 is a diagram for explaining the relationship between gaze point coordinates and a virtual camera position according to the first embodiment. FIG. [Figure 4] 4 is a flowchart of a process for calculating virtual camera parameters of the virtual camera control device 110 according to the first embodiment. [Diagram 5] FIG. 4 is a diagram showing an example of a GUI that displays a value of a controller according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of a joystick according to the first embodiment. [Figure 7] FIG. 2 is a diagram showing an example of a foot pedal according to the first embodiment. [Figure 8] FIG. 2 is a diagram showing an example of a controller having a knob according to the first embodiment. [Figure 9] FIG. 11 is a system configuration diagram of an image processing system 100 according to a second embodiment. [Figure 10] 11 is a flowchart of a process for calculating virtual camera parameters of the virtual camera control device 110 according to the second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a GUI for specifying virtual camera parameters that reflect the accelerator and brake in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the present disclosure, and all of the combinations of features described in the embodiments are not necessarily essential to the solution of the present disclosure.

[0010] <Embodiment 1> In this embodiment, a parameter for adjusting the change speed of a camera parameter of a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of images captured by a plurality of imaging devices will be described. The parameters for adjusting the change speed include a first adjustment parameter that is set in advance, and a second adjustment parameter that is changed as needed while the user is operating the virtual camera. By combining the above two adjustment parameters, it is possible to easily operate the virtual camera in response to the movement of the subject.

[0011] FIG. 1 is a system configuration diagram of an image processing system 100 according to the first embodiment of the present disclosure. The image processing system 100 includes a camera group 101, a 3D model generating device 102, a 3D model storage device 103, a virtual viewpoint image generating device 104, an external device 105, a virtual camera control device 110, an operation device 120, and a display device 106.

[0012] The image processing system 100 is a system that generates a virtual viewpoint image representing a scene from a specified virtual viewpoint based on a plurality of images based on imaging by a plurality of imaging devices and a specified virtual viewpoint. The virtual viewpoint image in this embodiment is also called a free viewpoint video, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, and for example, an image corresponding to a viewpoint selected by the user from a plurality of candidates is also included in the virtual viewpoint image. In addition, in this embodiment, the case where the virtual viewpoint is specified by a user operation will be mainly described, but the virtual viewpoint may be automatically specified based on the result of image analysis, etc. In addition, in this embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image.

[0013] The viewpoint information used to generate the virtual viewpoint image is information indicating the position and orientation (line of sight direction) of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including a parameter indicating the three-dimensional position of the virtual viewpoint and a parameter indicating the orientation of the virtual viewpoint in the pan, tilt, and roll directions. Note that the content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter indicating the size of the visual field (angle of view) of the virtual viewpoint. Furthermore, the viewpoint information may have a plurality of parameter sets. For example, the viewpoint information may have a plurality of parameter sets corresponding to a plurality of frames constituting a video of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of a plurality of consecutive time points.

[0014] The virtual camera is a virtual camera different from the multiple imaging devices actually installed around the imaging area, and is a concept for conveniently explaining the virtual viewpoint related to the generation of the virtual viewpoint image. That is, the virtual viewpoint image can be considered to be an image captured from a virtual viewpoint set in a virtual space associated with the imaging area. The position and orientation of the viewpoint in the virtual imaging can be expressed as the position and orientation of the virtual camera. In other words, the virtual viewpoint image can be said to be an image simulating an image captured by a camera when it is assumed that a camera exists at the position of the virtual viewpoint set in the space. In addition, in this embodiment, the content of the transition of the virtual viewpoint over time is expressed as a virtual camera path. However, it is not necessary to use the concept of a virtual camera to realize the configuration of this embodiment. That is, it is sufficient that at least information representing a specific position and information representing a direction in the space are set, and a virtual viewpoint image is generated according to the set information.

[0015] The camera group 101 performs synchronized shooting of all the cameras in order to generate a 3D model of the subject. In this embodiment, the camera group 101 is a plurality of imaging devices that capture images of an imaging area from a plurality of directions. The imaging area is, for example, a stadium where sports such as soccer and karate are held, or a stage where a concert or a play is held. The plurality of imaging devices are installed at different positions so as to surround such an imaging area, and capture images in synchronization. Note that the plurality of imaging devices do not need to be installed all around the imaging area, and may be installed only in a portion of the periphery of the imaging area depending on restrictions on the installation location, etc. The number of imaging devices is not limited to the example shown in the figure, and for example, when the imaging area is a soccer stadium, about 30 imaging devices may be installed around the stadium. Also, imaging devices with different functions, such as a telephoto camera and a wide-angle camera, may be installed.

[0016] In this embodiment, the multiple imaging devices are cameras each having an independent housing and capable of capturing images from a single viewpoint. However, this is not limited to this, and two or more imaging devices may be configured in the same housing. For example, a single camera having multiple lens groups and multiple sensors and capable of capturing images from multiple viewpoints may be installed as the multiple imaging devices.

[0017] The 3D model generating device 102 acquires images captured by the camera group 101 and generates a 3D model. The 3D model can be generated, for example, by a volume intersection method using silhouette information of a foreground object, or by multi-view stereo using triangulation. The methods of generating 3D models using the volume intersection method and multi-view stereo are well known, and therefore will not be described here.

[0018] The 3D model storage device 103 stores the 3D model generated by the 3D model generation device 102. In this embodiment, the 3D model storage device 103 is a server constructed by a dedicated PC, but is not limited to this and may be a virtual server constructed on a cloud.

[0019] The virtual viewpoint image generating device 104 acquires a 3D model from the 3D model storage device 103 based on the time code acquired from the time code providing unit 121. Furthermore, based on the virtual camera parameters acquired from the virtual camera parameter providing unit 118, the virtual viewpoint image generating device 104 generates a virtual viewpoint image so that the image is an image of the acquired 3D model captured by a virtual camera. Furthermore, the generated virtual viewpoint image is output to the external device 105 and the display device 106.

[0020] The virtual viewpoint image is generated, for example, by the following method. First, a plurality of images (multiple viewpoint images) are acquired by capturing images from different directions using a plurality of imaging devices. Next, a foreground image in which a foreground region corresponding to a predetermined object such as a person or a ball is extracted, and a background image in which a background region other than the foreground region is extracted are acquired from the multiple viewpoint images. In addition, a foreground model representing the three-dimensional shape of the predetermined object and texture data for coloring the foreground model are generated based on the foreground image, and texture data for coloring a background model representing the three-dimensional shape of the background such as a stadium is generated based on the background image. Then, the texture data is mapped onto the foreground model and the background model, and rendering is performed according to the virtual viewpoint indicated by the viewpoint information, thereby generating the virtual viewpoint image. However, the method of generating the virtual viewpoint image is not limited to this, and various methods can be used, such as a method of generating the virtual viewpoint image by projective transformation of the captured image without using a three-dimensional model.

[0021] A foreground image is an image in which an object area (foreground area) is extracted from an image captured by an imaging device. An object extracted as a foreground area is a dynamic object (moving body) that moves (its absolute position and shape can change) when images are captured from the same direction in a time series. An object can be, for example, a player or a referee on the field where a sport is being played, a ball in a ball game, or a singer, musician, performer, or presenter in a concert or entertainment event.

[0022] A background image is an image of at least a region (background region) different from the foreground object. Specifically, a background image is an image in a state where the foreground object has been removed from the captured image. The background refers to an imaged object that remains stationary or nearly stationary when images are captured from the same direction in chronological order. Such imaged objects include, for example, a stage for a concert or the like, a stadium where an event such as a competition is held, a structure such as a goal used in a ball game, a field, and the like. However, the background is at least a region different from the foreground object, and the imaged object may include other objects in addition to the object and the background.

[0023] The external device 105 receives the virtual viewpoint image generated by the virtual viewpoint image generating device 104 . The external device 105 is, for example, a display or an external server.

[0024] The display device 106 displays the virtual viewpoint image generated by the virtual viewpoint image generating device 104. The user who operates the virtual camera operates the virtual camera with reference to the virtual viewpoint image displayed on the display device 106. An auxiliary image indicating the camera parameters of the virtual camera generated by the auxiliary image generating unit 122 is displayed. It is assumed that the virtual viewpoint image and the auxiliary image are displayed side by side, but this is not limited to this. For example, an icon indicating the direction of the virtual camera in the virtual space may be displayed so as to be superimposed on the virtual viewpoint image. In this embodiment, as shown in FIG. 5, an auxiliary image indicating the values ​​of each camera parameter corresponding to the knob controller 133 and the accelerator and brake values ​​of the foot pedal 132 is displayed.

[0025] The virtual camera control device 110 is an information processing device including an operation information acquisition unit 111, a change speed determination unit 112, a reaction curve determination unit 113, an inertia parameter determination unit 114, a virtual camera parameter change amount determination unit 115, and a virtual camera parameter calculation unit 116. The virtual camera control device 110 further includes a virtual camera parameter storage unit 117, a virtual camera parameter providing unit 118, a time code storage unit 119, a time code calculation unit 120, a time code providing unit 121, and an auxiliary image generation unit 122. The virtual camera control device 110 controls the virtual camera based on operation information from the operation device 130, and transmits the virtual camera parameters to the virtual viewpoint image generation device 104. In addition, the auxiliary image indicating the camera parameters of the virtual camera to be controlled is transmitted to the display device 106.

[0026] The operation information acquisition unit 111 acquires operation information from the operation device 130. The operation information is an input made by the user to the operation device 130, for example, the angle at which a joystick is tilted (amount of tilt). Note that a joystick has been used as an example in the description, but any input that can acquire the amount of operation in response to a user's operation may be used. For example, the amount of depression of a seesaw switch may be the amount of operation.

[0027] The change speed determination unit 112 determines the change speed of each virtual camera parameter, which will be described later, based on the value acquired from the operation information acquisition unit 111. Specifically, the change speeds of eight virtual camera parameters, namely, the gaze point coordinate (X, Y, Z) [m], the virtual camera direction (Pan, Tilt, Roll) [degrees], the distance R [m] between the virtual camera and the gaze point, and the view angle Zoom [mm] of the virtual camera, are determined. Note that the gaze point is a point in the virtual space located on the optical axis of the virtual camera. Since the gaze point exists on the optical axis of the virtual camera, the user who operates the virtual camera can easily generate a camera path of the virtual camera that keeps the gaze point at the center of the virtual viewpoint image by aligning the gaze point position with the position of the subject of interest. Note that in this embodiment, the gaze point coordinate is treated as the virtual camera parameter, but is not limited to this, and the position (X, Y, Z) [m] of the virtual camera may be treated as the virtual camera parameter. A specific speed determination flow of the virtual camera parameters will be described in S430 to S441 of FIG. 4, which will be described later.

[0028] The response curve determination unit 113 determines the relationship between the tilt amount of the joystick and the change amount of the virtual camera parameter when the joystick 131 is operated. Note that this relationship is a sensitivity using a response curve, and even if the response curve changes, the maximum change amount of the virtual camera parameter for the maximum tilt amount of the joystick does not change. In this embodiment, the response curve is determined based on the Expo value of the knob controller 133. The larger the Expo value, the larger the change amount of the virtual camera parameter even if the tilt amount of the joystick is small, and the smaller the Expo value, the smaller the change amount of the virtual camera parameter when the tilt amount of the joystick is small. In other words, the larger the Expo value, the more sensitively the user's operation is reflected in the change amount, and the smaller the Expo value, the less sensitively the user's operation is reflected in the change amount. The response curve is, for example, a curve calculated by a sigmoid function or a curve calculated by a quadratic function, but is not limited to these.

[0029] The inertia parameter determination unit 114 determines the degree of inertia when the virtual camera moves. In this embodiment, the inertia parameter is determined based on the easing value of the knob controller 133. When the inertia parameter is 0, there is no inertia, so for example, the virtual camera stops immediately when the movement ends. As the inertia parameter increases, the virtual camera moves to a slower stop when the movement ends. Note that this inertia parameter may be one that provides inertia not only when the virtual camera moves, but also when the angle of view changes. Inertia in this embodiment refers to the degree of slippage and acceleration / deceleration that accompany the amount of operation and the elapsed time of the operation when moving the virtual camera.

[0030] Virtual camera parameter change amount determination unit 115 determines the change amount of the virtual camera parameters based on information from operation information acquisition unit 111, change speed determination unit 112, response curve determination unit 113, and inertia parameter determination unit 114. Specifically, the change amount is determined for each of eight parameters: gaze point coordinates (X, Y, Z) [m], virtual camera direction (Pan, Tilt, Roll) [degrees], distance R [m] between the virtual camera and the gaze point, and field angle Zoom [mm] of the virtual camera.

[0031] The virtual camera parameter calculation unit 116 calculates the virtual camera parameters in the current frame based on the information of the virtual camera parameter change amount determination unit 115 and the virtual camera parameter storage unit 117. The virtual camera parameters in the current frame are calculated by adding the change amount determined by the virtual camera parameter change amount determination unit 115 to the virtual camera parameters in the previous frame stored in the virtual camera parameter storage unit 117, and further performing conversion processing. The calculated virtual camera parameters are sent to the virtual camera parameter storage unit to update the virtual camera parameters stored in the virtual camera parameter storage unit 117. The conversion processing includes a process of calculating the virtual camera position from the gaze point coordinates (X, Y, Z) [m], the direction (Pan, Tilt, Roll) [degrees] of the virtual camera, and the distance R [m] between the virtual camera and the gaze point, among the above-mentioned eight virtual camera parameters. This conversion processing will be described with reference to FIG. 3.

[0032] The virtual camera parameter holding unit 117 transmits the virtual camera parameters of the previous frame to the virtual camera parameter calculation unit 116. After that, the virtual camera parameters transmitted from the virtual camera parameter calculation unit 116 are updated as held values.

[0033] The virtual camera parameter providing unit 118 transmits the virtual camera parameters acquired from the virtual camera parameter calculating unit 116 to the virtual viewpoint image generating device 104 and the auxiliary image generating unit 122 .

[0034] The time code of the previous frame is held in a time code holding unit 119. The time code is held in the format of hh:mm:ss:ff.

[0035] The time code calculation unit 120 obtains the time code of the previous frame stored in the time code storage unit 119, and calculates the time code of the current frame by adding one frame. Furthermore, the calculated time code is transmitted to the time code storage unit 119, and the time code stored in the time code storage unit 119 is updated. Note that, in this embodiment, the calculation is performed on the assumption that the playback speed is 100%, but the present disclosure is not limited to this. As an example, when the playback speed is 200%, the time code may be calculated by adding two frames to the time code stored in the time code storage unit 119. Also, the time code may be calculated by saving the time code linked to a certain scene and calling up the time code. In this case, the time code linked to the scene is calculated as the time code of the current frame. Note that the time code calculated by the time code calculation unit 120 is transmitted to the time code provision unit 121.

[0036] The time code providing unit 121 transmits the time code acquired from the time code calculation unit 120 to the virtual viewpoint image generating device 104 and the auxiliary image generating unit 122. Note that the present disclosure does not limit the means for transmitting the time code. As an example, the time code transmitted from the time code providing unit 121 may be added to the virtual camera parameters transmitted from the virtual camera parameter providing unit 118 and transmitted together.

[0037] The operating device 130 includes a joystick 131, a foot pedal 132, and a knob controller 133. Note that the operating device 130 is not limited to the above, and may be other operating members.

[0038] The joystick 131 is a hardware device for a user to operate a virtual camera by tilting or twisting the stick while holding it in his / her hand. FIG. 6 is a diagram showing an example of the joystick 131, which is a device such as a device 601 or a device 602. In this embodiment, two joysticks 603 and 604 are used, and the joystick 603 operates the position of the virtual camera's gaze point, and the joystick 604 operates the direction of the virtual camera. The seesaw switch 605 operates the angle of view of the virtual camera, and the seesaw switch 606 operates the distance R between the virtual camera and the gaze point. The joystick may also be provided with a boost button 607 that can be pressed while operating the joystick. In this embodiment, the speed of change of the virtual camera parameters can be changed while the boost button 607 is pressed, based on the value of a knob 512 that determines a boost value to be described later. In this embodiment, the joystick 131 may also be shaped like a game pad. In the case of a game pad, two joysticks are provided in one device, and it is common to operate them with the thumbs of the left and right hands, respectively. Additionally, functions corresponding to the seesaw switches 605 and 606 and the boost button 607 may be assigned to any button in the game pad.

[0039] The foot pedal 132 is a hardware device operated by the foot. FIG. 7 is a diagram showing an example of the foot pedal 132, which is a device such as an accelerator pedal 701 and a brake pedal 702. Since the foot pedal 132 is a device operated by using the foot, it is possible to operate it while operating the joystick 131 with both hands. In this embodiment, a parameter for increasing the speed of change of the virtual camera is assigned to the accelerator pedal 701, and a parameter for decreasing the speed of change of the virtual camera is assigned to the brake pedal 702. It is also assumed that both the accelerator pedal 701 and the brake pedal 702 are pedals that return to their initial values ​​when the feet are released. In other words, it indicates that the speed of change is increased or decreased only while the foot pedal 132 is being operated by the user.

[0040] The knob controller 133 is a hardware device equipped with knobs for determining the change speed of various parameters related to the operation of the virtual camera parameters. FIG. 8 is a diagram showing an example of the knob controller 133, which is a device equipped with a plurality of knobs. In this embodiment, the 12 knobs, knobs 801 to 812, correspond to the 12 graphics, icons 501 to 512, on the auxiliary image 500 in FIG. 5. For example, by turning the knob 801, the graphic of the icon 501, which is a GUI, also rotates in the same manner, and the corresponding value is displayed as a numerical value below the graphic of the icon 501 in FIG. 5. In the example of FIG. 5, the icons 501 to 508 represent the change speed of each virtual camera parameter individually. Furthermore, the icon 509 represents the change speed of all the virtual camera parameters. That is, by operating knobs 801 to 808 corresponding to icons 501 to 508, each virtual camera parameter can be adjusted, and by operating knob 809 corresponding to icon 509, each virtual camera parameter can be adjusted collectively. Note that the change speed adjusted by icons 501 to 508 and the change speed adjusted by icon 509 adjust parameters independent of each other. Specifically, the change speeds adjusted by icons 501 to 508 are not all set to the same value because icon 509 is operated. Also, icon 510 represents an Expo value that determines the relationship curve between the amount of operation of the controller and the amount of change of the virtual camera parameter. Icon 511 represents an Easing value that determines the degree of slippage (inertia) of the virtual camera. Icon 512 represents a boost value that determines the amount of change of the virtual camera parameter when boost button 607 is pressed. Note that icons 513 and 514 represent values ​​based on the input of accelerator pedal 701 and brake pedal 702.

[0041] It is assumed that the knob controller 133 is operated when the joystick 131 or foot pedal 132 is not being operated. It is also assumed that the knobs are operated before the joystick 131 or foot pedal 132 is operated to determine the values ​​corresponding to the knobs, thereby determining the maximum value for the rate of change of the virtual camera parameters in the subsequent operation. If there is no operation on the knob controller 133, the parameters previously input to the knob controller 133 are held.

[0042] 2 is a diagram showing the hardware configuration of virtual camera control device 110. Note that the hardware configurations of operation device 130 and virtual viewpoint image generation device 104 are also similar to the configuration of virtual camera control device 110 described below. Virtual camera control device 110 may include a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a display unit 205, an operation unit 206, a communication I / F 207, and a system bus 208.

[0043] 1 by controlling the entire virtual camera control device 110 using computer programs and data stored in the ROM 202 and the RAM 203. The virtual camera control device 110 may have one or more pieces of dedicated hardware different from the CPU 201, and at least a part of the processing by the CPU 201 may be executed by the dedicated hardware. Examples of such dedicated hardware include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

[0044] The ROM 202 stores programs that do not require modification.

[0045] The RAM 203 temporarily stores programs and data supplied from the auxiliary storage device 204, and data supplied from the outside via the communication I / F 207, and the like.

[0046] The auxiliary storage device 204 is configured, for example, by a hard disk drive, and stores various data such as image data and audio data.

[0047] The display unit 205 is configured with, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) and the like that allows the user to give instructions to the virtual camera control device 110.

[0048] The operation unit 206 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and receives operations by a user to input various instructions to the CPU 201. The CPU 201 operates as a display control unit that controls the display unit 205, and an operation control unit that controls the operation unit 206.

[0049] The communication I / F 207 is used for communication with devices external to the virtual camera control device 110, such as the virtual viewpoint image generating device 104, the display device 106, and the operation device 130. When the virtual camera control device 110 has a function of wirelessly communicating with external devices, the communication I / F 207 includes an antenna.

[0050] A system bus 208 connects each part of the virtual camera control device 110 and transmits information.

[0051] In this embodiment, the display unit 205 and the operation unit 206 are assumed to exist inside the virtual camera control device 110, but at least one of the display unit 205 and the operation unit 206 may exist outside the virtual camera control device 110 as a separate device.

[0052] 3 is a diagram showing the relationship between the gaze point coordinates and the position of the virtual camera, showing gaze point position 301, virtual camera position 302, Pan 303, Tilt 304, and R 305. For simplicity of explanation, gaze point 301 is described as existing at the origin (0,0,0), but gaze point 301 does not necessarily have to exist at the origin and can be freely specified in three-dimensional space.

[0053] In this embodiment, the virtual camera moves on a sphere of radius R centered on the gaze point 301 . The position on the sphere is uniquely determined by the values ​​of Pan 303 and Tilt 304. In other words, when the gaze point position is (X, Y, Z) [m], the virtual camera direction (Pan, Tilt, Roll) [degrees], and the distance from the gaze point to the virtual camera is R [m], the virtual camera coordinates (Xcam, Ycam, Zcam) are calculated by the following formula.

[0054]

number

[0055]

number

[0056]

number

[0057] The calculated Xcam, Ycam, Zcam and Pan, Tilt, Roll, and Zoom values ​​are transmitted to the virtual camera parameter providing unit 118.

[0058] FIG. 4 is a flowchart showing how virtual camera control device 110 calculates virtual camera parameters based on operation information of various hardware of operation device 130 and transmits them to virtual viewpoint image generation device 104 and auxiliary image generation unit 122.

[0059] Steps S401 and S413 represent repeated processing, and the processing from S402 to S412 is executed for each frame. In this embodiment, the frame rate is set to 59.94DF, that is, the processing from S402 to S412 is executed every (1 / 59.94) seconds. This processing interval depends on the frame rate.

[0060] In step S402, operation information for the joystick 131, the foot pedal 132, and the knob controller 133 is obtained from the operation device .

[0061] In step S403, the change speed of the virtual camera parameters is determined based on the operation information acquired in step S402. Note that the detailed flow of this step will be described later in steps S430 to S441.

[0062] In step S404, a response curve of the virtual camera is determined. That is, a relationship is determined between the tilt amount of the joystick and the change amount of the virtual camera parameters when the joystick 131 is operated. In this embodiment, the response curve is determined based on the Expo value of the knob controller 133.

[0063] In step S405, an inertia parameter of the virtual camera is determined. That is, the degree of inertia effect when the virtual camera moves is determined. In this embodiment, the inertia parameter is determined based on the easing value of the knob controller 133. When the inertia parameter is 0, there is no inertia, so the virtual camera immediately stops when the movement ends when there is no more operation on the virtual camera. As the inertia parameter increases, the virtual camera moves to a slower stop when the movement ends. Note that the inertia parameter may be one that gives inertia not only when the virtual camera moves, but also when the angle of view changes.

[0064] In step S406, the amount of change in the virtual camera parameters is determined based on the operation information acquired in S402, the change speed of the virtual camera parameters determined in S403, the response curve determined in S404, and the inertia parameters determined in S405.

[0065] In step S407, the virtual camera parameters for the current frame are calculated. That is, first, the change amounts of the virtual camera parameters determined in step S406 are added to the virtual camera parameters for the previous frame. Then, the virtual camera coordinates Xcam, Ycam, and Zcam are calculated using the calculation formula explained in Fig. 3. The calculated Xcam, Ycam, and Zcam and the values ​​of Pan, Tilt, Roll, and Zoom among the virtual camera parameters are set as parameters to be transmitted to the virtual viewpoint image generating device 104.

[0066] In step S408, the virtual camera parameters of the previous frame are updated with the virtual camera parameters calculated in S407, and the updated virtual camera parameters are stored.

[0067] In step S409, the values ​​of Xcam, Ycam, Zcam, Pan, Tilt, Roll, and Zoom calculated in S407 are transmitted to the virtual viewpoint image generating device 104.

[0068] In step S410, the time code of the previous frame stored in the time code storage unit 119 is obtained, and one frame is added to calculate the time code of the current frame. Note that, although the calculation is performed on the assumption that playback is performed at 100% playback speed in this embodiment, the present disclosure is not limited to this. As an example, when the playback speed is 200%, the time code may be calculated by adding two frames to the time code stored in the time code storage unit 119. Also, a time code may be stored in association with a certain scene, and the time code may be calculated by calling up the time code. In this case, the time code associated with the scene is calculated as the time code of the current frame.

[0069] In step S411, the time code calculated in S410 is stored in the time code storage unit 119.

[0070] In step S412, the time code calculated in S410 is transmitted to the virtual viewpoint image generating device 104. Note that the means for transmitting the time code is not important in the present disclosure. As an example, the time code calculated in S410 may be added to the virtual camera parameters transmitted from the virtual camera parameter providing unit 118 in S409 and transmitted together.

[0071] Steps S430 to S441 are a detailed version of the flow of S403.

[0072] Steps S431 and S440 represent repeated processes, and the processes from S432 to S439 are executed for each virtual camera parameter. That is, the processes from S432 to S439 are executed for each of the eight parameters, namely, the gaze point coordinates (X, Y, Z), the virtual camera direction (Pan, Tilt, Roll), the distance R between the virtual camera and the gaze point, and the angle of view Zoom. For the sake of simplicity, the processes from S432 to S439 will be described only for the gaze point X coordinate.

[0073] In step S432, an individual change speed parameter V1 in the X coordinate of the gaze point is calculated. That is, an individual change speed parameter V1 set for each virtual camera parameter is calculated based on the value of the icon 501 in Fig. 5. The value of the icon 501 is changed by operating the knob 801. In this embodiment, the value range of the icon 501 is from 0 to 100, and is used as it is as the value of V1.

[0074] In step S433, an overall change speed parameter V2 of the virtual camera parameters is calculated. That is, the change speed parameter V2 is calculated, which is the value of the icon 509 in Fig. 5 and is used in common for calculating the change amount of all virtual camera parameters. In this embodiment, the value range of the icon 509 is from 0 to 100, and is used as it is as the value of the overall change speed parameter V2.

[0075] In step S434, accelerator value V3 is acquired. That is, accelerator value V3 determined by how much accelerator pedal 701 in the device in FIG. 7 is depressed is acquired. Note that, by depressing accelerator pedal 701, icon 513 in FIG. 5 rotates in conjunction with depression, and is also reflected in the value below icon 513. Therefore, the user can adjust the depression of accelerator pedal 701 while looking at icon 513 and the value below icon 513. In this embodiment, accelerator value V3 takes a value of 1.0 to 4.0, and takes a value of 1.0 as an initial value when the foot is completely removed from accelerator pedal 701.

[0076] In step S435, the brake value V4 is acquired. That is, the brake value V4 determined by how much the brake pedal 702 in the device in FIG. 7 is depressed is acquired. Note that by depressing the brake pedal 702, the icon 514 in FIG. 5 rotates accordingly, and is also reflected in the value below the icon 514. Therefore, the user can adjust the degree of depression of the brake pedal 702 while looking at the icon 514 and the value below the icon 514. In this embodiment, the brake value V4 takes a value of 1.0 to 9.0, and takes a value of 1.0 as an initial value when the foot is completely released from the brake pedal 702.

[0077] In step S436, if the boost function is enabled, the process proceeds to S437, and if it is disabled, the process proceeds to S438. Note that in this embodiment, it is determined that the boost function is enabled when the boost button 607 is pressed, and disabled when it is not pressed.

[0078] In step S437, boost value V5 is obtained. That is, it is calculated by the following formula based on the boost value indicated by icon 512 in Fig. 5. At this time, boost value V5 is calculated by Boost, which is the value of icon 512, and the value range of Boost is -4.0 to 4.0. When icon 512 is pointing to the center, Boost is 0.0, when turned all the way to the left it is -4.0, and when turned all the way to the right it is 4.0. V5=2 Boost

[0079] In other words, the value range of V5 is 1 / 16 to 16.

[0080] In step S438, the boost value V5 is set to 1.0.

[0081] In step S439, the speed of the X coordinate of the gaze point is calculated. That is, based on the values ​​of V1, V2, V3, V4, and V5 determined in steps S432 to S438, the speed of change is calculated by the following formula. Note that the speed of change of the X coordinate of the gaze point is designated as Vx.

[0082]

number

[0083] In this embodiment, the change speed of the virtual camera parameters is calculated by multiplying the individual change speeds of the virtual camera parameters, the overall change speed, the accelerator value, the brake value, and the boost value. The reason for calculating by multiplication is to increase the influence that each parameter has on the change speed of the calculated virtual camera parameter. However, the calculation is not limited to the above, and each parameter may be added.

[0084] Therefore, in steps S430 to S441, the change rate of the virtual camera parameters is determined based on the operations on the joystick 131, foot pedal 132, and knob controller 133. By being able to determine the change rate using a plurality of operating members, the change rate can be adjusted to suit each scene even in a competition in which the change rate required in a plurality of imaging scenes varies greatly. Furthermore, since the change rate is determined based on the operation on the foot pedal 132, it becomes easy to operate the virtual camera in accordance with sudden movements of the subject.

[0085] In step S404, the amount of change in the virtual camera parameters is determined based on the Expo value set in response to the operation of knob 810, so that the amount of change can be set according to the user's preference. Since knob 810 is a member different from knobs 801 to 808 that set the individual change speeds of the virtual camera parameters, the sensitivity can be adjusted without changing the change speed. Furthermore, since the change speed can be changed without changing the sensitivity, the change speed of the virtual camera parameters can be changed without changing the sensitivity according to the user's preference, and the user's operation can be facilitated even when the change speed is changed in response to a sudden movement of the subject.

[0086] In step S405, the amount of change in the virtual camera parameters is determined based on the Easing value set in response to the operation of the knob 811, so that the image expression of the virtual viewpoint image can be changed in accordance with the movement of the scene or the subject. For example, in baseball shooting, in a scene where the batter hits the ball thrown by the pitcher, if the virtual camera is to be operated to follow the ball, the moving direction of the ball changes significantly before and after the batter hits it. In such a case, by setting the Easing value to a small value, it becomes possible to appropriately follow the ball, and furthermore, it is possible to produce a sudden change in the situation. As another example, in a scene where a runner runs through multiple bases, the direction in which the runner runs does not change significantly. Therefore, by setting the Easing value to a large value, it is possible to prevent screen sickness caused by a sudden screen change. By setting the Easing value to a member different from the knobs 801 to 808 that set the change speed of the virtual camera parameters, it becomes possible to set the change speed of the camera parameters while maintaining the image expression of the virtual viewpoint image to be shot. In this embodiment, the Easing value is set by the knob 811, but this is not limiting. For example, when a virtual camera is operated using a game controller with a built-in joystick, a function for changing the easing value may be assigned to a button on the game controller. This makes it possible to change the easing value even while operating the joystick, and allows the video expression to be changed according to the scene or subject, even when shooting a sport where scenes change frequently.

[0087] As described above, in this embodiment, by performing operations at the change speed of the virtual camera parameters previously set with the knob controller 133 while setting the accelerator and brake by operating the foot pedal 132, it becomes possible to perform operations that can respond to sudden speed changes of the subject.

[0088] <Embodiment 2> In the first embodiment, a technique has been described that can handle a sudden change in the speed of a subject while operating at a change speed previously set by the knob controller 133 by using the accelerator pedal 701 and the brake pedal 702. Although it has been described that the change speed of all virtual camera parameters changes by depressing the accelerator pedal 701 or the brake pedal 702, depending on the shooting environment of the virtual viewpoint image, there are cases where it is not desirable to change the change speed of all virtual camera parameters. For example, it is considered that the change speed of the virtual camera direction should not be changed in order to prevent screen sickness. Therefore, in the second embodiment, a configuration is described that allows selection of whether or not to reflect the accelerator value V3 corresponding to the accelerator pedal 701 and the brake value V4 corresponding to the brake pedal 702 in the calculation of the change speed for each virtual camera parameter.

[0089] 9 is a system configuration diagram of an image processing system 100 according to the second embodiment of the present disclosure. Note that configurations with the same reference numerals as in FIG. 1 are similar to those in FIG. 1, and therefore description thereof will be omitted.

[0090] The auxiliary image generating unit 901 generates an auxiliary image 1100 as shown in FIG. 11 in addition to the auxiliary image 500 in FIG.

[0091] FIG. 11 is a diagram showing an example of a GUI (auxiliary image 1100) for specifying virtual camera parameters that reflect the accelerator and brake. Check boxes 1101 to 1108 in FIG. 11 are check boxes for selecting virtual camera parameters whose change speed increases when the accelerator pedal 701 is depressed. For example, when check boxes 1102, 1103, and 1105 are checked, only the change speeds of the X coordinate of the gaze point, the Y coordinate of the gaze point, and the distance R from the gaze point to the virtual camera increase when the accelerator pedal 701 is depressed. However, the change speeds of virtual camera parameters that are not checked do not increase. Note that the degree to which the change speed increases changes based on how much the accelerator pedal 701 is depressed, as in the first embodiment, and when the accelerator pedal 701 is completely released, the change speed does not increase. Also, check boxes 1109 to 1116 are check boxes for selecting virtual camera parameters whose change speed decreases when the brake pedal 702 is depressed. For example, when the brake pedal 702 is depressed while all of the check boxes 1109 to 1117 are checked, the rate of change of all the virtual camera parameters decreases.

[0092] The accelerator reflection parameter storage unit 902 stores information indicating whether or not the accelerator value is reflected in the calculation of the change rate for each virtual camera parameter. This information is determined based on whether or not check boxes 1101 to 1108 in the auxiliary screen 1100 in FIG. 11 are checked.

[0093] The brake reflection parameter storage unit 903 stores information indicating whether or not to reflect the brake value in the calculation of the change speed for each virtual camera parameter. This information is determined based on whether or not check boxes 1110 to 1117 in the auxiliary screen 1100 in FIG. 11 are checked.

[0094] The change speed determination unit 904 determines the change speed for each virtual camera parameter based on the value acquired from the operation information acquisition unit 111 and information from the accelerator reflection parameter storage unit 922 and the brake reflection parameter storage unit 923. Specifically, the change speed is determined for each of eight parameters, namely, the gaze point coordinates (X, Y, Z) [m], the virtual camera direction (Pan, Tilt, Roll) [degrees], the distance R [m] between the virtual camera and the gaze point, and the angle of view Zoom [mm] of the virtual camera. A specific speed determination flow will be described later with reference to FIG. 10.

[0095] Fig. 10 is a flowchart of the process of calculating virtual camera parameters of the virtual camera control device 110 in the embodiment 2. Note that, since the processes in S401 to S413 and S430 to S440 in Fig. 4 are common to the embodiment 2, the description thereof will be omitted.

[0096] In step S1001, if the accelerator value is reflected in the calculation of the gaze point X coordinate in the accelerator reflection parameter storage unit 902, the process proceeds to S434, and if not, the process proceeds to S1002. Whether or not the accelerator value is reflected in the calculation of the gaze point X coordinate change speed is based on whether or not a check box 1101 in Fig. 11 is checked. In this embodiment, if the check box 1101 is checked, it is assumed that the accelerator value is reflected in the calculation of the gaze point X coordinate change speed, and the process proceeds to S434, and if not, the process proceeds to S1002.

[0097] In step S1002, the accelerator value V3 is set to 1.0, and then the process proceeds to step S1003.

[0098] In step S1003, if the brake value is reflected in the calculation of the gaze point X coordinate in the brake reflection parameter storage unit 903, the process proceeds to S435, and if not, the process proceeds to S1004. Whether or not the brake value is reflected in the calculation of the gaze point X coordinate change speed is based on whether or not a check box 1109 in Fig. 11 is checked. In this embodiment, if the check box 1109 is checked, it is assumed that the brake value is reflected in the calculation of the gaze point X coordinate change speed, and the process proceeds to S435, and if not, the process proceeds to S1004.

[0099] In step S1004, the brake value V4 is set to 1.0. Then, the process proceeds to step S439.

[0100] As described above, in this embodiment, a configuration has been described that allows selection of whether or not to reflect the accelerator value V3 corresponding to the accelerator pedal 701 and the brake value V4 corresponding to the brake pedal 702 in the calculation of the change rate for each virtual camera parameter. This makes it possible to accommodate cases where it is desired to change the change rate of only some of the virtual camera parameters using the accelerator or brake. That is, as an example, it is possible to accommodate a use case in which the change rate of only the gaze point coordinates (X, Y, Z) is improved while the change rates of Pan, Tilt, and Roll remain constant.

[0101] The present disclosure has been described above based on a number of embodiments, but the present disclosure is not limited to the above embodiments, and various modifications are possible based on the spirit of the present disclosure, and are not excluded from the scope of the present disclosure.

[0102] A computer program for implementing the functions of the above-described embodiment, part or all of the control in this embodiment, may be supplied to an image processing system or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the image processing system or the like may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present disclosure.

[0103] The disclosure of this embodiment includes the following configurations, methods, and programs.

[0104] (Configuration 1) A first acquisition means for acquiring input information corresponding to an input corresponding to an operation on a first member, the input changing virtual camera parameters including a parameter corresponding to a position of a virtual camera or a position of a gaze point corresponding to the virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by capturing an image of a subject using a plurality of imaging devices, a parameter corresponding to an orientation of the virtual camera, and a parameter corresponding to an angle of view of the virtual camera; a second acquisition means for acquiring a first adjustment parameter that adjusts a change amount of the virtual camera parameter corresponding to an operation amount of the operation, the first adjustment parameter being specified based on an operation on a second member, and that retains the first adjustment parameter acquired in a previous input when there is no operation on the second member; a third acquisition means for adjusting a change amount of the virtual camera parameter corresponding to an operation amount of the operation, and acquiring a second adjustment parameter that is specified based on an operation on a third member, and that is changed to an initial value when there is no operation on the third member; a determination means for determining a change amount of the virtual camera parameter based on the input information, the first adjustment parameter, and the second adjustment parameter; An information processing device comprising:

[0105] (Configuration 2) The information processing device according to configuration 1, wherein the first adjustment parameters adjust the amount of change in the amount of operation for parameters corresponding to X, Y, and Z indicating the position of the virtual camera, the distance between the position of the virtual camera and the position of the point of interest, pan, tilt, and roll indicating the orientation of the virtual camera, and the angle of view of the virtual camera.

[0106] (Configuration 3) The information processing device according to configuration 1 or 2, wherein the second adjustment parameter is a parameter for increasing a rate of change of the virtual camera parameter or a parameter for decreasing a rate of change of the virtual camera parameter.

[0107] (Configuration 4) The information processing device according to any one of configurations 1 to 3, wherein the second adjustment parameter is changed to an initial value when no operation is performed on the third member.

[0108] (Configuration 5) The information processing device according to any one of configurations 1 to 4, wherein the first member is a joystick.

[0109] (Configuration 6) The information processing device according to any one of configurations 1 to 5, wherein the second member is a knob.

[0110] (Configuration 7) The information processing device according to any one of configurations 1 to 6, wherein the third member is a foot pedal.

[0111] (Configuration 8) A fourth acquisition means for acquiring a third adjustment parameter which is a parameter for adjusting an amount of change for changing a position and an orientation of the virtual camera in response to an amount of operation of the operation, the third adjustment parameter being changed in response to an amount of operation of a fourth member; The information processing device according to any one of configurations 1 to 7, wherein the determination means determines the amount of change based on the input, the first adjustment parameter, the second adjustment parameter, and the third adjustment parameter.

[0112] (Configuration 9) A fifth acquisition means for acquiring a fourth adjustment parameter which is a parameter for changing sensitivity of the position of the virtual camera and the orientation of the virtual camera to an amount of operation of the operation; The information processing device according to any one of configurations 1 to 8, wherein the determination means determines the amount of change based on the input, the first adjustment parameter, the second adjustment parameter, and the fourth adjustment parameter.

[0113] (Configuration 10) The information processing device according to configuration 9, wherein the sensitivity is determined based on a reaction curve.

[0114] (Configuration 11) The information processing device according to any one of configurations 1 to 10, wherein the determination means determines change amounts of parameters corresponding to X, Y, and Z indicating the position of the virtual camera, the distance between the position of the virtual camera and the position of the point of interest, pan, tilt, and roll indicating the orientation of the virtual camera, and the angle of view of the virtual camera.

[0115] (Configuration 12) The information processing device according to any one of configurations 1 to 11, further comprising a setting means for setting whether or not to determine the amount of change of parameters corresponding to X, Y, Z indicating the position of the virtual camera, the distance between the position of the virtual camera and the position of the point of interest, pan, tilt, roll indicating the orientation of the virtual camera, and the angle of view of the virtual camera, in the determination means.

[0116] (Configuration 13) The information processing device according to any one of configurations 1 to 12, further comprising an output unit for outputting the virtual camera parameters to which the amount of change determined by the determination unit has been added.

[0117] (Configuration 14) The information processing device according to any one of configurations 1 to 13, further comprising a generation means for generating an auxiliary image indicating the first adjustment parameter acquired by the second acquisition means and the second adjustment parameter acquired by the third acquisition means.

[0118] (Configuration 15) The information processing device described in Configuration 14, wherein the auxiliary image indicates the first adjustment parameters corresponding to X, Y, Z indicating the position of the virtual camera, the distance between the position of the virtual camera and the position of the point of interest, pan, tilt, roll indicating the orientation of the virtual camera, and the angle of view of the virtual camera.

[0119] (Configuration 16) The information processing device according to any one of configurations 1 to 15, characterized in that the gaze point is determined to be located on the optical axis of the virtual camera.

[0120] (Configuration 17) A sixth acquisition means for acquiring a fifth adjustment parameter, which is a parameter for determining the amount of change, based on the amount of operation and an elapsed time of the operation; The information processing device according to any one of configurations 1 to 16, wherein the determination means determines the amount of change based on the input, the first adjustment parameter, the second adjustment parameter, and the fifth adjustment parameter.

[0121] (Method) A first acquisition step of acquiring input information corresponding to an input corresponding to an operation on a first member, the input changing virtual camera parameters including a parameter corresponding to a position of a virtual camera or a position of a gaze point corresponding to the virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by capturing an image of a subject with a plurality of imaging devices, a parameter corresponding to an orientation of the virtual camera, and a parameter corresponding to an angle of view of the virtual camera; a second acquisition step of acquiring a first adjustment parameter that is a parameter for adjusting a change amount of the virtual camera parameter with respect to an amount of the operation, the first adjustment parameter being specified based on an operation performed on a second member; a third acquisition step of acquiring a second adjustment parameter which is a parameter for adjusting a change amount of the virtual camera parameter in response to an amount of operation of the operation, the second adjustment parameter being changed in response to an amount of operation of a third member; a determination step of determining a change amount of the virtual camera parameter based on the input information, the first adjustment parameter, and the second adjustment parameter; An information processing method comprising:

[0122] (Program) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 17. [Explanation of symbols]

[0123] 110 Virtual camera control device 111 Operation information acquisition unit 112 Change rate determination unit 113 Response curve determination unit 114 Inertia parameter determination unit 116 Virtual camera parameter change amount determination unit 131 Joystick 132 Foot Pedal 133 Knob Controller 122 Auxiliary Image Generation Unit

Claims

1. an input corresponding to an operation on a first member, the input corresponding to a position of a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by capturing an image of a subject using a plurality of imaging devices, or a parameter corresponding to a position of a gaze point corresponding to the virtual camera; a first acquisition means for acquiring input information corresponding to an input for changing virtual camera parameters including a parameter corresponding to a direction of the virtual camera and a parameter corresponding to an angle of view of the virtual camera; a second acquisition means for acquiring a first adjustment parameter that adjusts a change amount of the virtual camera parameter corresponding to an amount of operation of the operation, the first adjustment parameter being specified based on an operation on a second member, and that retains the first adjustment parameter acquired in a previous input when there is no operation on the second member; a third acquisition means for adjusting a change amount of the virtual camera parameter corresponding to an operation amount of the operation, and acquiring a second adjustment parameter that is specified based on an operation performed on a third member, and that is changed to an initial value when there is no operation performed on the third member; a determination means for determining a change amount of the virtual camera parameter based on the input information, the first adjustment parameter, and the second adjustment parameter; An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the first adjustment parameters adjust the amount of change in the amount of operation of the operation for parameters corresponding to X, Y, and Z indicating the position of the virtual camera, the distance between the position of the virtual camera and the position of the point of interest, pan, tilt, and roll indicating the orientation of the virtual camera, and an angle of view of the virtual camera.

3. 2 . The information processing apparatus according to claim 1 , wherein the second adjustment parameter is a parameter for increasing a rate of change of the virtual camera parameter or a parameter for decreasing a rate of change of the virtual camera parameter.

4. The information processing apparatus according to claim 1 , wherein, when the third member is operated, the second adjustment parameter is determined based on an amount of operation of the third member.

5. The information processing apparatus according to claim 1 , wherein the first member is a joystick.

6. The information processing apparatus according to claim 1 , wherein the second member is a knob.

7. 2. The information processing apparatus according to claim 1, wherein the third member is a foot pedal.

8. a fourth acquisition means for acquiring a third adjustment parameter that is a parameter for adjusting an amount of change for changing a position and an orientation of the virtual camera in response to an amount of operation of the operation, the third adjustment parameter being changed in response to an amount of operation performed on a fourth member; 2 . The information processing apparatus according to claim 1 , wherein the determining unit determines the amount of change based on the input, the first adjustment parameter, the second adjustment parameter, and the third adjustment parameter.

9. a fifth acquisition means for acquiring a fourth adjustment parameter which is a parameter for changing sensitivity of a position of the virtual camera and a direction of the virtual camera to an amount of the operation; 2 . The information processing apparatus according to claim 1 , wherein the determining unit determines the amount of change based on the input, the first adjustment parameter, the second adjustment parameter, and the fourth adjustment parameter.

10. The information processing apparatus according to claim 9 , wherein the sensitivity is determined based on a response curve.

11. 2. The information processing device according to claim 1, wherein the determination means determines change amounts of parameters corresponding to X, Y, and Z indicating the position of the virtual camera, the distance between the position of the virtual camera and the position of the point of interest, pan, tilt, and roll indicating the orientation of the virtual camera, and the angle of view of the virtual camera.

12. The information processing device according to claim 1, further comprising a setting means for setting whether or not to determine amounts of change of parameters corresponding to X, Y, and Z indicating a position of the virtual camera, a distance between the position of the virtual camera and a position of the point of interest, pan, tilt, and roll indicating a direction of the virtual camera, and an angle of view of the virtual camera, in the determination means, based on the second adjustment parameter.

13. 2. The information processing apparatus according to claim 1, further comprising an output unit that outputs the virtual camera parameters to which the amount of change determined by the determination unit has been added.

14. 2. The information processing apparatus according to claim 1, further comprising a generating unit configured to generate an auxiliary image indicating the first adjustment parameter acquired by the second acquisition unit and the second adjustment parameter acquired by the third acquisition unit.

15. The information processing device according to claim 14, characterized in that the auxiliary image indicates the first adjustment parameters corresponding to X, Y, and Z indicating the position of the virtual camera, the distance between the position of the virtual camera and the position of the point of interest, pan, tilt, and roll indicating the orientation of the virtual camera, and the angle of view of the virtual camera.

16. The information processing apparatus according to claim 1 , wherein the gaze point is determined to be located on an optical axis of the virtual camera.

17. a sixth acquisition means for acquiring a fifth adjustment parameter that is a parameter for determining the amount of change based on the amount of operation and an elapsed time of the operation; 2 . The information processing apparatus according to claim 1 , wherein the determining unit determines the amount of change based on the input, the first adjustment parameter, the second adjustment parameter, and the fifth adjustment parameter.

18. an input corresponding to an operation on a first member, the input corresponding to a position of a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by capturing an image of a subject using a plurality of imaging devices, or a parameter corresponding to a position of a gaze point corresponding to the virtual camera; a first acquisition step of acquiring input information corresponding to an input for changing virtual camera parameters including a parameter corresponding to a direction of the virtual camera and a parameter corresponding to an angle of view of the virtual camera; a second acquisition step of acquiring a first adjustment parameter, the first adjustment parameter being a parameter for adjusting an amount of change of the virtual camera parameter in response to an amount of operation of the operation, the first adjustment parameter being specified based on an operation performed on a second member; a third acquisition step of acquiring a second adjustment parameter which is a parameter for adjusting a change amount of the virtual camera parameter in response to an amount of operation of the operation, the second adjustment parameter being changed in response to an amount of operation of a third member; a determination step of determining a change amount of the virtual camera parameter based on the input information, the first adjustment parameter, and the second adjustment parameter; An information processing method comprising:

19. A program for causing a computer to function as the information processing device according to any one of claims 1 to 17.

Images