Control device, control method, and program

Abstract

The goal is to achieve a comfortable level of blur in virtual perspective video for the viewer. [Solution] Operation information via an operating member of an input device that specifies a virtual camera in a virtual space, and angular velocity information from a gyro sensor equipped in the input device, are acquired on a frame-by-frame basis. Then, the orientation of the virtual camera is determined on a frame-by-frame basis based on the acquired operation information. Furthermore, the orientation of the virtual camera determined on a frame-by-frame basis is corrected based on angular velocity information from multiple frames.

Description

[Technical Field]

[0001] This disclosure relates to control technology for virtual cameras in a virtual space. [Background technology]

[0002] There is a technology that involves setting up multiple imaging devices at different locations and performing synchronized imaging. Using the multiple images obtained from this imaging, an image representing the view from a virtual camera (virtual viewpoint) that does not actually exist in the three-dimensional space of the object being imaged (virtual viewpoint image) is generated. During generation, the user can refer to the generated virtual viewpoint image on the UI screen and set the position and orientation of a new virtual camera by operating a controller, for example, equipped with a joystick. However, if it is desired to reproduce natural camera shake, such as that seen when using a handheld camera, in the virtual viewpoint image, this is difficult to achieve with joystick operation alone. One possible method to achieve this is to use a gyro sensor that detects the angle change (angular velocity) of rotation or tilt per unit of time.

[0003] The integration of gyro sensors into controllers is widely practiced in fields such as games. Patent Document 1 discloses a technique for determining the movement of a character swinging an item (sword) in a game, by using angular velocity information from a gyro sensor mounted on the controller to approximate the orientation of the item to the orientation of the controller. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2011-239985 [Overview of the project] [Problems that the invention aims to solve]

[0005] When determining the orientation of a virtual camera based on joystick operation and angular velocity detected by a gyro sensor, if unintended behavior is applied to the controller through other button operations, the angular velocity will also fluctuate accordingly. In such cases, the technology described in Patent Document 1 may result in, for example, the virtual camera pointing in an unintended direction, causing the subject of interest to move out of the field of view, or the viewing direction changing frequently, resulting in a virtual viewpoint image that is difficult to view.

[0006] This disclosure addresses the above-mentioned issues and aims to achieve a comfortable sense of blur in virtual viewpoint video for viewers. [Means for solving the problem]

[0007] The control device according to this disclosure is characterized by comprising: acquisition means for acquiring operation information via an operating member of an input device that specifies a virtual camera in a virtual space and angular velocity information of a gyro sensor provided by the input device on a frame-by-frame basis; determination means for determining the orientation of the virtual camera on a frame-by-frame basis based on the operation information; and correction means for correcting the orientation of the virtual camera determined on a frame-by-frame basis based on angular velocity information for multiple frames. [Effects of the Invention]

[0008] According to this disclosure, it is possible to achieve a comfortable sense of blur in virtual viewpoint video for the viewer. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram showing an example of the configuration of a virtual viewpoint image generation system. [Figure 2] A diagram showing the hardware resources of each device that makes up the virtual viewpoint image generation system. [Figure 3] A flowchart illustrating the operation flow for generating and outputting virtual camera parameters based on operation information and angular velocity information according to Embodiment 1. [Figure 4](a) and (b) are diagrams showing an example of a viewpoint input device. [Figure 5] A diagram for explaining that the position of a virtual camera can be uniquely specified based on a fixation point, a fixation point distance, and the orientation of the virtual camera. [Figure 6] A diagram for explaining the problems of Embodiment 2. [Figure 7] A flowchart showing the flow of an operation of generating and outputting virtual camera parameters based on operation information and angular velocity information according to Embodiment 2.

Mode for Carrying Out the Invention

[0010] Hereinafter, the present invention will be described in detail based on its preferred embodiments with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples and are not limited to the illustrated configurations.

[0011] [Embodiment 1] <System Configuration> FIG. 1 is a diagram showing an example of the configuration of a system for generating a virtual viewpoint image according to the present embodiment. The virtual viewpoint image generation system of the present embodiment includes a camera group 101, a 3D model generation device 102, a 3D model storage device 103, a virtual viewpoint image generation device 104, a display 105, a control device 110, and a viewpoint input device 120.

[0012] The camera group 101 composed of a plurality of imaging devices (cameras) is installed so as to surround an imaging space where a subject (object) exists, and performs imaging while synchronizing the entire unit time. In the present embodiment, each camera of the camera group 101 is assumed to perform imaging of a moving image at a frame rate of 60 fps.

[0013] The 3D model generation device 102 performs foreground extraction processing on a frame-by-frame basis on the moving images obtained from each camera of the camera group 101 to generate an image representing the silhouette of the foreground object (hereinafter referred to as "foreground silhouette"). Then, based on the generated silhouette image, it applies techniques such as VisualHull to generate data representing the three-dimensional shape of the object (generally called a "3D model") on a frame-by-frame basis. The data format of the 3D model is not particularly limited and can include polygon format, point cloud format, voxel format, etc. The 3D model storage device 103 stores the 3D model generated by the 3D model generation device 102.

[0014] The virtual viewpoint image generation device 104 generates virtual viewpoint images frame by frame based on virtual camera parameters acquired from the control device 110. The generated virtual viewpoint images are output to the display 105. Note that the output destination of the virtual viewpoint images does not necessarily have to be a display; for example, it could be a distribution provider or broadcasting station. The display 105 displays the virtual viewpoint images received from the virtual viewpoint image generation device 104.

[0015] The viewpoint input device 120 is an operating device for the user to specify the position and orientation of a virtual imaging device (virtual camera) in a virtual space corresponding to the imaging space of the camera group 101, and is equipped with a gyro sensor. The gyro sensor is a device that detects angular velocity per unit time. In this embodiment, it is designed to detect angular velocity in three axes: the X, Y, and Z axes, but a configuration that detects angular velocity in three axes in combination with a one-axis or two-axis gyro sensor is also acceptable. The user can continuously generate virtual viewpoint images corresponding to a desired virtual viewpoint by operating the viewpoint input device 120 while viewing the virtual viewpoint image displayed on the display 105 to specify the position and orientation of a new virtual camera. The viewpoint input device 120 can be any device equipped with a gyro sensor and an operating member such as a joystick or button for specifying the point of focus, position, and orientation of the virtual camera according to user operation. For example, it could be a gamepad held with both hands as described later, or a type held with one hand.

[0016] The control device 110 generates a set of parameters (virtual camera parameters) that define the position and orientation of the virtual camera based on the information input from the viewpoint input device 120, and outputs them to the virtual viewpoint image generation device 104. The virtual camera parameters in this embodiment shall include at least information on the position, orientation, and field of view of the virtual camera in the virtual space. The control device 110 has an input unit 111, a virtual camera parameter generation unit 112, and an output unit 119. The virtual camera parameter generation unit 112 consists of a gaze point determination unit 113, a gaze point distance determination unit 114, a field of view determination unit 115, an orientation determination unit 116, an orientation correction unit 117, and a position determination unit 118. Details of each part of the control device 110 will be described later. Note that the system configuration shown in Figure 1 is merely an example. For example, the internal configuration of the virtual camera parameter generation unit 112 of the control device 110 may consist of a determination unit that calculates and determines various parameters of the virtual camera based on operation information, and a correction unit that corrects the orientation of the virtual camera from the parameters determined by the determination unit.

[0017] <Hardware Configuration> Figure 2 shows the hardware resources of each device that constitutes the virtual viewpoint image generation system shown in Figure 1. Specifically, the 3D model generation device 102, the 3D model storage device 103, the virtual viewpoint image generation device 104, the control device 110, and the viewpoint input device 120 can be realized by the information processing device 200 shown in Figure 2.

[0018] The information processing device 200 includes a CPU 201, ROM 202, RAM 203, auxiliary storage device 204, display unit 205, operation unit 206, communication interface 207, and system bus 209.

[0019] The CPU 201 controls the entire information processing device 200 using programs and data stored in the ROM 202 and RAM 203. For example, each part 111 to 119 of the control device 110 is realized by the CPU 201 of the control device 110 executing a predetermined program. The information processing device 200 may have one or more dedicated hardware components separate from the CPU 201, and at least a portion of the processing performed by the CPU 201 may be executed by the dedicated hardware. Examples of such dedicated hardware include ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and DSPs (Digital Signal Processors).

[0020] ROM202 stores programs and other data that do not require modification. RAM203 temporarily stores programs and data supplied from auxiliary storage device 204, as well as data supplied from external sources via communication interface 207. Auxiliary storage device 204 is composed of, for example, a hard disk drive and stores various types of data such as image data and audio data.

[0021] The display unit 205 is composed of, for example, a liquid crystal display or LEDs, and displays a GUI (Graphical User Interface) for the user to give instructions to the information processing device 200. The operation unit 206 is composed of, for example, a keyboard, mouse, joystick, touch panel, etc., and receives various instructions from the user and inputs them to the CPU 201. The CPU 201 operates as a display control unit that controls the display unit 205 and an operation control unit that controls the operation unit 206. In this embodiment, the display unit 205 and the operation unit 206 are assumed to be located inside the information processing device 200, but at least one of the display unit 205 and the operation unit 206 may exist as a separate device outside the information processing device 200.

[0022] The communication interface 207 is used for communication between the information processing device 200 and external devices. If the information processing device 200 has the function of wirelessly communicating with external devices, the communication interface 207 is equipped with an antenna. The system bus 208 connects the various parts of the information processing device 200 and transmits information.

[0023] Figure 2 shows an example of the basic hardware that each device constituting the virtual viewpoint image generation system has in common. For example, the control device 110 may also be equipped with a gyro sensor, and the hardware configuration may differ depending on the purpose and function of the device in question.

[0024] <Operation Flow of Control Device> Next, referring to the system configuration diagram shown in Figure 1 and the flowchart shown in Figure 3, the operation of the control device 110 in generating and outputting virtual camera parameters based on the operation information and angular velocity information input from the viewpoint input device 120 will be explained in detail. In the following explanation, the symbol "S" means step. In this embodiment, where a virtual viewpoint image is generated based on a moving image captured at 60fps, each of the processes S301 to S312 will be executed frame by frame every (1 / 60) seconds.

[0025] In S301, the input unit 111 acquires operation information and angular velocity information related to the current frame (the frame of interest for processing) output from the viewpoint input device 120 via the communication I / F 207. Here, the operation information is information representing the operation content output by the user, such as tilting the joystick of the viewpoint input device 120. The angular velocity information is information indicating the angular velocity of the three axes (pan axis, tilt axis, and roll axis) detected by the gyro sensor mounted on the viewpoint input device 120. Here, pan is the angle of rotation parallel to the ground surface, tilt is the angle of rotation perpendicular to the ground surface, and roll is the angle of rotation relative to the optical axis of the virtual camera. In this embodiment, the angular velocity information is ω pan、 ω Tilt、 ω RollThese are expressed in degrees per second (°C / sec). Figures 4(a) and 4(b) show an example of the viewpoint input device 120, which is a gamepad held with both hands. The gamepad shown in Figures 4(a) and 4(b) has operating elements such as L1 / R1 buttons, L2 / R2 buttons, a directional pad, left / right sticks, and a round button. For example, the user tilts the left stick 405 in the desired direction to change the position of the point of focus (point of fixation) on the X and Y axes in the virtual space. Also, by pressing the L1 button 401, the position of the point of fixation on the Z axis is raised, and by pressing the R1 button 402, the position of the point of fixation on the Z axis is lowered. Furthermore, by pressing the L2 button 403, the user gradually decreases the distance from the virtual camera to the point of fixation (point of fixation distance), and by pressing the R2 button 404, the point of fixation distance is gradually increased. Furthermore, the user can gradually zoom in on the virtual camera's field of view by pressing the upper part of the directional pad 407, and gradually widen the field of view by pressing the lower part. The user can also change the pan and tilt of the virtual camera by tilting the right stick 406 in the desired direction. Additionally, the user can rotate the roll to the left by pressing the upper circular button 408a of the four circular buttons 408a to 408d, and rotate the roll to the right by pressing the right circular button 408b. The above operation method is just one example of how to operate the virtual camera, and the assignment of each button is not limited to this. The acquired operation information and angle information are sent to the virtual camera parameter generation unit 112. The virtual camera parameter generation unit 112 then executes the processes S302 to S310 using the respective function units 113 to 118.

[0026] In S302, the fixation point determination unit 113 determines the fixation point of the virtual camera based on the operation information of the current frame acquired in S301. Specifically, based on the operation information of the current frame, it calculates the change from the fixation point in the previous frame, adds the calculated change to the position of the fixation point in the previous frame, and sets the position of the fixation point in the current frame. It is assumed that the fixation point is always located on the optical axis of the virtual camera. In this embodiment, the fixation point is represented by a three-dimensional coordinate system consisting of three axes: the X axis, the Y axis, and the Z axis. The three-dimensional coordinate system (X,Y,Z) is represented by numerical values ​​that represent the distance from the origin, for example, X=4.0, Y=9.0, Z=1.5, with each unit being [m]. In this case, the origin (X,Y,Z)=(0,0,0) is, for example, the center of the 3D model generation range. When the X axis is specified to be parallel to the ground surface, the Y axis is specified to be parallel to the ground surface and perpendicular to the X axis, and the Z axis is specified to be perpendicular to the ground surface. The determined point of focus information is sent to the position determination unit 117.

[0027] In S303, the gaze point distance determination unit 114 determines the distance from the virtual camera to the gaze point determined in S302 (hereinafter referred to as "gaze point distance"). Specifically, based on the operation information of the current frame, it calculates the change from the gaze point distance in the previous frame, adds the calculated change to the gaze point distance in the previous frame, and obtains the gaze point distance in the current frame. In this embodiment, the unit of gaze point distance R is [m]. The determined gaze point distance information is sent to the position determination unit 117.

[0028] In S304, the field of view determination unit 115 determines the field of view of the virtual camera based on the operation information of the current frame acquired in S301. Specifically, it calculates the change in the field of view from the previous frame based on the operation information of the current frame, adds the calculated change to the field of view of the previous frame, and determines the field of view of the current frame. Note that there is a correlation between the field of view and focal length; a shorter focal length results in a wider angle, and a longer focal length results in a narrower angle (zoom). Once the focal length is determined, the field of view is uniquely determined. Therefore, in this embodiment, the field of view is identified by a numerical value representing the focal length (Zoom value), for example, a numerical value such as Zoom value = 6.0 (unit: [mm]). The determined field of view (≒focal length) information is sent to the position determination unit 117.

[0029] In S305, the attitude determination unit 116 determines the attitude of the virtual camera based on the operation information acquired in S301. Specifically, it calculates the change from the attitude in the previous frame based on the operation information of the current frame, adds the calculated change to the attitude in the previous frame, and determines the attitude in the current frame. In this embodiment, the attitude of the virtual camera is represented by three angles: pan, tilt, and roll, for example, (Pan,Tilt,Roll)=(20.0,10.0,2.0), with each value range being -180 degrees to 180 degrees. The determined attitude information is sent to the attitude correction unit 117.

[0030] In S306, the attitude correction unit 117 determines whether the function that corrects the attitude of the virtual camera based on angular velocity information from the gyro sensor (hereinafter referred to as the "gyro correction function") is turned on. The next process to be executed is then determined based on the result of this determination. Here, the on / off setting of the gyro correction function can be selected in advance by the user, for example, via a user interface screen (UI screen) not shown. If the gyro correction function is on, S307 is executed next; if it is off, S310 is executed next.

[0031] In S307, the attitude correction unit 117 determines the setting value of the correction strength when executing the gyro correction function. Then, the next process to be executed is distributed according to the determination result. Here, the correction strength setting can be set in advance by the user, for example, by configuring the screen to allow the user to select the strength after "On" is selected on the UI screen mentioned above. In this embodiment, it is assumed that either "Strong" or "Soft" is set as the correction strength setting value. If the correction strength setting value is "Strong", S308 is executed next, and if it is "Soft", S309 is executed next.

[0032] In S308, the attitude correction unit 117 corrects the attitude of the virtual camera determined in S305 based on the angular velocity information of the current frame acquired in S301. In this embodiment, processing is assumed to be done at a frame rate of 60fps. In this case, the correction amount is calculated by dividing the input angular velocity of the current frame by "60". In this embodiment, the attitude of the virtual camera is specified by pan, tilt, and roll, so the correction amounts (dPan, dTilt, dRoll) are obtained by the following equations (1) to (3).

[0033]

number

[0034]

number

[0035]

number

[0036] [Number]

[0037] [Number]

[0038] [Number] In S309, the posture correction unit 117 corrects the posture of the virtual camera determined in S305 based on the angular velocity information of the current frame acquired in S301 and the angular velocity information of the immediately previous frame. Posture correction based on angular velocity information may sometimes result in an overly harsh and unpleasant video. In such a case, the user selects "Weak (Soft)". In this step, first, the correction amounts (dPan, dTilt, dRoll) are obtained using the above equations (1) to (3) for a plurality of frames including the current frame and the immediately previous frame. Then, the smoothed values (dPan sm , dTilt sm , dRoll sm ) of the correction amounts obtained for each frame are added to the posture (Pan, Tilt, Roll) determined by the posture determination unit 116. That is, the corrected posture (Pan_corr, Tilt_corr, Roll_corr) in this step is represented by the following equations (7) to (9), and the blur in a plurality of frames is smoothed.

[0039]

number

[0040]

number

[0041]

number

[0042] There is no limit to how many frames of angular velocity information, including past frames, should be considered. For example, considering the angular velocity information of the current frame plus the three most recent frames (a total of four frames) can reproduce a reasonable sense of camera shake. Alternatively, it is sufficient to smooth the blur across multiple frames; for example, the median value can be used instead of the average value.

[0043] In S310, the position determination unit 118 determines the position of the virtual camera based on the point of focus, point of focus distance, and orientation of the virtual camera determined in the processes up to this point. In this embodiment, the position of the virtual camera is represented by a three-dimensional coordinate system consisting of three axes: the X, Y, and Z axes, similar to the point of focus. Figure 5 illustrates how the position of the virtual camera can be uniquely determined based on the point of focus, point of focus distance, and orientation of the virtual camera. In Figure 5, the black circle 501 indicates the point of focus, and the double arrow 505 indicates the distance from the virtual camera 502 to the point of focus 501. From Figure 5, it can be seen that the position of the virtual camera 502 is determined so that it faces the point of focus, based on the point of focus distance 505, the angle 503 in the Pan direction of the virtual camera 502, and the angle 504 in the Tilt direction of the virtual camera 502. For the sake of explanation, in Figure 5 the position of the point of focus 501 is set to the origin (0,0,0), but the point of focus can be any position in the virtual space. The three-dimensional coordinates (Xcam, Ycam, Zcam) representing the position of the virtual camera are determined differently depending on whether the gyro correction function is on or off. If the gyro correction function is on, the coordinates are determined using the following equations (10) to (12) with the attitude (Pan_corr, Tilt_corr, Roll_corr) corrected by S308 or SS309.

[0044]

number

[0045]

number

[0046]

number

[0047] On the other hand, if the gyro correction function is turned off, the three-dimensional coordinates (Xcam, Ycam, Zcam) representing the position of the virtual camera can be obtained using the following equations (13) to (15) with the attitude (Pan, Tilt, Roll) determined in S305.

[0048]

number

[0049]

number

[0050]

number

[0051] In S311, the output unit 119 transmits the virtual camera parameters for the current frame, generated by the virtual camera parameter generation unit 112, to the virtual viewpoint image generation device 104 via the communication interface 207.

[0052] In S312, it is determined whether to continue generating virtual camera parameters. For example, if operation information and angular velocity information continue to be input from the viewpoint input device 120 and virtual camera parameter generation is to continue, the process returns to S301 and continues. On the other hand, if the input of operation information and angular velocity information from the viewpoint input device 120 stops and virtual camera parameter generation is to end, this flow is exited.

[0053] The above describes the virtual camera parameter generation process according to this embodiment. In this embodiment, the correction strength setting is limited to two levels, "Strong" and "Soft," but is not limited to these. For example, the correction strength may be selectable from three or more levels, such as "Strong," "Normal," and "Soft." Alternatively, the user may only be allowed to select whether to turn the gyro correction function on or off, and the correction strength may be fixed to a recommended setting (e.g., "Soft").

[0054] As described above, according to this embodiment, by correcting the orientation of the virtual camera, which is determined based on the operation information of the operating member of the viewpoint input device, based on the angular velocity information of the gyro sensor mounted on the viewpoint input device, it becomes possible to reproduce a natural sense of blur in the virtual viewpoint image. In the above embodiment, the correction strength setting was expressed in two stages: "Strong" and "Soft," but this can also be rephrased as turning the correction amount adjustment function on or off. In the above embodiment, when the correction strength setting is "Strong," the smoothing process is not performed, and when the correction strength setting is "Soft," the smoothing process is performed, so it can also be said that this sets the on / off state of the smoothing process.

[0055] [Embodiment 2] In Embodiment 1, the orientation of the virtual camera, determined based on the operation information of the operating member, was corrected based on the angular velocity information of the gyro sensor. However, in the method of Embodiment 1, the amount of change (movement) of the virtual camera's position when the orientation is corrected increases as the distance to the point of focus increases. This can lead to problems such as motion sickness. Therefore, Embodiment 2 describes a method in which the amount of movement of the virtual camera does not become too large when correcting the orientation of the virtual camera based on the angular velocity information of the gyro sensor. Note that the explanation of the system configuration and other contents common to Embodiment 1 will be omitted, and the following will focus on the differences.

[0056] Before describing this embodiment, the problems of this embodiment will be explained with reference to Figure 6. In Figure 6, the black circle 601 indicates the point of fixation, the double arrow 607 indicates the point of fixation distance of camera 602, and the double arrow 617 indicates the point of fixation distance of camera 612. Cameras 602 and 612 represent virtual cameras before attitude correction, while cameras 603 and 613 represent virtual cameras after attitude correction. The dashed line 604 extending from camera 602 towards the point of fixation 601 indicates the optical axis of virtual camera 602, and the dashed line 604' extending from camera 612 towards the point of fixation 601 indicates the optical axis of virtual camera 612. The dashed line 605 extending from camera 603 towards the point of fixation 601 indicates the optical axis of virtual camera 603, and the dashed line 605' extending from camera 613 towards the point of fixation 601 indicates the optical axis of virtual camera 613. The straight double arrow 607 indicates the gaze distance of virtual cameras 602 and 603, and the straight double arrow 617 indicates the gaze distance of virtual cameras 612 and 613. Furthermore, the double arrow 606 on the arc indicates the distance traveled from the uncorrected virtual camera 602 to the corrected virtual camera 603, and the double arrow 616 on the arc indicates the distance traveled from the uncorrected virtual camera 612 to the corrected virtual camera 613. From Figure 6, it can be seen that the travel distance increases in proportion to the gaze distance. Then, the angular velocity information (ω) from the input gyro sensor... pan, ω Tilt, ω Roll Even if the values ​​of ) are the same, if the gaze distance is large, the position of the virtual camera may change significantly more than the user would expect. This embodiment aims to control the amount of correction so that the position of the virtual camera does not change too much when correcting the attitude of the virtual camera.

[0057] <Operation Flow of Control Device> Next, the operation of the control device 110 in this embodiment, which generates and outputs virtual camera parameters based on the operation information and angular velocity information input from the viewpoint input device 120, will be described in detail. In the following description, the symbol "S" means step. Also, similar to the flow in Figure 3 of Embodiment 1, the series of processes shown in the flow in Figure 7 will be executed in frame units every (1 / 60) seconds.

[0058] Steps S301 to S307 are the same as in Embodiment 1, so their explanation will be omitted. If the correction strength setting value in the judgment in S307 is "Strong", then S801 is executed next; if it is "Soft", then S802 is executed next.

[0059] In S701, the attitude correction unit 117 corrects the attitude of the virtual camera determined in S305 based on the angular velocity information of the current frame acquired in S301 and the gaze point distance determined in S303. This embodiment also assumes processing at a frame rate of 60fps. In this case, the correction amount is calculated by dividing the input angular velocity of the current frame by "60", but in this embodiment, the correction amount is calculated so that it decreases as the gaze point distance increases. Specifically, the correction amounts (dPan, dTilt, dRoll) are obtained by the following equations (16) to (18).

[0060]

number

[0061]

number

[0062]

number

[0063] In S702, the attitude correction unit 117 corrects the attitude of the virtual camera determined in S305 based on the angular velocity information of the current frame and the most recent past frame, and the gaze point distance determined in S303. Specifically, first, the correction amount (dPan, dTilt, dRoll) is calculated for multiple frames, including the current frame, using the aforementioned equations (16) to (18). Next, the average value of the calculated correction amount (dPan ave, dTilt ave, dRoll ave The angular velocity is calculated. Then, the average value obtained is added to the attitude (Pan, Tilt, Roll) determined by the attitude determination unit 116 to obtain the corrected attitude (Pan_corr,Tilt_corr,Roll_corr). Similar to S309 in the embodiment, there is no limitation on how many past frames of angular velocity information should be considered.

[0064] The following steps S310 and S311 are the same as in Embodiment 1, so their explanation will be omitted. The above describes the virtual camera control process according to this embodiment. Note that the above equations (16) to (18) for calculating the correction amount are just examples, and any calculation formula that decreases the correction amount as the gaze point distance increases is acceptable. For example, an exponential function such as those shown in the following equations (19) to (21) may be used.

[0065]

number

[0066]

number

[0067]

number

[0068] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0069] Furthermore, this disclosure includes the following configurations and methods.

[0070] [Configuration 1] An acquisition means for acquiring, on a frame-by-frame basis, operation information via an operating member of an input device that specifies a virtual camera in a virtual space, and angular velocity information from a gyro sensor provided by the input device. A determination means for determining the orientation of the virtual camera on a frame-by-frame basis based on the aforementioned operation information, Correction means for correcting the orientation of the virtual camera, which is determined on a frame-by-frame basis, based on angular velocity information from multiple frames, A control device characterized by having the following features.

[0071] [Configuration 2] The control device according to configuration 1, characterized in that the determination means determines the point of focus of the virtual camera on a frame-by-frame basis based on the operation information.

[0072] [Configuration 3] The control device according to configuration 2, characterized in that the correction means corrects the determined pose of the virtual camera so that the determined point of focus does not change.

[0073] [Structure 4] The control device according to any one of configurations 1 to 3, characterized in that the strength of the correction in the correction means can be selected by the user.

[0074] [Composition 5] The control device according to any one of configurations 1 to 3, characterized in that the strength of the correction in the correction means is fixed.

[0075] [Composition 6] The orientation of the virtual camera is determined by pan, tilt, and roll. The correction means is Using the following equations (1) to (3), the correction amounts for pan, tilt, and roll (dPan, dTilt, dRoll) are calculated for each of the aforementioned multiple frames.

[0076]

number

[0077]

number

[0078]

number

[0079]

number

[0080]

number

[0081]

number

[0082] [Composition 7] The control device according to configuration 6, characterized in that the correction means performs the smoothing by calculating the average value of the correction amount obtained for each frame.

[0083] [Structure 8] The control device according to any one of configurations 1 to 7, characterized in that the angular velocity information for multiple frames is angular velocity information for a total of four frames, consisting of the current frame to be processed and the three most recent past frames.

[0084] [Composition 9] A control device according to any one of configurations 1 to 8, characterized in that it can select whether or not to perform correction by the correction means.

[0085] [Configuration 10] An acquisition means for acquiring operation information via an operating member of an input device that specifies a virtual camera in a virtual space, and angular velocity information of a gyro sensor provided by the input device, A determination means for determining the pose of the virtual camera based on the aforementioned operation information, Correction means for correcting the attitude of the virtual camera determined based on the angular velocity information, It has, The correction means provides a correction amount when the distance from the virtual camera to the point of fixation is greater than the first distance, and the correction amount when the distance from the virtual camera to the point of fixation is greater than the first distance, for a second distance. A control device characterized by the following features.

[0086] [Composition 11] The control device according to configuration 10, characterized in that the determination means determines the point of focus of the virtual camera based on the operation information.

[0087] [Composition 12] The control device according to configuration 11, characterized in that the correction means corrects the determined pose of the virtual camera so that the determined point of focus does not change.

[0088] [Composition 13] The orientation of the virtual camera is determined by pan, tilt, and roll. The correction means is Using equations (7) to (9) below, calculate the correction amounts (dPan, dTilt, dRoll) for pan, tilt, and roll respectively.

[0089]

number

[0090]

number

[0091]

number

[0092]

number

[0093]

number

[0094]

number

[0095] [Composition 14] The control device according to configuration 2 or 12, further comprising a calculation means for calculating the position of the virtual camera based on the point of focus determined by the determination means, the distance to the point of focus determined by the determination means, and the pose of the virtual camera corrected by the correction means.

[0096] [Composition 15] The control device according to any one of configurations 1 to 14, further comprising output means for outputting virtual camera parameters for generating a virtual viewpoint image generated based on a plurality of captured images, the parameters for which information such as the position of the virtual camera, the orientation of the virtual camera, and the field of view of the virtual camera are included.

[0097] [Composition 16] The virtual space corresponds to an imaging space in which multiple imaging devices perform imaging in synchronization. The virtual camera parameters are used to generate a virtual viewpoint image based on the multiple captured images obtained by the multiple imaging devices performing the imaging. The control device according to configuration 15, characterized by the above.

[0098] [Method 1]] An acquisition step that acquires, on a frame-by-frame basis, operation information via an operating component of an input device that specifies a virtual camera in a virtual space, and angular velocity information from a gyro sensor provided by the input device. A decision step in which the orientation of the virtual camera is determined on a frame-by-frame basis based on the aforementioned operation information, A correction step in which the orientation of the virtual camera, which is determined on a frame-by-frame basis, is corrected based on angular velocity information from multiple frames, A method for controlling a virtual camera, characterized by including the following:

[0099] [Method 2] An acquisition step of acquiring operation information via an operating member of an input device that specifies a virtual camera in a virtual space, and angular velocity information of a gyro sensor provided by the input device, A decision step to determine the pose of the virtual camera based on the aforementioned operation information, A correction step to correct the attitude of the virtual camera determined based on the angular velocity information, Includes, In the correction described above, the correction amount when the distance from the virtual camera to the point of fixation is greater than the first distance (a second distance) is smaller than the correction amount when the distance from the virtual camera to the point of fixation is a first distance. A method for controlling a virtual camera, characterized by the features described above.

[0100] [Composition 19] A program for causing a computer to function as a control device as described in any one of the items in Configuration 16.

Claims

1. An acquisition means for acquiring, on a frame-by-frame basis, operation information via an operating member of an input device that specifies a virtual camera in a virtual space, and angular velocity information from a gyro sensor provided by the input device. A determination means for determining the orientation of the virtual camera on a frame-by-frame basis based on the aforementioned operation information, Correction means for correcting the orientation of the virtual camera, which is determined on a frame-by-frame basis, based on angular velocity information from multiple frames, A control device characterized by having the following features.

2. The control device according to claim 1, characterized in that the determination means determines the point of focus of the virtual camera on a frame-by-frame basis based on the operation information.

3. The control device according to claim 2, characterized in that the correction means corrects the determined pose of the virtual camera so that the determined point of focus does not change.

4. The control device according to claim 1, characterized in that the strength of the correction in the correction means can be selected by the user.

5. The control device according to claim 1, characterized in that the strength of the correction in the correction means is fixed.

6. The orientation of the virtual camera is determined by pan, tilt, and roll. The correction means is Using the following equations (1) to (3), the correction amounts for pan, tilt, and roll (dPan, dTilt, dRoll) are calculated for each of the multiple frames: [Math 1] [Math 2] [Math 3] Here, in equations (1) to (3) above, fr represents the frame rate, and ω pan , ω Tilt , ω Roll This represents the angular velocity information for pan, tilt, and roll, respectively. Based on the calculated correction amounts (dPan, dTilt, dRoll), the corrected attitude (Pan_corr, Tilt_corr, Roll_corr) is calculated using the following equations (4) to (6). [Math 4] [Math 5] [Math 6] Here, in equations (4) to (6), dPan sm dTilt sm dRoll sm This represents the value obtained by smoothing the correction amount obtained for each frame for the aforementioned multiple frames. The control device according to feature 1.

7. The control device according to claim 6, characterized in that the correction means performs the smoothing by calculating the average value of the correction amount obtained for each frame.

8. The control device according to claim 1, characterized in that the angular velocity information for multiple frames is angular velocity information for a total of four frames, consisting of the current frame to be processed and the three most recent past frames.

9. The control device according to claim 1, characterized in that it allows the user to select whether or not to perform correction by the correction means.

10. An acquisition means for acquiring operation information via an operating member of an input device that specifies a virtual camera in a virtual space, and angular velocity information of a gyro sensor provided by the input device, A determination means for determining the pose of the virtual camera based on the aforementioned operation information, Correction means for correcting the attitude of the virtual camera determined based on the angular velocity information, It has, The correction means provides a correction amount when the distance from the virtual camera to the point of fixation is greater than the first distance, and the correction amount when the distance from the virtual camera to the point of fixation is greater than the first distance, for a second distance. A control device characterized by the following features.

11. The control device according to claim 10, characterized in that the determination means determines the point of focus of the virtual camera based on the operation information.

12. The control device according to claim 11, characterized in that the correction means corrects the determined pose of the virtual camera so that the determined point of focus does not change.

13. The orientation of the virtual camera is determined by pan, tilt, and roll. The correction means is Using equations (7) to (9) below, calculate the correction amounts (dPan, dTilt, dRoll) for pan, tilt, and roll respectively. [Number 7] [Number 8] [Number 9] Here, in the above formulas (7) to (9), fr represents the frame rate, and ω pan , ω Tilt , ω Roll represents the angular velocity information for each of pan, tilt, and roll, "R" represents the distance from the virtual camera to the fixation point of the virtual camera, and "C" is a constant for adjusting the value of "R". Based on the calculated correction amounts (dPan, dTilt, dRoll), the corrected attitude (Pan_corr, Tilt_corr, Roll_corr) is calculated using the following equations (10) to (12). [Number 10] [Math 11] [Math 12] The control device according to feature 9.

14. The control device according to claim 2, further comprising a calculation means for calculating the position of the virtual camera based on the point of focus determined by the determination means, the distance to the point of focus determined by the determination means, and the pose of the virtual camera corrected by the correction means.

15. The control device according to claim 1, further comprising output means for outputting virtual camera parameters for generating a virtual viewpoint image generated based on a plurality of captured images, the parameters of which include at least information on the position of the virtual camera, the orientation of the virtual camera, and the field of view of the virtual camera.

16. The virtual space corresponds to an imaging space in which multiple imaging devices perform imaging in synchronization. The virtual camera parameters are used to generate a virtual viewpoint image based on the multiple captured images obtained by the multiple imaging devices performing the imaging. The control device according to claim 15.

17. An acquisition step that acquires, on a frame-by-frame basis, operation information via an operating component of an input device that specifies a virtual camera in a virtual space, and angular velocity information from a gyro sensor provided by the input device. A decision step in which the orientation of the virtual camera is determined on a frame-by-frame basis based on the aforementioned operation information, A correction step in which the orientation of the virtual camera, which is determined on a frame-by-frame basis, is corrected based on angular velocity information from multiple frames, A method for controlling a virtual camera, characterized by including the following:

18. An acquisition step of acquiring operation information via an operating member of an input device that specifies a virtual camera in a virtual space, and angular velocity information of a gyro sensor provided by the input device, A decision step to determine the pose of the virtual camera based on the aforementioned operation information, A correction step to correct the attitude of the virtual camera determined based on the angular velocity information, Includes, In the correction described above, the correction amount when the distance from the virtual camera to the point of fixation is greater than the first distance (a second distance) is smaller than the correction amount when the distance from the virtual camera to the point of fixation is a first distance. A method for controlling a virtual camera characterized by the following features.

19. A program for causing a computer to execute the control method described in claim 17.

20. A program for causing a computer to execute the control method described in claim 18.

Images