Control apparatus, control method, storage medium, and program product

Abstract

The present disclosure provides a control device, a control method, a storage medium, and a program product, and aims to achieve a good camera shake feeling to a viewer in a virtual viewpoint image. For each frame, operation information for specifying a virtual camera in a virtual space via an operation member of an input device and angular velocity information from a gyroscope sensor included in the input device are obtained. Based on the obtained operation information, a posture of the virtual camera is determined for each frame. Further, the posture of the virtual camera determined for each frame is corrected based on the angular velocity information corresponding to the plurality of frames.

Description

Technical Field

[0001] This disclosure relates to control technology for virtual cameras in virtual space. Background Technology

[0002] A technique exists that involves arranging multiple camera devices at different locations to simultaneously capture images, and using these multiple images to generate an image (virtual viewpoint image) indicating the scene seen from a virtual camera (virtual viewpoint) that does not actually exist in the three-dimensional space where the image is captured. In this generation, for example, a user views the generated virtual viewpoint image on a UI screen while operating a controller with a joystick to set the position or orientation of the virtual camera. However, when it is necessary to reproduce the natural camera shake feel, as if shooting a virtual viewpoint image with a handheld camera, this is difficult to achieve using only a joystick. As a method to achieve this, the use of a gyroscope sensor for sensing the angular change (angular velocity) of rotation or tilt per unit time is considered.

[0003] For example, in the field of video games, it is common practice to install gyroscope sensors on controllers. Japanese Patent Application Publication No. 2011-239985 discloses the following technology: using angular velocity information from a gyroscope sensor installed on a controller to determine the character's action of shaking an item (sword) in a video game, so that the item's posture is approximately consistent with the posture of the controller. Summary of the Invention

[0004] The control device according to this disclosure is characterized by comprising: one or more memory storing instructions; and one or more processors executing the instructions to: for each frame, obtain, via an operating member of an input device, operational information for a virtual camera in a specified virtual space and angular velocity information from a gyroscope sensor included in the input device; for each frame, determine the attitude of the virtual camera based on the operational information; and correct the attitude of the virtual camera determined for each frame based on the angular velocity information corresponding to a plurality of frames.

[0005] The features of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. The following description of embodiments is given by way of example. Attached Figure Description

[0006] Figure 1 This is a diagram illustrating an example of the structure of a virtual viewpoint image generation system;

[0007] Figure 2 This is a diagram showing the hardware resources of the various devices that form the virtual viewpoint image generation system;

[0008] Figure 3This is a flowchart illustrating the operation process of generating and outputting virtual camera parameters based on operation information and angular velocity information according to the first embodiment;

[0009] Figure 4A and Figure 4B This is a diagram illustrating an example of a viewpoint input device;

[0010] Figure 5 It is a diagram showing how the position of a virtual camera can be uniquely specified based on the gaze point, the gaze point distance, and the virtual camera's pose;

[0011] Figure 6 This is a diagram illustrating the problem of the second embodiment; and

[0012] Figure 7 This is a flowchart illustrating the operation process of generating and outputting virtual camera parameters based on operation information and angular velocity information according to the second embodiment. Detailed Implementation

[0013] The present disclosure will be described in detail below with reference to the accompanying drawings and preferred embodiments. The structures shown in the following embodiments are merely exemplary, and the present disclosure is not limited to the structures illustrated herein.

[0014] When determining the virtual camera's attitude based on joystick operation and angular velocity sensed by a gyroscope sensor, if the controller is given unexpected behavior through other button operations, the angular velocity changes accordingly. In this case, as the inventor examined the aforementioned technology in Japanese Patent Application Publication No. 2011-239985, it was found that when the virtual camera faces an unexpected direction and the subject of interest moves out of the viewpoint, or when the viewing direction changes frequently, it becomes difficult to see the virtual viewpoint image.

[0015] First Embodiment

[0016] System Structure

[0017] Figure 1 This is a diagram illustrating an example of the structure of a system for generating virtual viewpoint images according to this embodiment. The virtual viewpoint image generation system of this embodiment includes a camera group 101, a 3D model generation device 102, a 3D model storage device 103, a virtual viewpoint image generation device 104, a display 105, a control device 110, and a viewpoint input device 120.

[0018] A camera group 101, consisting of multiple imaging devices (cameras), is arranged to surround the imaging space existing around the subject, and all cameras capture images synchronously with each other in time. In this embodiment, it is assumed that each camera in the camera group 101 captures moving images at a frame rate of 60fps.

[0019] The 3D model generation device 102 performs foreground extraction processing on each frame of the moving images acquired by each camera in the camera group 101, and generates an image indicating the outline of the subject to be used as the foreground (hereinafter referred to as "foreground outline"). Based on the generated outline image, methods such as Visual Hull are applied to generate data indicating the three-dimensional shape of the subject for each frame (commonly referred to as "3D model"). The data format of the 3D model can be polygon format, point cloud format, or voxel format, etc., without any restrictions. The 3D model storage device 103 stores the 3D model generated by the 3D model generation device 102.

[0020] The virtual viewpoint image generation device 104 generates virtual viewpoint images for each frame based on virtual camera parameters obtained from the control device 110. The generated virtual viewpoint images are output to the display 105. However, the destination of the output virtual viewpoint images is not necessarily limited to the display, and can be, for example, a streaming service or a broadcasting station. The display 105 displays the virtual viewpoint images received from the virtual viewpoint image generation device 104.

[0021] The viewpoint input device 120 is an operating device used by a user to specify the position or orientation of a virtual camera device (virtual camera) in a virtual space corresponding to the camera space of the camera group 101, and the viewpoint input device 120 includes a gyroscope sensor. A gyroscope sensor is a device used to sense angular velocity per unit time. In this embodiment, it is assumed that 3-axis angular velocities related to the X, Y, and Z axes can be sensed. However, 3-axis angles can be sensed by a combination of a 1-axis gyroscope sensor or a 2-axis gyroscope sensor. The user can continuously generate virtual viewpoint images corresponding to the desired virtual viewpoint by operating the viewpoint input device 120 to specify a new virtual camera position and orientation while viewing a virtual viewpoint image displayed on the display 105. Incidentally, the viewpoint input device 120 only needs to include a gyroscope sensor and operating components (such as joysticks or buttons) for specifying the gaze point, position, and orientation of the virtual camera according to user operation. For example, the viewpoint input device 120 can be a two-handed gamepad as described later, or it can be a one-handed gamepad.

[0022] The control device 110 generates a set of parameters (virtual camera parameters) for specifying the position and orientation of a virtual camera based on information input from the viewpoint input device 120, and outputs them to the virtual viewpoint image generation device 104. In this embodiment, it is assumed that the virtual camera parameters include at least information related to the position, orientation, and viewing angle of the virtual camera in virtual space. The control device 110 includes an input unit 111, a virtual camera parameter generation unit 112, and an output unit 119. The virtual camera parameter generation unit 112 is formed by a gaze point determination unit 113, a gaze point distance determination unit 114, a viewing angle determination unit 115, an orientation determination unit 116, an orientation correction unit 117, and a position determination unit 118. The various units of the control device 110 will be described in detail later. It should be noted that... Figure 1 The system structure shown is merely an example. For instance, the internal structure of the virtual camera parameter generation unit 112 of the control device 110 can be formed by a determination unit for calculating and determining various parameters of the virtual camera based on operational information and a correction unit for correcting the attitude of the virtual camera among the parameters determined by the determination unit.

[0023] Hardware Structure

[0024] Figure 2 It shows the formation Figure 1 The diagram shows the hardware resources of each device in the virtual viewpoint image generation system. That is, the 3D model generation device 102, the 3D model storage device 103, the virtual viewpoint image generation device 104, the control device 110, and the viewpoint input device 120 can be... Figure 2 The information processing device 200 shown is used to implement this.

[0025] The information processing device 200 includes a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a display unit 205, an operation unit 206, a communication I / F 207, and a system bus 208.

[0026] CPU 201 uses programs or data stored in ROM 202 or RAM 203 to control the entire information processing device 200. For example, the units 111 to 119 of the control device 110 are controlled by the CPU 201 executing a predetermined program. However, the information processing device 200 may include one or more dedicated hardware components different from the CPU 201, such that at least a portion of the processing performed by the CPU 201 is performed by the dedicated hardware. Examples of dedicated hardware include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and digital signal processors (DSPs).

[0027] ROM 202 stores programs that do not need to be changed, for example. RAM 203 temporarily stores programs or data provided by auxiliary storage device 204, or data provided from the outside via communication I / F 207. Auxiliary storage device 204 is formed, for example, by a hard disk drive, and stores various types of data such as image data and sound data.

[0028] The display unit 205 is formed, for example, by a liquid crystal display or an LED, and displays a graphical user interface (GUI) for the user to give instructions to the information processing device 200. The operation unit 206 is formed, for example, by a keyboard, mouse, joystick, or touch panel, and inputs various instructions to the CPU 201 in response to user operations. The CPU 201 operates as a display control unit for controlling the display unit 205 and an operation control unit for controlling the operation unit 206. In this embodiment, it is assumed that the display unit 205 and the operation unit 206 are located inside the information processing device 200, but at least one of the display unit 205 and the operation unit 206 can exist as a separate device outside the information processing device 200.

[0029] Communication I / F 207 is used for communication between information processing device 200 and external devices. When information processing device 200 has the function of wireless communication with external devices, communication I / F 207 includes an antenna. System bus 208 connects the units of information processing device 200 to each other to transmit information.

[0030] It should be noted that Figure 2 Examples of basic hardware commonly included in the various devices that form a virtual viewpoint image generation system are shown. The hardware architecture may vary depending on the purpose or function of the target device; for example, control device 110 may also include a gyroscope sensor.

[0031] Operating procedures for control equipment

[0032] Next, refer to Figure 1 The aforementioned system structure diagram and Figure 3 The flowchart shown describes in detail the operation of the control device 110 for generating and outputting virtual camera parameters based on operation information and angular velocity information input from the viewpoint input device 120. In the following description, the symbol "S" denotes a step. In this embodiment, which generates a virtual viewpoint image based on motion images captured at 60fps, each of the processes in S301 to S312 is performed for each frame every 1 / 60th of a second.

[0033] In S301, the input unit 111 obtains operation information and angular velocity information related to the current frame (the frame of interest as the object to be processed) output from the viewpoint input device 120 via the communication I / F 207. Here, the operation information indicates the operation content generated by, for example, tilting the joystick of the viewpoint input device 120. The angular velocity information indicates the angular velocity related to three axes (pan, tilt, and roll) sensed by a gyroscope sensor mounted on the viewpoint input device 120. Here, pan is the rotation angle parallel to the ground, tilt is the rotation angle perpendicular to the ground, and roll is the rotation angle around the optical axis of the virtual camera. In this embodiment, the angular velocity information is expressed in ω, each in [° / s]. pan ω Tilt and ω Roll To express. Figure 4A and Figure 4B A two-handed game board is shown as an example of a viewpoint input device 120. Figure 4A and Figure 4B The game board shown includes control components such as L1 / R1 buttons, L2 / R2 buttons, a D-pad (cross-pad), left / right sticks, and face buttons. For example, the user tilts the left stick 405 in the desired direction to change the position of the virtual camera's focused point (gaze point) on the X and Y axes. Furthermore, the user presses the L1 button 401 to raise the gaze point's position on the Z-axis, and presses the R1 button 402 to lower the gaze point's position on the Z-axis. Additionally, the user presses the L2 button 403 to gradually decrease the distance between the virtual camera and the gaze point (gaze point distance), and presses the R2 button 404 to gradually increase the gaze point distance. Furthermore, the user presses the upper part of the D-pad 407 to gradually decrease the virtual camera's field of view, and presses the lower part to gradually increase the virtual camera's field of view. Finally, the user tilts the right stick 406 in the desired direction to change the virtual camera's pan and tilt. Furthermore, among the four face buttons 408a to 408d, the user presses the upper face button 408a to rotate the tilt to the left, and presses the right face button 408b to rotate the tilt to the right. The above is an example of the operation method of the virtual camera; the mapping of each button is not limited to this. The obtained operation information and angular velocity information are sent to the virtual camera parameter generation unit 112. The virtual camera parameter generation unit 112 then uses the units in functional units 113 to 118 to execute the processes in S302 to S310.

[0034] In S302, the gaze point determination unit 113 determines the gaze point of the virtual camera based on the operation information related to the current frame obtained in S301. More specifically, the position of the gaze point in the current frame is obtained by calculating the change in the gaze point relative to the previous frame based on the operation information related to the current frame and adding the calculated change to the position of the gaze point in the previous frame. It is assumed that the gaze point always exists on the optical axis of the virtual camera. In this embodiment, the gaze point is represented by three-dimensional coordinates on three axes (X, Y, and Z axes). The three-dimensional coordinates (X, Y, Z) are represented by numerical values ​​indicating the distance from the origin, such as X = 4.0, Y = 9.0, and Z = 1.5, each in units of [m]. In this case, the origin (X, Y, Z) = (0, 0, 0) is, for example, the center of the 3D model generation range. It is also assumed that when the X-axis is specified as parallel to the ground, the Y-axis is specified as parallel to the ground and perpendicular to the X-axis, and the Z-axis is specified as perpendicular to the ground. Information related to the determined gaze point is sent to the position determination unit 117.

[0035] In S303, the gaze distance determination unit 114 determines the distance between the virtual camera and the gaze point determined in S302 (hereinafter referred to as "gaze distance"). More specifically, the gaze distance in the current frame is obtained by calculating the change in gaze distance relative to the previous frame based on operation information related to the current frame and adding the calculated change to the gaze distance in the previous frame. In this embodiment, the unit of gaze distance R is [m]. Information related to the determined gaze distance is sent to the position determination unit 117.

[0036] In S304, the viewing angle determination unit 115 determines the viewing angle of the virtual camera based on the operation information related to the current frame obtained in S301. More specifically, the viewing angle in the current frame is obtained by calculating the change in viewing angle relative to the previous frame based on the operation information related to the current frame and adding the calculated change to the viewing angle in the previous frame. Incidentally, there is a relationship where the viewing angle increases as the focal length decreases and decreases as the focal length increases (zooming), and the viewing angle is uniquely determined by the focal length. Therefore, in this embodiment, the viewing angle is specified by a value indicating the focal length (zoom value), such as zoom value = 6.0 (in [mm]). Information related to the determined viewing angle (≈ focal length) is sent to the position determination unit 117.

[0037] In S305, the attitude determination unit 116 determines the attitude of the virtual camera based on the operation information obtained in S301. More specifically, the attitude in the current frame is obtained by calculating the change in attitude relative to the previous frame based on the operation information related to the current frame and adding the calculated change to the attitude in the previous frame. In this embodiment, the attitude of the virtual camera is represented by the values ​​of three angles: pan, tilt, and roll, such as (Pan, Tilt, Roll) = (20.0, 10.0, 2.0), each ranging from -180° to 180°. Information related to the determined attitude is sent to the position determination unit 117.

[0038] In S306, the attitude correction unit 117 determines whether the function of correcting the attitude of the virtual camera based on angular velocity information from the gyroscope sensor (hereinafter referred to as the "gyroscope correction function") is enabled. The next steps branch according to the determination result. Here, the user only needs to pre-select the setting to enable or disable the gyroscope correction function, for example, via a user interface screen (UI screen) not shown. If the gyroscope correction function is enabled, S307 is executed next. If the gyroscope correction function is disabled, S310 is executed next.

[0039] In S307, the attitude correction unit 117 determines a setting value for the correction intensity used to perform the gyroscope correction function. The next steps branch according to the determination result. Here, the user only needs to pre-select the correction intensity setting, for example, using a screen structure that allows intensity selection after selecting "Enable" in the aforementioned UI screen. In this embodiment, it is assumed that "Strong" or "Weak" is set as the correction intensity setting value. If the correction intensity setting value is "Strong," then S308 is executed. If the setting value is "Weak," then S309 is executed.

[0040] In S308, the attitude correction unit 117 corrects the attitude of the virtual camera determined in S305 based on the angular velocity information related to the current frame obtained in S301. In this embodiment, it is assumed that processing is performed at a frame rate of 60 fps. In this case, the correction amount is calculated by dividing the input angular velocity in the current frame by 60. In this embodiment, since the attitude of the virtual camera is specified by pan, tilt, and roll, the correction amount (dPan, dTilt, dRoll) is obtained by the following equations (1) to (3).

[0041]

[0042] In equations (1) to (3) above, fr indicates the frame rate, and in this embodiment, the value of fr is 60. For example, it is assumed here that the input angular velocity information is (ω pan, ω tilt, ω roll = (20.0, 30.2, -12.0). In this case, the correction amount (dPan, dTilt, dRoll) = (0.333, 0.503, -0.2) is obtained by the above equations (1) to (3). The correction amount (dPan, dTilt, dRoll) calculated therefrom is added to the attitude (Pan, Tilt, Roll) determined by the attitude determination unit 116 to obtain the corrected attitude in this step. That is, the corrected attitude (Pan_corr, Tilt_corr, Roll_corr) is represented by the following equations (4) to (6).

[0043] Pan_corr=Pan+dPan…Equation (4)

[0044] Tilt_corr=Tilt+dTilt…Formula (5)

[0045] Roll_corr=Roll+dRoll…Formula (6)

[0046] In S309, the attitude correction unit 117 corrects the attitude of the virtual camera determined in S305 based on the angular velocity information related to the current frame and the immediately preceding frame obtained in S301. Overly strong attitude correction based on angular velocity information may actually make the image difficult to see. In this case, the user selects "weak". In this step, firstly, the correction amount (dPan, dTilt, dRoll) is obtained for each of the multiple frames, including the current frame and the immediately preceding frame, using the above equations (1) to (3). Then, the value (dPan) obtained by smoothing the correction amount obtained for the corresponding frame is... sm ,dTilt sm dRoll sm The corrected pose (Pan, Tilt, Roll) is added to the pose determination unit 116. That is, the corrected pose (Pan_corr, Tilt_corr, Roll_corr) in this step is represented by the following equations (7) to (9), where the fluctuations in multiple frames are smoothed.

[0047] Pan_corr=Pan+dPan SM …Formula (7)

[0048] Tilt_corr = Tilt + dTilt SM …Formula (8)

[0049] Roll_corr = Roll + dRoll SM …Formula (9)

[0050] Here, as a smoothing method, for example, consider calculating the average and using (dPan) ave ,dTilt ave dRoll ave ) as the correction amount (dPan) above sm ,dTilt sm dRoll sm However, only smoothing fluctuations across multiple frames is needed, and this method is not limited to using the average value. For example, an intermediate value could be used instead of the average.

[0051] Regarding angular velocity information relating to multiple frames, including previous frames, there is no limit to the number of frames to be considered. For example, by considering angular velocity information corresponding to a total of four frames (the current frame and three immediately preceding frames), a moderate sense of camera shake can be reproduced. Furthermore, it is only necessary to smooth out fluctuations across multiple frames. For example, median values ​​can be used instead of average values.

[0052] In S310, the position determination unit 118 determines the position of the virtual camera based on the gaze point, gaze point distance, and virtual camera pose that have been determined through processing up to the present. In this embodiment, similar to the gaze point, the position of the virtual camera is represented by three-dimensional coordinates on three axes (X, Y, and Z axes). Figure 5 This diagram illustrates how the position of a virtual camera can be uniquely specified based on the gaze point, gaze distance, and the virtual camera's pose. Figure 5 In the image, black dot 501 indicates the gaze point, and double-headed arrow 505 indicates the distance between virtual camera 502 and gaze point 501. Figure 5 It can be understood that the position of the virtual camera 502 is determined based on the gaze point distance 505, the angle 503 of the virtual camera 502 in the pan direction, and the angle 504 of the virtual camera 502 in the pitch direction, so that the virtual camera 502 faces the gaze point. Although, for the sake of explanation, the position of the gaze point 501 is... Figure 5 The origin is (0,0,0), but the position of the gaze point can be any location in the virtual space. The method for obtaining the three-dimensional coordinates (Xcam, Ycam, Zcam) indicating the position of the virtual camera differs depending on whether the gyroscope correction function is enabled or disabled. When the gyroscope correction function is enabled, the three-dimensional coordinates can be obtained using the pose (Pan_corr, Tilt_corr, Roll_corr) corrected in S308 or S309 using the following equations (10) to (12).

[0053]

[0054] In equations (10) to (12) above, R indicates the gaze distance determined in S303, and X, Y and Z indicate the three-dimensional coordinates (X, Y, Z) of the gaze point determined in S302.

[0055] On the other hand, when the gyroscope correction function is disabled, the three-dimensional coordinates (Xcam, Ycam, Zcam) indicating the position of the virtual camera are obtained by using the attitude (Pan, Tilt, Roll) determined in S305 using the following equations (13) to (15).

[0056]

[0057] In this way, virtual camera parameters are generated for each frame by the actions of each unit in the virtual camera parameter generation unit 112.

[0058] In S311, the output unit 119 transmits the virtual camera parameters of the current frame generated by the virtual camera parameter generation unit 112 to the virtual viewpoint image generation device 104 via the communication I / F 207.

[0059] In S312, it is determined whether to continue generating virtual camera parameters. For example, if operation information and angular velocity information are being continuously input from the viewpoint input device 120 and virtual camera parameters are being generated, the process returns to S301 to continue. Conversely, if the input of operation information and angular velocity information from the viewpoint input device 120 is stopped and the generation of virtual camera parameters is completed, the process exits the flow.

[0060] This concludes the virtual camera parameter generation process according to this embodiment. Incidentally, in this embodiment, the correction intensity setting is two levels, "strong" and "weak," but is not limited to this. For example, the correction intensity can be selected from three or more levels such as "strong," "normal," and "weak." Alternatively, the user can be allowed to enable or disable the gyroscope correction function, and the correction intensity can be fixed at a recommended setting (e.g., "weak").

[0061] As described above, according to this embodiment, the pose of the virtual camera, determined based on the operation information from the operation component of the viewpoint input device, is corrected based on the angular velocity information from the gyroscope sensor mounted on the viewpoint input device, thereby enabling the reproduction of a natural sense of camera shake in the virtual viewpoint image.

[0062] Incidentally, in the above embodiments, the setting value of the correction intensity is represented by two levels, "strong" and "weak," which are equivalent to enabling and disabling the function for adjusting the correction amount. In the above embodiments, when the setting value of the correction intensity is "strong," smoothing processing is not performed, and when the setting value of the correction intensity is "weak," smoothing processing is performed. Therefore, it can be said that smoothing processing is enabled and disabled.

[0063] Second Embodiment

[0064] In the first embodiment, the pose of the virtual camera, determined based on operational information from the operating mechanism, is corrected based on angular velocity information from a gyroscope sensor. However, in the method of the first embodiment, the amount of position change (movement) of the virtual camera increases with the distance from the gaze point when correcting the pose. This may lead to adverse effects, such as a tendency towards so-called motion sickness. Therefore, as a second embodiment, the following aspect is described: preventing excessive increase in the amount of movement of the virtual camera when correcting the pose based on angular velocity information from a gyroscope sensor. It should be noted that common aspects such as system structure shared by the first and second embodiments are not described, and the following mainly describes the differences.

[0065] Before describing this embodiment, refer to Figure 6 This explains the problems with this embodiment. Figure 6 In the diagram, black dot 601 indicates the gaze point, double-headed arrow 607 indicates the gaze distance of camera 602, and double-headed arrow 617 indicates the gaze distance of camera 612. Cameras 602 and 612 indicate the virtual cameras before attitude correction, and cameras 603 and 613 indicate the virtual cameras after attitude correction. Furthermore, a dashed line 604 extending from camera 602 to gaze point 601 indicates the optical axis of virtual camera 602, and a dashed line 604' extending from camera 612 to gaze point 601 indicates the optical axis of virtual camera 612. Similarly, a dashed line 605 extending from camera 603 to gaze point 601 indicates the optical axis of virtual camera 603, and a dashed line 605' extending from camera 613 to gaze point 601 indicates the optical axis of virtual camera 613. A straight double-headed arrow 607 indicates the gaze distance between virtual cameras 602 and 603, and a straight double-headed arrow 617 indicates the gaze distance between virtual cameras 612 and 613. Furthermore, the curved double-headed arrow 606 indicates the distance traveled from the virtual camera 602 before correction to the virtual camera 603 after correction, and the curved double-headed arrow 616 indicates the distance traveled from the virtual camera 612 before correction to the virtual camera 613 after correction. Figure 6 As can be seen, the movement distance increases proportionally with the distance to the gaze point. Therefore, even with angular velocity information (ω) from the gyroscope sensor...pan ω tilt ω roll Even with the same input values, a large gaze distance can cause the virtual camera's position to change significantly relative to the user's expectations. This embodiment aims to control the correction amount to prevent excessive changes in the virtual camera's position while correcting its pose.

[0066] Operating procedures for control equipment

[0067] Next, the operation of the control device 110 according to this embodiment for generating and outputting virtual camera parameters based on operation information and angular velocity information input from the viewpoint input device 120 will be described in detail. In the following description, the symbol "S" denotes a step. Compared with the first embodiment... Figure 3 The process is the same, assuming Figure 7 The process shown in the flowchart is performed for each frame every 1 / 60th of a second.

[0068] S301 to S307 are the same as in the first embodiment, and therefore will not be described again here. If the setting value of the correction intensity is determined to be "strong" in S307, S801 is executed next. If the setting value is "weak", S802 is executed next.

[0069] In S701, the attitude correction unit 117 corrects the attitude of the virtual camera determined in S305 based on the angular velocity information related to the current frame obtained in S301 and the gaze distance determined in S303. Also in this embodiment, it is assumed that processing is performed at a frame rate of 60 fps. In this case, the correction amount is calculated by dividing the input angular velocity in the current frame by 60. Furthermore, in this embodiment, the calculation is performed such that the correction amount decreases as the gaze distance increases. More specifically, the correction amount (dPan, dTilt, dRoll) is obtained using the following equations (16) to (18).

[0070]

[0071] In equations (16) to (18) above, fr indicates the frame rate, R indicates the gaze distance determined in S303, and C is a constant used to adjust the reduction of the correction amount according to the R value, which has a value in the range of 0 < 1. Here it is assumed that the gaze distance R = 9 [m] and the angular velocity information (ω pan ω tilt ω roll= (20.0, 30.2, -12.0). In this case, the correction amount (dPan, dTilt, dRoll) = (0.033, 0.05, -0.02) is obtained by the above equations (16) to (18). The correction amount is approximately 1 / 10 of the correction amount (dPan, dTilt, dRoll) = (0.333 + 0.503, -0.2) calculated based on the same angular velocity information by the method of the first embodiment described above (see the description of S308). The correction amount (dPan, dTilt, dRoll) calculated therefrom is added to the attitude (Pan, Tilt, Roll) determined by the attitude determination unit 116 to obtain the corrected attitude (Pan_corr, Tilt_corr, Roll_corr).

[0072] In S702, the attitude correction unit 117 corrects the attitude of the virtual camera determined in S305 based on the angular velocity information related to the current frame and the immediate preceding frame, as well as the gaze distance determined in S303. More specifically, firstly, the correction amount (dPan, dTilt, dRoll) is calculated for each frame in a plurality of frames, including the current frame, using the above equations (16) to (18). Next, the average value (dPan) of the calculated correction amount is obtained. ave ,dTilt ave dRoll ave Then, the obtained average value is added to the attitude (Pan, Tilt, Roll) determined by the attitude determination unit 116 to obtain the corrected attitude (Pan_corr, Tilt_corr, Roll_corr). Similar to S309 in the first embodiment, the number of previous frames to be considered for the angular velocity information is not limited.

[0073] The subsequent steps S310 and S311 are the same as those in the first embodiment, and therefore will not be described again here. This concludes the control processing of the virtual camera according to this embodiment. It should be noted that the above equations (16) to (18) used to calculate the correction amount are merely examples and can be any calculation formula, as long as the correction amount decreases as the gaze distance increases. For example, the exponential function shown in the following equations (19) to (21) can be used.

[0074] dPan=ω Pan / 60×0.5^R…Formula (19)

[0075] dTilt=ω Tilt / 60×0.5^R…Formula (20)

[0076] dRoll=ω Roll / 60×0.5^R…Formula (21)

[0077] As described above, according to this embodiment, the amount of pose correction for the virtual camera is controlled based on the gaze distance. This prevents excessive movement of the virtual camera's position when the pose correction is based on angular velocity information from a gyroscope sensor.

[0078] Other embodiments

[0079] Embodiments of this disclosure can also be implemented by a computer of a system or device that includes a storage medium (also more fully referred to as a "non-transitory computer-readable storage medium") recording computer-executable instructions (e.g., one or more programs) to perform one or more functions in the above embodiments and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing one or more functions in the above embodiments, and by the following method, wherein the computer of the system or device performs the above methods by, for example, reading from the storage medium and executing computer-executable instructions to perform one or more functions in the above embodiments and / or controlling the one or more circuits to perform one or more functions in the above embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include a network of separate computers or separate processors to read and execute the computer-executable instructions. These computer-executable instructions may, for example, be provided to the computer from a network or storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), the storage unit of a distributed computer system, or optical discs (such as CDs, DVDs, or Blu-ray discs). TM One or more of the following: flash storage devices and memory cards.

[0080] While this disclosure has been described with reference to exemplary embodiments, it should be understood that this disclosure is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

[0081] According to this disclosure, a good sense of camera shake for the viewer can be achieved in virtual viewpoint images.

Claims

1. A control device comprising: one or more memories that store instructions; and one or more processors that execute the instructions to: obtain operation information for specifying a virtual camera in a virtual space via an operation member of an input device, and angular velocity information from a gyro sensor included in the input device; determine a pose of the virtual camera based on the operation information; and correct the determined pose of the virtual camera based on the angular velocity information, wherein the correction is made so that an amount of correction in a case where a distance between the virtual camera and a gaze point of the virtual camera is a second distance that is larger than a first distance is smaller than an amount of correction in a case where the distance between the virtual camera and the gaze point of the virtual camera is the first distance. 2.The control device according to claim 1, wherein the one or more processors execute the instructions to: determine a gaze point of the virtual camera based on the operation information for each frame. 3.The control device according to claim 2, wherein the determined pose of the virtual camera for each frame is corrected so that the determined gaze point for each frame does not change. 4.The control device according to any one of claims 1 to 3, wherein the angular velocity information corresponding to a plurality of frames is angular velocity information corresponding to a total of four frames, the four frames being a current frame to be processed and three immediately preceding frames. 5.The control device according to claim 2, wherein the one or more processors execute the instructions to: calculate a position of the virtual camera based on the determined gaze point, a distance to the determined gaze point, and the corrected pose of the virtual camera. 6.The control device according to claim 1, wherein the one or more processors execute the instructions to: output a virtual camera parameter for generating a virtual viewpoint image based on a plurality of captured images, the virtual camera parameter including at least information about a position of the virtual camera, a pose of the virtual camera, and a view angle of the virtual camera. 7.The control device according to claim 6, wherein the virtual space corresponds to a photographing space in which a plurality of photographing devices photograph in synchronization with each other, and the virtual camera parameter is used to generate a virtual viewpoint image based on a plurality of captured images obtained by photographing by the plurality of photographing devices. 8.The control device according to claim 1, wherein the one or more processors execute the instructions to: determine a gaze point of the virtual camera based on the operation information. 9.A control method of a virtual camera, comprising the steps of: obtaining operation information for specifying the virtual camera in a virtual space via an operation member of an input device, and angular velocity information from a gyro sensor included in the input device; determining a pose of the virtual camera based on the operation information; and correcting the determined pose of the virtual camera based on the angular velocity information, wherein the correction is performed so that an amount of correction in a case where a distance between the virtual camera and a point of gaze of the virtual camera is a second distance that is larger than a first distance is smaller than an amount of correction in a case where the distance between the virtual camera and the point of gaze of the virtual camera is the first distance.

10. A non-transitory computer-readable storage medium storing a program for causing a computer to perform a control method of a virtual camera, the control method comprising the steps of: obtaining operation information for specifying the virtual camera in a virtual space via an operation member of an input device, and angular velocity information from a gyro sensor included in the input device; determining a posture of the virtual camera based on the operation information; and correcting the determined posture of the virtual camera based on the angular velocity information, wherein the correction is performed so that an amount of correction in a case where a distance between the virtual camera and a point of gaze of the virtual camera is a second distance that is larger than a first distance is smaller than an amount of correction in a case where the distance between the virtual camera and the point of gaze of the virtual camera is the first distance.

11. A computer program product comprising a program for causing a computer to perform a control method of a virtual camera, the control method comprising the steps of: obtaining operation information for specifying the virtual camera in a virtual space via an operation member of an input device, and angular velocity information from a gyro sensor included in the input device; determining a posture of the virtual camera based on the operation information; and correcting the determined posture of the virtual camera based on the angular velocity information, wherein the correction is performed so that an amount of correction in a case where a distance between the virtual camera and a point of gaze of the virtual camera is a second distance that is larger than a first distance is smaller than an amount of correction in a case where the distance between the virtual camera and the point of gaze of the virtual camera is the first distance.

Images